system

Abstract

We provide the system. [Solution] Image acquisition method for obtaining facial images, A feature extraction means for extracting features from acquired facial images, A matching means that compares extracted facial features with pre-registered data, Access control means that perform access control based on the matching result, A recording and alarm means that records the verification results and issues an alarm in case of an abnormality, A means of notifying another terminal of the authentication result using a remote notification means, A system that includes means for linking an access control system with a physical unlocking device.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In multiple facilities, it is necessary to improve the convenience of users and enhance security by centralizing the admission and departure systems that were conventionally often managed individually. Furthermore, it is required to reduce the loss of physical admission cards and the burden of carrying them, and enable comprehensive management in cooperation with other systems.

Means for Solving the Problems

[0005] The system according to the present invention includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the images, and a matching means for comparing the extracted features with pre-registered data. Furthermore, it includes an access control means for performing access control based on the matching results, and a recording alarm means for recording the matching results and issuing an alarm in the event of an anomaly. This enables centralized entry and exit management of facilities without the use of physical media.

[0006] A "face image" is digital data in the form of a photograph or video showing a person's face.

[0007] "Image acquisition means" refers to a device or function for acquiring facial images using a camera or sensor.

[0008] A "feature extraction method" is a process or function that extracts specific patterns or features from acquired facial images and quantifies them.

[0009] A "matching method" is a function that compares extracted features with features in a pre-registered database and evaluates the degree of match.

[0010] "Access control means" refers to systems and processes for managing each person's entry and exit from a building based on the results of facial recognition matching.

[0011] A "recording and alarm system" is a function that records authentication results and abnormal events, and issues alarms to administrators or other relevant parties as needed.

[0012] An "electronic data management system" refers to any system that uses digital data to organize, store, and utilize information. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the language used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention provides a facial recognition system for streamlining entry and exit management at various facilities. This system acquires a user's facial image using an image acquisition means and generates feature data from that image using a feature extraction means. The generated feature data is transmitted to a server and compared with data in a pre-registered database. The server's matching means determines whether the feature data matches the registered information. If a match is found, the server sends an access control signal to the terminal, and access from the terminal is authorized.

[0035] As a concrete example, when a user enters a specific building, a camera on a terminal installed at the entrance captures the user's face. The terminal temporarily stores this face image, performs feature extraction, and then sends the feature data to a server. The server immediately compares the features with a database, and if a match is found, it returns an authentication success signal to the terminal. Upon receiving this signal, the terminal unlocks the door, and the user can enter the facility.

[0036] The server records all authentication logs, and if authentication fails or an unregistered face is recognized, an alert is sent to the administrator via a recording alarm system. This system allows users to enter facilities quickly and securely without carrying a physical card. In addition, entry and exit records for each facility are centrally managed, leading to improved security.

[0037] The following describes the processing flow.

[0038] Step 1:

[0039] The device uses its camera to capture an image of the user's face when they stand in front of the entrance gate. It captures images not only from the center of the face, but also from several angles to improve recognition accuracy.

[0040] Step 2:

[0041] The device extracts features from the acquired facial image. Using a feature extraction algorithm, it identifies facial feature points and formats them as digital data. The feature data is immediately sent to the server.

[0042] Step 3:

[0043] The server receives feature data sent from the terminal. Based on the received data, it compares and matches it with features in a pre-registered employee database. If the matching result is a match, it is processed as a "success"; otherwise, it is processed as a "failure".

[0044] Step 4:

[0045] The server sends the verification result back to the terminal. If authentication is successful, it sends a signal to grant access; if it fails, it sends a signal to deny access.

[0046] Step 5:

[0047] Upon receiving a success signal, the terminal unlocks the physical gates and doors. The user is then granted access and can enter the facility.

[0048] Step 6:

[0049] The server logs all authentication events. If an unauthorized authentication or anomaly occurs, the logging alarm system immediately notifies the administrator.

[0050] (Example 1)

[0051] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0052] Current access control systems require physical authentication methods such as cards, which presents security risks and management challenges due to card loss or forgery. Furthermore, when using facial recognition, an efficient and secure authentication mechanism is required. Additionally, monitoring and prompt response to abnormal access are crucial from a security perspective, but current systems fail to adequately achieve these.

[0053] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0054] In this invention, the server includes means for acquiring an image of a person using an image acquisition device, means for extracting characteristic information from the acquired image of the person, and means for comparing the extracted characteristic information with information stored in advance. This allows users to efficiently and securely manage their access by facial recognition without using physical cards. Furthermore, a warning is issued in the event of an anomaly, enabling a rapid security response.

[0055] An "image acquisition device" is a device used to acquire images of people, and is a photographic device that includes cameras and sensors.

[0056] "Characteristic information" refers to numerical data, such as facial feature points, extracted from an acquired image of a person.

[0057] "Comparison" is a process that compares extracted characteristic information with previously stored information and analyzes the degree of similarity.

[0058] A "digital information management system" is an information system for organizing, storing, and managing electronic data.

[0059] "Authentication result" refers to information indicating the success or failure of authentication, based on the results of the comparison process.

[0060] A "communication network" is a network used for sending and receiving data, and includes infrastructure such as the internet and local networks.

[0061] This invention is a system for managing entry and exit using facial recognition of people, and consists of an image acquisition device, characteristic information extraction means, comparison means, and the like.

[0062] First, the user faces the image acquisition device at the entrance. The image acquisition device uses a camera to acquire high-resolution images. The hardware can use a standard high-performance camera. This ensures that accurate images are obtained regardless of gender, age, or other environmental conditions.

[0063] The acquired images are processed by the terminal. The terminal, as software, uses open-source image processing libraries, such as OpenCV, to extract characteristic information from the images. Here, facial feature points and shapes are analyzed and converted into numerical data. This data serves as a template for human identification.

[0064] The server receives characteristic information sent from the terminal and compares it with the stored database. The database contains pre-registered user characteristic information, and the server performs the matching process. Python and Java (registered trademark) are used as development languages ​​for the matching process, enabling rapid processing.

[0065] One concrete example is implementing this system at the entrance of a library to allow students to enter smoothly. In this case, students would only need to face the camera for authentication and would not need to carry a physical card. Furthermore, if an attempt at unauthorized access is made, the server would immediately detect it and send an alert to the administrator.

[0066] An example of a prompt to the generated AI model would be: "Please tell me how to streamline the facial recognition process in the access control system. Also, please explain in detail what steps should be taken in the event of an emergency."

[0067] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0068] Step 1:

[0069] The device acquires the user's face image using an image acquisition device. The input consists of physical environmental information and the user's face itself. The camera captures this environmental information as video data and generates a high-resolution face image. Specifically, the camera is activated on the device, and a picture is automatically taken when the user faces the camera.

[0070] Step 2:

[0071] The device extracts characteristic information using the acquired facial image. The input is the facial image acquired in step 1, and the output is characteristic information that quantifies the facial features. Image processing software (e.g., OpenCV) is used to extract feature points such as the position and distance of the eyes, nose, and mouth. Specifically, the image data is analyzed in real time, and its characteristics are extracted.

[0072] Step 3:

[0073] The terminal sends the extracted characteristic information to the server. The input is the characteristic information from step 2, and the output is the status indicating that the transmission to the server is complete. A secure protocol (e.g., HTTPS) is used for communication. Specifically, the data is converted into packets on the terminal side and sent to the server via the network.

[0074] Step 4:

[0075] The server compares the received characteristic information with data stored in a pre-stored database. The input consists of the characteristic information from step 3 and existing data in the database. The server analyzes these using a matching algorithm and determines the degree of match. Specifically, a database query is executed, and a high-speed comparison process is performed.

[0076] Step 5:

[0077] The server sends an authentication result to the terminal based on the comparison result. The input is the comparison result from step 4, and the output is an access control signal. If there is a match, an access approval signal is sent to the terminal; otherwise, a rejection signal is sent. Specifically, a signal is sent from the server to the terminal, and the terminal performs an action according to the received signal.

[0078] Step 6:

[0079] The terminal controls the physical door based on the authentication result from the server. The input is the authentication result from step 5, and the output is the opening and closing of the door. If access is approved, the terminal issues a door unlock command; if denied, it sounds an access denied sound. Specifically, the control circuit inside the terminal activates, and the door lock system physically operates.

[0080] (Application Example 1)

[0081] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0082] In modern society, while there is a demand for improved security in facilities and residences, the challenge lies in realizing access control systems that enhance authentication accuracy without compromising convenience. Furthermore, there is a lack of mechanisms for quickly and remotely verifying authentication results. Additionally, technologies that utilize facial recognition to safely and smoothly control physical barriers are also necessary.

[0083] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0084] In this invention, the server includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the acquired facial images, a matching means for comparing the extracted facial features with pre-registered data, and a means for linking the access control system with a physical unlocking device. This enables rapid and accurate access control, and by notifying another terminal of the authentication result using a remote notification means, external security monitoring becomes possible. Furthermore, by controlling the physical barrier based on the authentication result, smooth access is possible while maintaining high security.

[0085] "Image acquisition means" refers to a device or system that has the function of acquiring facial images using a camera or sensor.

[0086] "Feature extraction means" refers to a technical method or device for extracting specific feature data from acquired facial images.

[0087] A "matching device" is a device or system responsible for the process of comparing extracted facial feature data with information in a pre-registered database to determine whether or not there is a match.

[0088] "Access control means" refers to a device or system that has the function of managing permissions within a system or access to specific areas based on verification results.

[0089] A "recording and alarming device" is a device or system that records the results of authentication and has the function of reporting or issuing an alarm in the event of an abnormal situation or when an unregistered face is recognized.

[0090] A "remote notification means" is a device or system that has a mechanism for notifying another terminal of the authentication result via the internet or other communication means.

[0091] An "access control system" is a device or system that has a set of functions to manage people's entry into and exit from a facility or area and to control access based on authentication.

[0092] A "physical unlocking device" is a device or system that has the function of unlocking a physical lock based on authentication and opening and closing barriers or gates.

[0093] The system used to implement this application example employs facial recognition technology for access control and utilizes the following hardware and software.

[0094] The server is equipped with a camera device (e.g., a general-purpose camera or sensor) for acquiring facial images, and uses an image processing library (e.g., OpenCV) to extract facial features from the acquired images.

[0095] The extracted feature data is matched with pre-registered data on a cloud-based database service (e.g., AWS® DynamoDB) via the network. This matching is performed using a face recognition algorithm (e.g., a deep learning model using Dlib).

[0096] Based on the verification results, instructions are sent to an access control terminal via the network. This works in conjunction with a physical unlocking device (e.g., a smart lock system) to control the opening and closing of doors and gates. If an abnormality occurs during verification, a security notification is issued using a recording alarm system.

[0097] Furthermore, as a means of remote notification, a real-time notification service (e.g., Firebase Notifications) is used to immediately send authentication results to the user's or administrator's device. This enables rapid response from external sources.

[0098] As a concrete example, when a user approaches their home, the installed camera recognizes their face, notifies their smartphone of the result, and automatically unlocks the front door. This feature allows the user to enter smoothly without having to take out their key. Furthermore, if unexpected authentication occurs, an immediate warning is issued, allowing for corrective action.

[0099] An example of a prompt message is, "Write a process to identify users requesting access to a specific area using facial recognition and automate their entry and exit."

[0100] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0101] Step 1:

[0102] The device acquires a facial image from the camera. The input here is real-time image data from the camera, and the output is a facial image for subsequent processing. The device stores this image in temporary memory.

[0103] Step 2:

[0104] The server receives the face image and performs feature extraction using a face recognition algorithm. The input here is the face image obtained in step 1, and the output is extracted data containing facial feature points. In this process, OpenCV is used to quantify features such as the positions of the eyes and nose from the image.

[0105] Step 3:

[0106] The server sends the extracted feature data to a cloud database and compares it with pre-registered user data. The input is feature data, and the output is the authentication match or mismatch result. A cloud service is used to perform the matching operation with the feature data in the database.

[0107] Step 4:

[0108] The server generates an access instruction to the physical unlocking device based on the verification result. The input here is the verification result, and the output is an unlock or restriction instruction signal. Specifically, upon successful authentication, a smart lock control signal is sent.

[0109] Step 5:

[0110] The server sends the authentication result to the user or administrator's terminal using remote notification methods. The input is the matching result and notification content, and the output is notification data via a real-time notification service. Firebase Notifications are used in this process.

[0111] Step 6:

[0112] When an anomaly is detected, the server activates a recording and alarm system, logs the event, and issues an alarm. Inputs are an anomaly detection flag and the target image, while outputs are the alarm and recorded log. This enables monitoring and tracking of unauthorized access.

[0113] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0114] This invention combines a facial recognition access control system with an emotion engine to recognize the user's emotional state and process accordingly. The system acquires a user's facial image using an image acquisition means and generates facial features from the image using a feature extraction means. This information is transmitted to a server and compared with pre-registered data by a matching means. If the matching is successful, an access control means grants the user access.

[0115] Furthermore, it incorporates an emotion engine that analyzes the user's facial expressions to evaluate their emotions. Emotion analysis is performed in real time, identifying various emotional states, such as whether the user is stressed or happy. This information is transmitted to a server for recording and management. If the emotion data exceeds a certain range or is deemed abnormal, a recording alarm system will issue an alarm and notify the administrator.

[0116] As a concrete example, when a user enters a building, the terminal's camera captures their face. The terminal extracts facial features, analyzes emotions from facial expressions, and sends the data to a server. The server verifies the facial data, grants access, stores the emotion data, and sends an alert to the administrator if any anomalies are detected. This system goes beyond simple entry and exit management, enabling security and environmental adjustments based on user emotions.

[0117] The following describes the processing flow.

[0118] Step 1:

[0119] The device uses a camera to photograph the face of a user attempting to enter the building. Images are acquired from multiple angles to collect sufficient data to improve recognition accuracy.

[0120] Step 2:

[0121] The device extracts features from the acquired facial image. It identifies facial feature points and formats them as digital data. Furthermore, it uses an emotion engine to analyze the user's facial expressions and generate emotion data.

[0122] Step 3:

[0123] The device sends feature data and sentiment data to the server. The data is designed to be processed by the server immediately.

[0124] Step 4:

[0125] The server compares the feature data received from the terminal with pre-registered information in the database. It determines whether the information matches the registered information and decides whether access is permitted.

[0126] Step 5:

[0127] Once the server completes the facial data matching, it sends the authentication result back to the device. If access is permitted, the server sends an authentication success signal.

[0128] Step 6:

[0129] Upon receiving a signal indicating successful authentication, the terminal unlocks the gate or door. The user can then enter the facility.

[0130] Step 7:

[0131] The server records emotional data, and if an anomaly detection system detects changes or abnormalities in emotions, it uses recording alarm mechanisms to alert the administrator. In this way, the user's psychological state is continuously monitored, ensuring safety.

[0132] (Example 2)

[0133] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0134] Conventional access control systems use facial recognition technology for access control, but they have the drawback of not being able to take into account the user's emotional state, making it difficult to respond based on emotions or to detect abnormal conditions early. Furthermore, introducing emotion recognition requires enhanced security and environmental adjustments.

[0135] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0136] In this invention, the server includes an image acquisition means for acquiring facial images, an emotion analysis means for recognizing emotional states, and a recording alarm means for recording and notifying of abnormalities when the emotional state exceeds a specific range. This enables flexible access control and rapid abnormality notification based on the user's emotional state.

[0137] "Image acquisition means" refers to devices and technologies for capturing and acquiring images of a user's face.

[0138] "Feature extraction means" refers to software or technology used to identify specific feature points from acquired facial images and extract them as data.

[0139] "Matching means" refers to a technology or process that compares extracted facial features with pre-registered data to determine whether they match or not.

[0140] "Access control means" refers to systems or technologies for managing the opening and closing of physical or digital entrances and exits based on matching results.

[0141] "Emotional analysis methods" refer to technologies and processes that analyze and recognize various emotional states from a user's facial expressions.

[0142] "Recording and alarming means" refers to technologies and processes that record recognition results such as emotional states and issue alarms to administrators and related systems when an anomaly is detected.

[0143] An "electronic data processing device" refers to a digital system or device that comprehensively utilizes the above-mentioned methods to perform facial recognition and emotion analysis.

[0144] This invention is an advanced access control system that combines facial recognition and emotion analysis. It recognizes the user's face to control access, while simultaneously monitoring, recording, and managing the user's emotional state.

[0145] Specifically, the device uses a high-resolution camera to acquire an image of the user's face. The camera automatically adjusts the exposure and focus to obtain a clear image. First, facial features are extracted from this image using an image processing library such as OpenCV. In this feature extraction, feature points such as the eyes, nose, and mouth are converted into data.

[0146] The extracted feature data is sent from the device to the server. The server uses cloud-based facial recognition technology to compare it with pre-registered data. AWS Rekognition and similar technologies are used for this process, and authentication is considered successful if a match is found. Based on this authentication result, the server sends a signal to control the physical door or gate, granting the user access.

[0147] Furthermore, the device is equipped with an emotion analysis module that uses technologies such as Azure® Face API to analyze the user's emotions in real time from their facial expressions. This emotion data quantifies whether the user is feeling joy, anger, or stress, and sends it to the server.

[0148] The server records this emotional data in a database and notifies the administrator if there are emotional changes that exceed the set criteria. This notification is sent via email or system alert, allowing for a quick response.

[0149] As a concrete example, when a user visits an office building, the device takes a picture of their face and analyzes their emotions in real time. Based on this information, the device can appropriately control the user's access and adjust the environment based on their emotions (for example, by adjusting the room temperature or changing the music).

[0150] The generative AI model uses prompts such as the following:

[0151] "Please analyze the emotions expressed by the person in this photograph."

[0152] "Please output the real-time sentiment analysis results from the building's entry system."

[0153] This invention enables more flexible and sophisticated management that goes beyond simple entry and exit control, and is based on user emotions.

[0154] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0155] Step 1:

[0156] The device acquires the user's facial image. The device is installed at the facility's entrance, and when a user stands in front of the camera, the high-resolution camera activates and captures a facial image. At this time, appropriate exposure and focus are automatically adjusted. The input is the facial image captured by the camera, and the output is high-resolution digital image data.

[0157] Step 2:

[0158] The device extracts features from the acquired facial image. Specifically, it uses the OpenCV library to identify feature points such as the positions of the eyes, nose, and mouth, and extracts them as numerical data. The input is facial image data, and the output is facial feature data.

[0159] Step 3:

[0160] The terminal sends the extracted feature data to the server. The feature data is transferred to the server via a secure protocol. The input is digitized feature data, and the output is the digital data transferred to the server.

[0161] Step 4:

[0162] The server uses the received feature data to compare it with registered data. It utilizes a facial recognition system and cloud-based recognition technology to compare it with pre-registered data. The input is the transmitted feature data, and the output is the result of successful or unsuccessful authentication.

[0163] Step 5:

[0164] If authentication is successful, the server sends an access control signal to the terminal. This is how the physical gate or door opens. The input is the result of successful authentication, and the output is the activation signal of the access control device.

[0165] Step 6:

[0166] The device performs emotion analysis using facial images. It utilizes the Azure Face API to analyze emotions such as joy, anger, and stress in real time. The input is a real-time facial image, and the output is numerical data indicating the emotional state.

[0167] Step 7:

[0168] The server receives emotional data and determines anomalies based on predefined criteria. If the emotional state exceeds the criteria, the server issues an alarm and notifies the administrator. The input is emotional state data, and the output is the issuance of an alarm or the execution of a notification.

[0169] Step 8:

[0170] The server analyzes the accumulated data and uses it to improve the system. Generative AI models are used to improve recognition accuracy and add new features. The input is the accumulated data set, and the output is the specifications of the improved algorithms and new features.

[0171] (Application Example 2)

[0172] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0173] Current facial recognition systems primarily manage user entry and exit, with few systems that understand and utilize users' emotional states. Therefore, it is difficult to respond to visitors' and customers' emotions, limiting the potential for improving satisfaction and preventing potential problems. Consequently, there is a growing need for technology that can identify user emotions in real time and respond appropriately based on that information.

[0174] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0175] In this invention, the server includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the acquired facial images, a matching means for comparing the extracted facial features with pre-registered data, an access control means for performing access control based on the matching results, a recording and alarm means for recording the matching results and issuing an alarm in case of an anomaly, an emotion analysis means for analyzing the extracted emotion data and recording the information, and a means for performing corresponding processing based on the analyzed emotion data. This enables flexible responses that reflect the user's emotional state.

[0176] "Image acquisition means" refers to the device or process for acquiring a user's facial image.

[0177] A "feature extraction method" refers to an algorithm or device for analyzing and identifying specific patterns or shapes from acquired facial images.

[0178] A "matching means" is a function or process that compares extracted features with pre-registered data and evaluates the degree of agreement.

[0179] "Access control means" refers to a system for granting or denying a user physical or digital access based on the matching results.

[0180] A "recording and alarming means" is a device or program that has the function of recording information from verification results and abnormal emotions, and issuing an alarm as needed.

[0181] "Emotional analysis methods" refer to technologies and systems that detect a user's emotional state from their facial expressions and record and evaluate it as data.

[0182] "Means of handling response processing" refers to a function or process that implements specific countermeasures for the user based on the analyzed sentiment data.

[0183] In this invention, a user terminal acquires a facial image when it enters a physical store. First, the user terminal uses its internal camera to acquire a facial image of the user and extracts features for facial recognition. A facial recognition library such as OpenCV is used for this process. The extracted feature data is compared with pre-registered data, and the result is sent to the server. If the comparison is successful, the server authenticates the user and notifies the store's corresponding processing system.

[0184] Furthermore, the user terminal uses an emotion analysis engine, such as Microsoft® Azure Emotion API, to analyze the user's emotional state in real time based on the acquired facial image. This emotional data is sent to a server for recording and management. If the emotional data exceeds a certain threshold, for example, if the system determines that the user is experiencing stress, the response system sends an alert to store staff, instructing them to provide special assistance to the user.

[0185] As a concrete example, a smartphone application can simultaneously perform facial recognition and sentiment analysis when a user enters a store, enabling it to provide services tailored to the user's state. This can improve the customer experience and prevent potential problems.

[0186] An example of a prompt to input into a generative AI model is: "Please propose an application that combines customer facial recognition and sentiment analysis to provide a personalized shopping experience. Please also provide details on the specific hardware, software, and data processing methods."

[0187] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0188] Step 1:

[0189] The user terminal uses a camera to acquire an image of the user's face. At this stage, the input is the user's face as seen through the camera, and the output is digitized facial image data.

[0190] Step 2:

[0191] The device uses a face recognition library (such as OpenCV) to extract necessary features from the acquired face image. The input to this process is face image data, and the output is extracted feature data. The feature data is identified from the face image using a proprietary algorithm.

[0192] Step 3:

[0193] The extracted feature data is sent to the server and compared with a pre-registered database. The input for this step is the feature data and database information, and the output is the matching result. The server calculates the match rate and determines whether authentication is successful or not.

[0194] Step 4:

[0195] The server determines access rights based on the matching result. The input is the matching result, and the output is the access permission or denial status. The access control system uses this result to manage access to physical gates and networks.

[0196] Step 5:

[0197] The user's device sends the facial image to an emotion analysis engine (such as the Microsoft Azure Emotion API) to evaluate the emotion. The input is facial image data, and the output is data indicating the emotional state. Emotions can be determined based on changes in facial expression and facial features.

[0198] Step 6:

[0199] The server records the analyzed sentiment data, and if it exceeds a certain threshold, the response system is activated. The input is sentiment data, and the output is recording to the management database and sending an alert notification. The response system sends a notification to the store staff in the event of an anomaly and instructs them on the necessary actions.

[0200] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0201] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0202] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0203] [Second Embodiment]

[0204] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0205] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0206] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0207] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0208] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0209] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0210] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0211] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0212] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0213] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0214] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0215] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0216] This invention provides a facial recognition system for streamlining entry and exit management at various facilities. This system acquires a user's facial image using an image acquisition means and generates feature data from that image using a feature extraction means. The generated feature data is transmitted to a server and compared with data in a pre-registered database. The server's matching means determines whether the feature data matches the registered information. If a match is found, the server sends an access control signal to the terminal, and access from the terminal is authorized.

[0217] As a concrete example, when a user enters a specific building, a camera on a terminal installed at the entrance captures the user's face. The terminal temporarily stores this face image, performs feature extraction, and then sends the feature data to a server. The server immediately compares the features with a database, and if a match is found, it returns an authentication success signal to the terminal. Upon receiving this signal, the terminal unlocks the door, and the user can enter the facility.

[0218] The server records all authentication logs, and if authentication fails or an unregistered face is recognized, an alert is sent to the administrator via a recording alarm system. This system allows users to enter facilities quickly and securely without carrying a physical card. In addition, entry and exit records for each facility are centrally managed, leading to improved security.

[0219] The following describes the processing flow.

[0220] Step 1:

[0221] The device uses its camera to capture an image of the user's face when they stand in front of the entrance gate. It captures images not only from the center of the face, but also from several angles to improve recognition accuracy.

[0222] Step 2:

[0223] The device extracts features from the acquired facial image. Using a feature extraction algorithm, it identifies facial feature points and formats them as digital data. The feature data is immediately sent to the server.

[0224] Step 3:

[0225] The server receives feature data sent from the terminal. Based on the received data, it compares and matches it with features in a pre-registered employee database. If the matching result is a match, it is processed as a "success"; otherwise, it is processed as a "failure".

[0226] Step 4:

[0227] The server sends the verification result back to the terminal. If authentication is successful, it sends a signal to grant access; if it fails, it sends a signal to deny access.

[0228] Step 5:

[0229] Upon receiving a success signal, the terminal unlocks the physical gates and doors. The user is then granted access and can enter the facility.

[0230] Step 6:

[0231] The server logs all authentication events. If an unauthorized authentication or anomaly occurs, the logging alarm system immediately notifies the administrator.

[0232] (Example 1)

[0233] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0234] Current access control systems require physical authentication methods such as cards, which presents security risks and management challenges due to card loss or forgery. Furthermore, when using facial recognition, an efficient and secure authentication mechanism is required. Additionally, monitoring and prompt response to abnormal access are crucial from a security perspective, but current systems fail to adequately achieve these.

[0235] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0236] In this invention, the server includes means for acquiring an image of a person using an image acquisition device, means for extracting characteristic information from the acquired image of the person, and means for comparing the extracted characteristic information with information stored in advance. This allows users to efficiently and securely manage their access by facial recognition without using physical cards. Furthermore, a warning is issued in the event of an anomaly, enabling a rapid security response.

[0237] An "image acquisition device" is a device used to acquire images of people, and is a photographic device that includes cameras and sensors.

[0238] "Characteristic information" refers to numerical data, such as facial feature points, extracted from an acquired image of a person.

[0239] "Comparison" is a process that compares extracted characteristic information with previously stored information and analyzes the degree of similarity.

[0240] A "digital information management system" is an information system for organizing, storing, and managing electronic data.

[0241] "Authentication result" refers to information indicating the success or failure of authentication, based on the results of the comparison process.

[0242] A "communication network" is a network used for sending and receiving data, and includes infrastructure such as the internet and local networks.

[0243] This invention is a system for managing entry and exit using facial recognition of people, and consists of an image acquisition device, characteristic information extraction means, comparison means, and the like.

[0244] First, the user faces the image acquisition device at the entrance. The image acquisition device uses a camera to acquire high-resolution images. The hardware can use a standard high-performance camera. This ensures that accurate images are obtained regardless of gender, age, or other environmental conditions.

[0245] The acquired images are processed by the terminal. The terminal, as software, uses open-source image processing libraries, such as OpenCV, to extract characteristic information from the images. Here, facial feature points and shapes are analyzed and converted into numerical data. This data serves as a template for human identification.

[0246] The server receives characteristic information sent from the terminal and compares it with the stored database. The database contains pre-registered user characteristic information, and the server performs the matching process using this information. Python and Java are used as development languages ​​for the matching process, enabling rapid processing.

[0247] One concrete example is implementing this system at the entrance of a library to allow students to enter smoothly. In this case, students would only need to face the camera for authentication and would not need to carry a physical card. Furthermore, if an attempt at unauthorized access is made, the server would immediately detect it and send an alert to the administrator.

[0248] An example of a prompt to the generated AI model would be: "Please tell me how to streamline the facial recognition process in the access control system. Also, please explain in detail what steps should be taken in the event of an emergency."

[0249] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0250] Step 1:

[0251] The device acquires the user's face image using an image acquisition device. The input consists of physical environmental information and the user's face itself. The camera captures this environmental information as video data and generates a high-resolution face image. Specifically, the camera is activated on the device, and a picture is automatically taken when the user faces the camera.

[0252] Step 2:

[0253] The device extracts characteristic information using the acquired facial image. The input is the facial image acquired in step 1, and the output is characteristic information that quantifies the facial features. Image processing software (e.g., OpenCV) is used to extract feature points such as the position and distance of the eyes, nose, and mouth. Specifically, the image data is analyzed in real time, and its characteristics are extracted.

[0254] Step 3:

[0255] The terminal sends the extracted characteristic information to the server. The input is the characteristic information from step 2, and the output is the status indicating that the transmission to the server is complete. A secure protocol (e.g., HTTPS) is used for communication. Specifically, the data is converted into packets on the terminal side and sent to the server via the network.

[0256] Step 4:

[0257] The server compares the received characteristic information with data stored in a pre-stored database. The input consists of the characteristic information from step 3 and existing data in the database. The server analyzes these using a matching algorithm and determines the degree of match. Specifically, a database query is executed, and a high-speed comparison process is performed.

[0258] Step 5:

[0259] The server sends an authentication result to the terminal based on the comparison result. The input is the comparison result from step 4, and the output is an access control signal. If there is a match, an access approval signal is sent to the terminal; otherwise, a rejection signal is sent. Specifically, a signal is sent from the server to the terminal, and the terminal performs an action according to the received signal.

[0260] Step 6:

[0261] The terminal controls the physical door based on the authentication result from the server. The input is the authentication result from step 5, and the output is the opening and closing of the door. If access is approved, the terminal issues a door unlock command; if denied, it sounds an access denied sound. Specifically, the control circuit inside the terminal activates, and the door lock system physically operates.

[0262] (Application Example 1)

[0263] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0264] In modern society, while there is a demand for improved security in facilities and residences, the challenge lies in realizing access control systems that enhance authentication accuracy without compromising convenience. Furthermore, there is a lack of mechanisms for quickly and remotely verifying authentication results. Additionally, technologies that utilize facial recognition to safely and smoothly control physical barriers are also necessary.

[0265] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0266] In this invention, the server includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the acquired facial images, a matching means for comparing the extracted facial features with pre-registered data, and a means for linking the access control system with a physical unlocking device. This enables rapid and accurate access control, and by notifying another terminal of the authentication result using a remote notification means, external security monitoring becomes possible. Furthermore, by controlling the physical barrier based on the authentication result, smooth access is possible while maintaining high security.

[0267] "Image acquisition means" refers to a device or system that has the function of acquiring facial images using a camera or sensor.

[0268] "Feature extraction means" refers to a technical method or device for extracting specific feature data from acquired facial images.

[0269] A "matching device" is a device or system responsible for the process of comparing extracted facial feature data with information in a pre-registered database to determine whether or not there is a match.

[0270] "Access control means" refers to a device or system that has the function of managing permissions within a system or access to specific areas based on verification results.

[0271] A "recording and alarming device" is a device or system that records the results of authentication and has the function of reporting or issuing an alarm in the event of an abnormal situation or when an unregistered face is recognized.

[0272] A "remote notification means" is a device or system that has a mechanism for notifying another terminal of the authentication result via the internet or other communication means.

[0273] An "access control system" is a device or system that has a set of functions to manage people's entry into and exit from a facility or area and to control access based on authentication.

[0274] A "physical unlocking device" is a device or system that has the function of unlocking a physical lock based on authentication and opening and closing barriers or gates.

[0275] The system used to implement this application example employs facial recognition technology for access control and utilizes the following hardware and software.

[0276] The server is equipped with a camera device (e.g., a general-purpose camera or sensor) for acquiring facial images, and uses an image processing library (e.g., OpenCV) to extract facial features from the acquired images.

[0277] The extracted feature data is compared with pre-registered data on a cloud-based database service (e.g., AWS DynamoDB) via a network. This comparison is performed using a face recognition algorithm (e.g., a deep learning model using Dlib).

[0278] Based on the comparison result, an instruction is sent to the access control terminal through the network. This causes it to be interlocked with a physical unlocking device (e.g., a smart lock system) to control the opening and closing of doors and gates. If there is an abnormality in the comparison, a security notification is sent using a recording and warning means.

[0279] Furthermore, as a remote notification means, a real-time notification service (e.g., Firebase Notifications) is used to immediately send the authentication result to the user or administrator's terminal. This enables a prompt response from the outside.

[0280] As a specific example, when the user approaches their home, the installed camera recognizes the face, the result is notified to the smartphone, and the front door lock is automatically unlocked. With this function, the user can smoothly enter the room without taking out the key. Also, when an unexpected authentication is performed, a warning is immediately issued and a response can be made.

[0281] An example of a prompt sentence is "Describe the process of identifying users who request access to a specific area through face recognition and automating entry and exit."

[0282] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0283] Step 1:

[0284] The terminal acquires a face image from the camera device. The input here is real-time image data from the camera, and the output is a face image for subsequent processing. The terminal saves this image in temporary memory.

[0285] Step 2:

[0286] The server receives the face image and performs feature extraction using a face recognition algorithm. The input here is the face image obtained in Step 1, and the output is the extracted data including the facial feature points. In this process, OpenCV is used to digitize features such as the positions of eyes and nose from the image.

[0287] Step 3:

[0288] The server sends the extracted feature data to the cloud database and compares it with the pre-registered user data. The input is the feature data, and the output is the result of authentication match or mismatch. Using cloud services, a matching operation is performed with the feature data in the database.

[0289] Step 4:

[0290] Based on the comparison result, the server generates an access instruction to the physical unlocking device. The input here is the comparison result, and the output is an instruction signal for unlocking or restriction. Specifically, when authentication is successful, a smart lock control signal is sent.

[0291] Step 5:

[0292] The server uses remote notification means to send the authentication result to the user or administrator's terminal. The input is the comparison result and the notification content, and the output is the notification data via the real-time notification service. In this process, Firebase Notifications is utilized.

[0293] Step 6:

[0294] When an anomaly is detected, the server activates the recording and warning means, records it in the log, and issues a warning. The input is the anomaly detection flag and the target image, and the output is the warning and the recorded log. This enables the monitoring and tracking of unauthorized access.

[0295] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0296] This invention combines a facial recognition access control system with an emotion engine to recognize the user's emotional state and process accordingly. The system acquires a user's facial image using an image acquisition means and generates facial features from the image using a feature extraction means. This information is transmitted to a server and compared with pre-registered data by a matching means. If the matching is successful, an access control means grants the user access.

[0297] Furthermore, it incorporates an emotion engine that analyzes the user's facial expressions to evaluate their emotions. Emotion analysis is performed in real time, identifying various emotional states, such as whether the user is stressed or happy. This information is transmitted to a server for recording and management. If the emotion data exceeds a certain range or is deemed abnormal, a recording alarm system will issue an alarm and notify the administrator.

[0298] As a concrete example, when a user enters a building, the terminal's camera captures their face. The terminal extracts facial features, analyzes emotions from facial expressions, and sends the data to a server. The server verifies the facial data, grants access, stores the emotion data, and sends an alert to the administrator if any anomalies are detected. This system goes beyond simple entry and exit management, enabling security and environmental adjustments based on user emotions.

[0299] The following describes the processing flow.

[0300] Step 1:

[0301] The device uses a camera to photograph the face of a user attempting to enter the building. Images are acquired from multiple angles to collect sufficient data to improve recognition accuracy.

[0302] Step 2:

[0303] The terminal extracts features from the acquired face image. It identifies the facial feature points and formalizes them as digital data. Furthermore, it uses an emotion engine to analyze the emotion from the user's expression and generates emotion data.

[0304] Step 3:

[0305] The terminal sends the feature data and emotion data to the server. The data is designed to be processed immediately by the server.

[0306] Step 4:

[0307] The server compares the feature data received from the terminal with the pre-registered information in the database. It determines whether it matches the registered information and decides whether access is permitted.

[0308] Step 5:

[0309] When the server completes the verification of the face data, it returns the authentication result to the terminal. If access is permitted, the server sends a signal of successful authentication.

[0310] Step 6:

[0311] When the terminal receives the signal of successful authentication, it unlocks the gate or door lock. The user can enter the facility.

[0312] Step 7:

[0313] The server records the emotion data and, when a change or abnormality in the emotion is recognized by the anomaly detection system, it issues an alarm to the administrator by utilizing the recording alarm means. In this way, the psychological state of the user is continuously monitored to ensure safety.

[0314] (Example 2)

[0315] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0316] Conventional access control systems use facial recognition technology for access control, but they have the drawback of not being able to take into account the user's emotional state, making it difficult to respond based on emotions or to detect abnormal conditions early. Furthermore, introducing emotion recognition requires enhanced security and environmental adjustments.

[0317] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0318] In this invention, the server includes an image acquisition means for acquiring facial images, an emotion analysis means for recognizing emotional states, and a recording alarm means for recording and notifying of abnormalities when the emotional state exceeds a specific range. This enables flexible access control and rapid abnormality notification based on the user's emotional state.

[0319] "Image acquisition means" refers to devices and technologies for capturing and acquiring images of a user's face.

[0320] "Feature extraction means" refers to software or technology used to identify specific feature points from acquired facial images and extract them as data.

[0321] "Matching means" refers to a technology or process that compares extracted facial features with pre-registered data to determine whether they match or not.

[0322] "Access control means" refers to systems or technologies for managing the opening and closing of physical or digital entrances and exits based on matching results.

[0323] "Emotional analysis methods" refer to technologies and processes that analyze and recognize various emotional states from a user's facial expressions.

[0324] "Recording and alarming means" refers to technologies and processes that record recognition results such as emotional states and issue alarms to administrators and related systems when an anomaly is detected.

[0325] An "electronic data processing device" refers to a digital system or device that comprehensively utilizes the above-mentioned methods to perform facial recognition and emotion analysis.

[0326] This invention is an advanced access control system that combines facial recognition and emotion analysis. It recognizes the user's face to control access, while simultaneously monitoring, recording, and managing the user's emotional state.

[0327] Specifically, the device uses a high-resolution camera to acquire an image of the user's face. The camera automatically adjusts the exposure and focus to obtain a clear image. First, facial features are extracted from this image using an image processing library such as OpenCV. In this feature extraction, feature points such as the eyes, nose, and mouth are converted into data.

[0328] The extracted feature data is sent from the device to the server. The server uses cloud-based facial recognition technology to compare it with pre-registered data. AWS Rekognition and similar technologies are used for this process, and authentication is considered successful if a match is found. Based on this authentication result, the server sends a signal to control the physical door or gate, granting the user access.

[0329] Furthermore, the device is equipped with an emotion analysis module that uses technologies such as the Azure Face API to analyze the user's emotions from their facial expressions in real time. This emotion data quantifies whether the user is feeling joy, anger, or stress, and sends it to the server.

[0330] The server records this emotional data in a database and notifies the administrator if there are emotional changes that exceed the set criteria. This notification is sent via email or system alert, allowing for a quick response.

[0331] As a concrete example, when a user visits an office building, the device takes a picture of their face and analyzes their emotions in real time. Based on this information, the device can appropriately control the user's access and adjust the environment based on their emotions (for example, by adjusting the room temperature or changing the music).

[0332] The generative AI model uses prompts such as the following:

[0333] "Please analyze the emotions expressed by the person in this photograph."

[0334] "Please output the real-time sentiment analysis results from the building's entry system."

[0335] This invention enables more flexible and sophisticated management that goes beyond simple entry and exit control, and is based on user emotions.

[0336] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0337] Step 1:

[0338] The device acquires the user's facial image. The device is installed at the facility's entrance, and when a user stands in front of the camera, the high-resolution camera activates and captures a facial image. At this time, appropriate exposure and focus are automatically adjusted. The input is the facial image captured by the camera, and the output is high-resolution digital image data.

[0339] Step 2:

[0340] The device extracts features from the acquired facial image. Specifically, it uses the OpenCV library to identify feature points such as the positions of the eyes, nose, and mouth, and extracts them as numerical data. The input is facial image data, and the output is facial feature data.

[0341] Step 3:

[0342] The terminal sends the extracted feature data to the server. The feature data is transferred to the server via a secure protocol. The input is digitized feature data, and the output is the digital data transferred to the server.

[0343] Step 4:

[0344] The server uses the received feature data to compare it with registered data. It utilizes a facial recognition system and cloud-based recognition technology to compare it with pre-registered data. The input is the transmitted feature data, and the output is the result of successful or unsuccessful authentication.

[0345] Step 5:

[0346] If authentication is successful, the server sends an access control signal to the terminal. This is how the physical gate or door opens. The input is the result of successful authentication, and the output is the activation signal of the access control device.

[0347] Step 6:

[0348] The device performs emotion analysis using facial images. It utilizes the Azure Face API to analyze emotions such as joy, anger, and stress in real time. The input is a real-time facial image, and the output is numerical data indicating the emotional state.

[0349] Step 7:

[0350] The server receives emotional data and determines anomalies based on predefined criteria. If the emotional state exceeds the criteria, the server issues an alarm and notifies the administrator. The input is emotional state data, and the output is the issuance of an alarm or the execution of a notification.

[0351] Step 8:

[0352] The server analyzes the accumulated data and uses it to improve the system. Generative AI models are used to improve recognition accuracy and add new features. The input is the accumulated data set, and the output is the specifications of the improved algorithms and new features.

[0353] (Application Example 2)

[0354] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0355] Current facial recognition systems primarily manage user entry and exit, with few systems that understand and utilize users' emotional states. Therefore, it is difficult to respond to visitors' and customers' emotions, limiting the potential for improving satisfaction and preventing potential problems. Consequently, there is a growing need for technology that can identify user emotions in real time and respond appropriately based on that information.

[0356] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0357] In this invention, the server includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the acquired facial images, a matching means for comparing the extracted facial features with pre-registered data, an access control means for performing access control based on the matching results, a recording and alarm means for recording the matching results and issuing an alarm in case of an anomaly, an emotion analysis means for analyzing the extracted emotion data and recording the information, and a means for performing corresponding processing based on the analyzed emotion data. This enables flexible responses that reflect the user's emotional state.

[0358] "Image acquisition means" refers to the device or process for acquiring a user's facial image.

[0359] A "feature extraction method" refers to an algorithm or device for analyzing and identifying specific patterns or shapes from acquired facial images.

[0360] A "matching means" is a function or process that compares extracted features with pre-registered data and evaluates the degree of agreement.

[0361] "Access control means" refers to a system for granting or denying a user physical or digital access based on the matching results.

[0362] A "recording and alarming means" is a device or program that has the function of recording information from verification results and abnormal emotions, and issuing an alarm as needed.

[0363] "Emotional analysis methods" refer to technologies and systems that detect a user's emotional state from their facial expressions and record and evaluate it as data.

[0364] "Means of handling response processing" refers to a function or process that implements specific countermeasures for the user based on the analyzed sentiment data.

[0365] In this invention, a user terminal acquires a facial image when it enters a physical store. First, the user terminal uses its internal camera to acquire a facial image of the user and extracts features for facial recognition. A facial recognition library such as OpenCV is used for this process. The extracted feature data is compared with pre-registered data, and the result is sent to the server. If the comparison is successful, the server authenticates the user and notifies the store's corresponding processing system.

[0366] Furthermore, the user terminal uses an emotion analysis engine, such as the Microsoft Azure Emotion API, to analyze the user's emotional state in real time based on the acquired facial image. This emotional data is sent to a server for recording and management. If the emotional data exceeds a certain threshold, for example, if the system determines that the user is experiencing stress, the response processing system sends an alert to store staff, instructing them to provide special assistance to the user.

[0367] As a concrete example, a smartphone application can simultaneously perform facial recognition and sentiment analysis when a user enters a store, enabling it to provide services tailored to the user's state. This can improve the customer experience and prevent potential problems.

[0368] An example of a prompt to input into a generative AI model is: "Please propose an application that combines customer facial recognition and sentiment analysis to provide a personalized shopping experience. Please also provide details on the specific hardware, software, and data processing methods."

[0369] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0370] Step 1:

[0371] The user terminal uses a camera to acquire an image of the user's face. At this stage, the input is the user's face as seen through the camera, and the output is digitized facial image data.

[0372] Step 2:

[0373] The device uses a face recognition library (such as OpenCV) to extract necessary features from the acquired face image. The input to this process is face image data, and the output is extracted feature data. The feature data is identified from the face image using a proprietary algorithm.

[0374] Step 3:

[0375] The extracted feature data is sent to the server and compared with a pre-registered database. The input for this step is the feature data and database information, and the output is the matching result. The server calculates the match rate and determines whether authentication is successful or not.

[0376] Step 4:

[0377] The server determines access rights based on the matching result. The input is the matching result, and the output is the access permission or denial status. The access control system uses this result to manage access to physical gates and networks.

[0378] Step 5:

[0379] The user's device sends the facial image to an emotion analysis engine (such as the Microsoft Azure Emotion API) to evaluate the emotion. The input is facial image data, and the output is data indicating the emotional state. Emotions can be determined based on changes in facial expression and facial features.

[0380] Step 6:

[0381] The server records the analyzed sentiment data, and if it exceeds a certain threshold, the response system is activated. The input is sentiment data, and the output is recording to the management database and sending an alert notification. The response system sends a notification to the store staff in the event of an anomaly and instructs them on the necessary actions.

[0382] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0383] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0384] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0385] [Third Embodiment]

[0386] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0387] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0388] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0389] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0390] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0392] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0393] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0394] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0395] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0396] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0397] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0398] This invention provides a facial recognition system for streamlining entry and exit management at various facilities. This system acquires a user's facial image using an image acquisition means and generates feature data from that image using a feature extraction means. The generated feature data is transmitted to a server and compared with data in a pre-registered database. The server's matching means determines whether the feature data matches the registered information. If a match is found, the server sends an access control signal to the terminal, and access from the terminal is authorized.

[0399] As a concrete example, when a user enters a specific building, a camera on a terminal installed at the entrance captures the user's face. The terminal temporarily stores this face image, performs feature extraction, and then sends the feature data to a server. The server immediately compares the features with a database, and if a match is found, it returns an authentication success signal to the terminal. Upon receiving this signal, the terminal unlocks the door, and the user can enter the facility.

[0400] The server records all authentication logs, and if authentication fails or an unregistered face is recognized, an alert is sent to the administrator via a recording alarm system. This system allows users to enter facilities quickly and securely without carrying a physical card. In addition, entry and exit records for each facility are centrally managed, leading to improved security.

[0401] The following describes the processing flow.

[0402] Step 1:

[0403] The device uses its camera to capture an image of the user's face when they stand in front of the entrance gate. It captures images not only from the center of the face, but also from several angles to improve recognition accuracy.

[0404] Step 2:

[0405] The device extracts features from the acquired facial image. Using a feature extraction algorithm, it identifies facial feature points and formats them as digital data. The feature data is immediately sent to the server.

[0406] Step 3:

[0407] The server receives feature data sent from the terminal. Based on the received data, it compares and matches it with features in a pre-registered employee database. If the matching result is a match, it is processed as a "success"; otherwise, it is processed as a "failure".

[0408] Step 4:

[0409] The server sends the verification result back to the terminal. If authentication is successful, it sends a signal to grant access; if it fails, it sends a signal to deny access.

[0410] Step 5:

[0411] Upon receiving a success signal, the terminal unlocks the physical gates and doors. The user is then granted access and can enter the facility.

[0412] Step 6:

[0413] The server logs all authentication events. If an unauthorized authentication or anomaly occurs, the logging alarm system immediately notifies the administrator.

[0414] (Example 1)

[0415] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0416] Current access control systems require physical authentication methods such as cards, which presents security risks and management challenges due to card loss or forgery. Furthermore, when using facial recognition, an efficient and secure authentication mechanism is required. Additionally, monitoring and prompt response to abnormal access are crucial from a security perspective, but current systems fail to adequately achieve these.

[0417] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0418] In this invention, the server includes means for acquiring an image of a person using an image acquisition device, means for extracting characteristic information from the acquired image of the person, and means for comparing the extracted characteristic information with information stored in advance. This allows users to efficiently and securely manage their access by facial recognition without using physical cards. Furthermore, a warning is issued in the event of an anomaly, enabling a rapid security response.

[0419] An "image acquisition device" is a device used to acquire images of people, and is a photographic device that includes cameras and sensors.

[0420] "Characteristic information" refers to numerical data, such as facial feature points, extracted from an acquired image of a person.

[0421] "Comparison" is a process that compares extracted characteristic information with previously stored information and analyzes the degree of similarity.

[0422] A "digital information management system" is an information system for organizing, storing, and managing electronic data.

[0423] "Authentication result" refers to information indicating the success or failure of authentication, based on the results of the comparison process.

[0424] A "communication network" is a network used for sending and receiving data, and includes infrastructure such as the internet and local networks.

[0425] This invention is a system for managing entry and exit using facial recognition of people, and consists of an image acquisition device, characteristic information extraction means, comparison means, and the like.

[0426] First, the user faces the image acquisition device at the entrance. The image acquisition device uses a camera to acquire high-resolution images. The hardware can use a standard high-performance camera. This ensures that accurate images are obtained regardless of gender, age, or other environmental conditions.

[0427] The acquired images are processed by the terminal. The terminal, as software, uses open-source image processing libraries, such as OpenCV, to extract characteristic information from the images. Here, facial feature points and shapes are analyzed and converted into numerical data. This data serves as a template for human identification.

[0428] The server receives characteristic information sent from the terminal and compares it with the stored database. The database contains pre-registered user characteristic information, and the server performs the matching process using this information. Python and Java are used as development languages ​​for the matching process, enabling rapid processing.

[0429] One concrete example is implementing this system at the entrance of a library to allow students to enter smoothly. In this case, students would only need to face the camera for authentication and would not need to carry a physical card. Furthermore, if an attempt at unauthorized access is made, the server would immediately detect it and send an alert to the administrator.

[0430] An example of a prompt to the generated AI model would be: "Please tell me how to streamline the facial recognition process in the access control system. Also, please explain in detail what steps should be taken in the event of an emergency."

[0431] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0432] Step 1:

[0433] The device acquires the user's face image using an image acquisition device. The input consists of physical environmental information and the user's face itself. The camera captures this environmental information as video data and generates a high-resolution face image. Specifically, the camera is activated on the device, and a picture is automatically taken when the user faces the camera.

[0434] Step 2:

[0435] The device extracts characteristic information using the acquired facial image. The input is the facial image acquired in step 1, and the output is characteristic information that quantifies the facial features. Image processing software (e.g., OpenCV) is used to extract feature points such as the position and distance of the eyes, nose, and mouth. Specifically, the image data is analyzed in real time, and its characteristics are extracted.

[0436] Step 3:

[0437] The terminal sends the extracted characteristic information to the server. The input is the characteristic information from step 2, and the output is the status indicating that the transmission to the server is complete. A secure protocol (e.g., HTTPS) is used for communication. Specifically, the data is converted into packets on the terminal side and sent to the server via the network.

[0438] Step 4:

[0439] The server compares the received characteristic information with data stored in a pre-stored database. The input consists of the characteristic information from step 3 and existing data in the database. The server analyzes these using a matching algorithm and determines the degree of match. Specifically, a database query is executed, and a high-speed comparison process is performed.

[0440] Step 5:

[0441] The server sends an authentication result to the terminal based on the comparison result. The input is the comparison result from step 4, and the output is an access control signal. If there is a match, an access approval signal is sent to the terminal; otherwise, a rejection signal is sent. Specifically, a signal is sent from the server to the terminal, and the terminal performs an action according to the received signal.

[0442] Step 6:

[0443] The terminal controls the physical door based on the authentication result from the server. The input is the authentication result from step 5, and the output is the opening and closing of the door. If access is approved, the terminal issues a door unlock command; if denied, it sounds an access denied sound. Specifically, the control circuit inside the terminal activates, and the door lock system physically operates.

[0444] (Application Example 1)

[0445] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0446] In modern society, while there is a demand for improved security in facilities and residences, the challenge lies in realizing access control systems that enhance authentication accuracy without compromising convenience. Furthermore, there is a lack of mechanisms for quickly and remotely verifying authentication results. Additionally, technologies that utilize facial recognition to safely and smoothly control physical barriers are also necessary.

[0447] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0448] In this invention, the server includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the acquired facial images, a matching means for comparing the extracted facial features with pre-registered data, and a means for linking the access control system with a physical unlocking device. This enables rapid and accurate access control, and by notifying another terminal of the authentication result using a remote notification means, external security monitoring becomes possible. Furthermore, by controlling the physical barrier based on the authentication result, smooth access is possible while maintaining high security.

[0449] "Image acquisition means" refers to a device or system that has the function of acquiring facial images using a camera or sensor.

[0450] "Feature extraction means" refers to a technical method or device for extracting specific feature data from acquired facial images.

[0451] A "matching device" is a device or system responsible for the process of comparing extracted facial feature data with information in a pre-registered database to determine whether or not there is a match.

[0452] "Access control means" refers to a device or system that has the function of managing permissions within a system or access to specific areas based on verification results.

[0453] A "recording and alarming device" is a device or system that records the results of authentication and has the function of reporting or issuing an alarm in the event of an abnormal situation or when an unregistered face is recognized.

[0454] A "remote notification means" is a device or system that has a mechanism for notifying another terminal of the authentication result via the internet or other communication means.

[0455] An "access control system" is a device or system that has a set of functions to manage people's entry into and exit from a facility or area and to control access based on authentication.

[0456] A "physical unlocking device" is a device or system that has the function of unlocking a physical lock based on authentication and opening and closing barriers or gates.

[0457] The system used to implement this application example employs facial recognition technology for access control and utilizes the following hardware and software.

[0458] The server is equipped with a camera device (e.g., a general-purpose camera or sensor) for acquiring facial images, and uses an image processing library (e.g., OpenCV) to extract facial features from the acquired images.

[0459] The extracted feature data is matched against pre-registered data on a cloud-based database service (e.g., AWS DynamoDB) via the network. This matching is performed using a face recognition algorithm (e.g., a deep learning model using Dlib).

[0460] Based on the verification results, instructions are sent to an access control terminal via the network. This works in conjunction with a physical unlocking device (e.g., a smart lock system) to control the opening and closing of doors and gates. If an abnormality occurs during verification, a security notification is issued using a recording alarm system.

[0461] Furthermore, as a means of remote notification, a real-time notification service (e.g., Firebase Notifications) is used to immediately send authentication results to the user's or administrator's device. This enables rapid response from external sources.

[0462] As a concrete example, when a user approaches their home, the installed camera recognizes their face, notifies their smartphone of the result, and automatically unlocks the front door. This feature allows the user to enter smoothly without having to take out their key. Furthermore, if unexpected authentication occurs, an immediate warning is issued, allowing for corrective action.

[0463] An example of a prompt message is, "Write a process to identify users requesting access to a specific area using facial recognition and automate their entry and exit."

[0464] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0465] Step 1:

[0466] The device acquires a facial image from the camera. The input here is real-time image data from the camera, and the output is a facial image for subsequent processing. The device stores this image in temporary memory.

[0467] Step 2:

[0468] The server receives the face image and performs feature extraction using a face recognition algorithm. The input here is the face image obtained in step 1, and the output is extracted data containing facial feature points. In this process, OpenCV is used to quantify features such as the positions of the eyes and nose from the image.

[0469] Step 3:

[0470] The server sends the extracted feature data to a cloud database and compares it with pre-registered user data. The input is feature data, and the output is the authentication match or mismatch result. A cloud service is used to perform the matching operation with the feature data in the database.

[0471] Step 4:

[0472] The server generates an access instruction to the physical unlocking device based on the verification result. The input here is the verification result, and the output is an unlock or restriction instruction signal. Specifically, upon successful authentication, a smart lock control signal is sent.

[0473] Step 5:

[0474] The server sends the authentication result to the user or administrator's terminal using remote notification methods. The input is the matching result and notification content, and the output is notification data via a real-time notification service. Firebase Notifications are used in this process.

[0475] Step 6:

[0476] When an anomaly is detected, the server activates a recording and alarm system, logs the event, and issues an alarm. Inputs are an anomaly detection flag and the target image, while outputs are the alarm and recorded log. This enables monitoring and tracking of unauthorized access.

[0477] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0478] This invention combines a facial recognition access control system with an emotion engine to recognize the user's emotional state and process accordingly. The system acquires a user's facial image using an image acquisition means and generates facial features from the image using a feature extraction means. This information is transmitted to a server and compared with pre-registered data by a matching means. If the matching is successful, an access control means grants the user access.

[0479] Furthermore, it incorporates an emotion engine that analyzes the user's facial expressions to evaluate their emotions. Emotion analysis is performed in real time, identifying various emotional states, such as whether the user is stressed or happy. This information is transmitted to a server for recording and management. If the emotion data exceeds a certain range or is deemed abnormal, a recording alarm system will issue an alarm and notify the administrator.

[0480] As a concrete example, when a user enters a building, the terminal's camera captures their face. The terminal extracts facial features, analyzes emotions from facial expressions, and sends the data to a server. The server verifies the facial data, grants access, stores the emotion data, and sends an alert to the administrator if any anomalies are detected. This system goes beyond simple entry and exit management, enabling security and environmental adjustments based on user emotions.

[0481] The following describes the processing flow.

[0482] Step 1:

[0483] The device uses a camera to photograph the face of a user attempting to enter the building. Images are acquired from multiple angles to collect sufficient data to improve recognition accuracy.

[0484] Step 2:

[0485] The device extracts features from the acquired facial image. It identifies facial feature points and formats them as digital data. Furthermore, it uses an emotion engine to analyze the user's facial expressions and generate emotion data.

[0486] Step 3:

[0487] The device sends feature data and sentiment data to the server. The data is designed to be processed by the server immediately.

[0488] Step 4:

[0489] The server compares the feature data received from the terminal with pre-registered information in the database. It determines whether the information matches the registered information and decides whether access is permitted.

[0490] Step 5:

[0491] Once the server completes the facial data matching, it sends the authentication result back to the device. If access is permitted, the server sends an authentication success signal.

[0492] Step 6:

[0493] Upon receiving a signal indicating successful authentication, the terminal unlocks the gate or door. The user can then enter the facility.

[0494] Step 7:

[0495] The server records emotional data, and if an anomaly detection system detects changes or abnormalities in emotions, it uses recording alarm mechanisms to alert the administrator. In this way, the user's psychological state is continuously monitored, ensuring safety.

[0496] (Example 2)

[0497] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0498] Conventional access control systems use facial recognition technology for access control, but they have the drawback of not being able to take into account the user's emotional state, making it difficult to respond based on emotions or to detect abnormal conditions early. Furthermore, introducing emotion recognition requires enhanced security and environmental adjustments.

[0499] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0500] In this invention, the server includes an image acquisition means for acquiring facial images, an emotion analysis means for recognizing emotional states, and a recording alarm means for recording and notifying of abnormalities when the emotional state exceeds a specific range. This enables flexible access control and rapid abnormality notification based on the user's emotional state.

[0501] "Image acquisition means" refers to devices and technologies for capturing and acquiring images of a user's face.

[0502] "Feature extraction means" refers to software or technology used to identify specific feature points from acquired facial images and extract them as data.

[0503] "Matching means" refers to a technology or process that compares extracted facial features with pre-registered data to determine whether they match or not.

[0504] "Access control means" refers to systems or technologies for managing the opening and closing of physical or digital entrances and exits based on matching results.

[0505] "Emotional analysis methods" refer to technologies and processes that analyze and recognize various emotional states from a user's facial expressions.

[0506] "Recording and alarming means" refers to technologies and processes that record recognition results such as emotional states and issue alarms to administrators and related systems when an anomaly is detected.

[0507] An "electronic data processing device" refers to a digital system or device that comprehensively utilizes the above-mentioned methods to perform facial recognition and emotion analysis.

[0508] This invention is an advanced access control system that combines facial recognition and emotion analysis. It recognizes the user's face to control access, while simultaneously monitoring, recording, and managing the user's emotional state.

[0509] Specifically, the device uses a high-resolution camera to acquire an image of the user's face. The camera automatically adjusts the exposure and focus to obtain a clear image. First, facial features are extracted from this image using an image processing library such as OpenCV. In this feature extraction, feature points such as the eyes, nose, and mouth are converted into data.

[0510] The extracted feature data is sent from the device to the server. The server uses cloud-based facial recognition technology to compare it with pre-registered data. AWS Rekognition and similar technologies are used for this process, and authentication is considered successful if a match is found. Based on this authentication result, the server sends a signal to control the physical door or gate, granting the user access.

[0511] Furthermore, the device is equipped with an emotion analysis module that uses technologies such as the Azure Face API to analyze the user's emotions from their facial expressions in real time. This emotion data quantifies whether the user is feeling joy, anger, or stress, and sends it to the server.

[0512] The server records this emotional data in a database and notifies the administrator if there are emotional changes that exceed the set criteria. This notification is sent via email or system alert, allowing for a quick response.

[0513] As a concrete example, when a user visits an office building, the device takes a picture of their face and analyzes their emotions in real time. Based on this information, the device can appropriately control the user's access and adjust the environment based on their emotions (for example, by adjusting the room temperature or changing the music).

[0514] The generative AI model uses prompts such as the following:

[0515] "Please analyze the emotions expressed by the person in this photograph."

[0516] "Please output the real-time sentiment analysis results from the building's entry system."

[0517] This invention enables more flexible and sophisticated management that goes beyond simple entry and exit control, and is based on user emotions.

[0518] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0519] Step 1:

[0520] The device acquires the user's facial image. The device is installed at the facility's entrance, and when a user stands in front of the camera, the high-resolution camera activates and captures a facial image. At this time, appropriate exposure and focus are automatically adjusted. The input is the facial image captured by the camera, and the output is high-resolution digital image data.

[0521] Step 2:

[0522] The device extracts features from the acquired facial image. Specifically, it uses the OpenCV library to identify feature points such as the positions of the eyes, nose, and mouth, and extracts them as numerical data. The input is facial image data, and the output is facial feature data.

[0523] Step 3:

[0524] The terminal sends the extracted feature data to the server. The feature data is transferred to the server via a secure protocol. The input is digitized feature data, and the output is the digital data transferred to the server.

[0525] Step 4:

[0526] The server uses the received feature data to compare it with registered data. It utilizes a facial recognition system and cloud-based recognition technology to compare it with pre-registered data. The input is the transmitted feature data, and the output is the result of successful or unsuccessful authentication.

[0527] Step 5:

[0528] If authentication is successful, the server sends an access control signal to the terminal. This is how the physical gate or door opens. The input is the result of successful authentication, and the output is the activation signal of the access control device.

[0529] Step 6:

[0530] The device performs emotion analysis using facial images. It utilizes the Azure Face API to analyze emotions such as joy, anger, and stress in real time. The input is a real-time facial image, and the output is numerical data indicating the emotional state.

[0531] Step 7:

[0532] The server receives emotional data and determines anomalies based on predefined criteria. If the emotional state exceeds the criteria, the server issues an alarm and notifies the administrator. The input is emotional state data, and the output is the issuance of an alarm or the execution of a notification.

[0533] Step 8:

[0534] The server analyzes the accumulated data and uses it to improve the system. Generative AI models are used to improve recognition accuracy and add new features. The input is the accumulated data set, and the output is the specifications of the improved algorithms and new features.

[0535] (Application Example 2)

[0536] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0537] Current facial recognition systems primarily manage user entry and exit, with few systems that understand and utilize users' emotional states. Therefore, it is difficult to respond to visitors' and customers' emotions, limiting the potential for improving satisfaction and preventing potential problems. Consequently, there is a growing need for technology that can identify user emotions in real time and respond appropriately based on that information.

[0538] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0539] In this invention, the server includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the acquired facial images, a matching means for comparing the extracted facial features with pre-registered data, an access control means for performing access control based on the matching results, a recording and alarm means for recording the matching results and issuing an alarm in case of an anomaly, an emotion analysis means for analyzing the extracted emotion data and recording the information, and a means for performing corresponding processing based on the analyzed emotion data. This enables flexible responses that reflect the user's emotional state.

[0540] "Image acquisition means" refers to the device or process for acquiring a user's facial image.

[0541] A "feature extraction method" refers to an algorithm or device for analyzing and identifying specific patterns or shapes from acquired facial images.

[0542] A "matching means" is a function or process that compares extracted features with pre-registered data and evaluates the degree of agreement.

[0543] "Access control means" refers to a system for granting or denying a user physical or digital access based on the matching results.

[0544] A "recording and alarming means" is a device or program that has the function of recording information from verification results and abnormal emotions, and issuing an alarm as needed.

[0545] "Emotional analysis methods" refer to technologies and systems that detect a user's emotional state from their facial expressions and record and evaluate it as data.

[0546] "Means of handling response processing" refers to a function or process that implements specific countermeasures for the user based on the analyzed sentiment data.

[0547] In this invention, a user terminal acquires a facial image when it enters a physical store. First, the user terminal uses its internal camera to acquire a facial image of the user and extracts features for facial recognition. A facial recognition library such as OpenCV is used for this process. The extracted feature data is compared with pre-registered data, and the result is sent to the server. If the comparison is successful, the server authenticates the user and notifies the store's corresponding processing system.

[0548] Furthermore, the user terminal uses an emotion analysis engine, such as the Microsoft Azure Emotion API, to analyze the user's emotional state in real time based on the acquired facial image. This emotional data is sent to a server for recording and management. If the emotional data exceeds a certain threshold, for example, if the system determines that the user is experiencing stress, the response processing system sends an alert to store staff, instructing them to provide special assistance to the user.

[0549] As a concrete example, a smartphone application can simultaneously perform facial recognition and sentiment analysis when a user enters a store, enabling it to provide services tailored to the user's state. This can improve the customer experience and prevent potential problems.

[0550] An example of a prompt to input into a generative AI model is: "Please propose an application that combines customer facial recognition and sentiment analysis to provide a personalized shopping experience. Please also provide details on the specific hardware, software, and data processing methods."

[0551] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0552] Step 1:

[0553] The user terminal uses a camera to acquire an image of the user's face. At this stage, the input is the user's face as seen through the camera, and the output is digitized facial image data.

[0554] Step 2:

[0555] The device uses a face recognition library (such as OpenCV) to extract necessary features from the acquired face image. The input to this process is face image data, and the output is extracted feature data. The feature data is identified from the face image using a proprietary algorithm.

[0556] Step 3:

[0557] The extracted feature data is sent to the server and compared with a pre-registered database. The input for this step is the feature data and database information, and the output is the matching result. The server calculates the match rate and determines whether authentication is successful or not.

[0558] Step 4:

[0559] The server determines access rights based on the matching result. The input is the matching result, and the output is the access permission or denial status. The access control system uses this result to manage access to physical gates and networks.

[0560] Step 5:

[0561] The user's device sends the facial image to an emotion analysis engine (such as the Microsoft Azure Emotion API) to evaluate the emotion. The input is facial image data, and the output is data indicating the emotional state. Emotions can be determined based on changes in facial expression and facial features.

[0562] Step 6:

[0563] The server records the analyzed sentiment data, and if it exceeds a certain threshold, the response system is activated. The input is sentiment data, and the output is recording to the management database and sending an alert notification. The response system sends a notification to the store staff in the event of an anomaly and instructs them on the necessary actions.

[0564] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0565] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0566] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0567] [Fourth Embodiment]

[0568] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0569] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0570] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0571] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0572] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0573] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0574] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0575] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0576] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0577] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0578] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0579] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0580] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0581] This invention provides a facial recognition system for streamlining entry and exit management at various facilities. This system acquires a user's facial image using an image acquisition means and generates feature data from that image using a feature extraction means. The generated feature data is transmitted to a server and compared with data in a pre-registered database. The server's matching means determines whether the feature data matches the registered information. If a match is found, the server sends an access control signal to the terminal, and access from the terminal is authorized.

[0582] As a concrete example, when a user enters a specific building, a camera on a terminal installed at the entrance captures the user's face. The terminal temporarily stores this face image, performs feature extraction, and then sends the feature data to a server. The server immediately compares the features with a database, and if a match is found, it returns an authentication success signal to the terminal. Upon receiving this signal, the terminal unlocks the door, and the user can enter the facility.

[0583] The server records all authentication logs, and if authentication fails or an unregistered face is recognized, an alert is sent to the administrator via a recording alarm system. This system allows users to enter facilities quickly and securely without carrying a physical card. In addition, entry and exit records for each facility are centrally managed, leading to improved security.

[0584] The following describes the processing flow.

[0585] Step 1:

[0586] The device uses its camera to capture an image of the user's face when they stand in front of the entrance gate. It captures images not only from the center of the face, but also from several angles to improve recognition accuracy.

[0587] Step 2:

[0588] The device extracts features from the acquired facial image. Using a feature extraction algorithm, it identifies facial feature points and formats them as digital data. The feature data is immediately sent to the server.

[0589] Step 3:

[0590] The server receives feature data sent from the terminal. Based on the received data, it compares and matches it with features in a pre-registered employee database. If the matching result is a match, it is processed as a "success"; otherwise, it is processed as a "failure".

[0591] Step 4:

[0592] The server sends the verification result back to the terminal. If authentication is successful, it sends a signal to grant access; if it fails, it sends a signal to deny access.

[0593] Step 5:

[0594] Upon receiving a success signal, the terminal unlocks the physical gates and doors. The user is then granted access and can enter the facility.

[0595] Step 6:

[0596] The server logs all authentication events. If an unauthorized authentication or anomaly occurs, the logging alarm system immediately notifies the administrator.

[0597] (Example 1)

[0598] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0599] Current access control systems require physical authentication methods such as cards, which presents security risks and management challenges due to card loss or forgery. Furthermore, when using facial recognition, an efficient and secure authentication mechanism is required. Additionally, monitoring and prompt response to abnormal access are crucial from a security perspective, but current systems fail to adequately achieve these.

[0600] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0601] In this invention, the server includes means for acquiring an image of a person using an image acquisition device, means for extracting characteristic information from the acquired image of the person, and means for comparing the extracted characteristic information with information stored in advance. This allows users to efficiently and securely manage their access by facial recognition without using physical cards. Furthermore, a warning is issued in the event of an anomaly, enabling a rapid security response.

[0602] An "image acquisition device" is a device used to acquire images of people, and is a photographic device that includes cameras and sensors.

[0603] "Characteristic information" refers to numerical data, such as facial feature points, extracted from an acquired image of a person.

[0604] "Comparison" is a process that compares extracted characteristic information with previously stored information and analyzes the degree of similarity.

[0605] A "digital information management system" is an information system for organizing, storing, and managing electronic data.

[0606] "Authentication result" refers to information indicating the success or failure of authentication, based on the results of the comparison process.

[0607] A "communication network" is a network used for sending and receiving data, and includes infrastructure such as the internet and local networks.

[0608] This invention is a system for managing entry and exit using facial recognition of people, and consists of an image acquisition device, characteristic information extraction means, comparison means, and the like.

[0609] First, the user faces the image acquisition device at the entrance. The image acquisition device uses a camera to acquire high-resolution images. The hardware can use a standard high-performance camera. This ensures that accurate images are obtained regardless of gender, age, or other environmental conditions.

[0610] The acquired images are processed by the terminal. The terminal, as software, uses open-source image processing libraries, such as OpenCV, to extract characteristic information from the images. Here, facial feature points and shapes are analyzed and converted into numerical data. This data serves as a template for human identification.

[0611] The server receives characteristic information sent from the terminal and compares it with the stored database. The database contains pre-registered user characteristic information, and the server performs the matching process using this information. Python and Java are used as development languages ​​for the matching process, enabling rapid processing.

[0612] One concrete example is implementing this system at the entrance of a library to allow students to enter smoothly. In this case, students would only need to face the camera for authentication and would not need to carry a physical card. Furthermore, if an attempt at unauthorized access is made, the server would immediately detect it and send an alert to the administrator.

[0613] An example of a prompt to the generated AI model would be: "Please tell me how to streamline the facial recognition process in the access control system. Also, please explain in detail what steps should be taken in the event of an emergency."

[0614] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0615] Step 1:

[0616] The device acquires the user's face image using an image acquisition device. The input consists of physical environmental information and the user's face itself. The camera captures this environmental information as video data and generates a high-resolution face image. Specifically, the camera is activated on the device, and a picture is automatically taken when the user faces the camera.

[0617] Step 2:

[0618] The device extracts characteristic information using the acquired facial image. The input is the facial image acquired in step 1, and the output is characteristic information that quantifies the facial features. Image processing software (e.g., OpenCV) is used to extract feature points such as the position and distance of the eyes, nose, and mouth. Specifically, the image data is analyzed in real time, and its characteristics are extracted.

[0619] Step 3:

[0620] The terminal sends the extracted characteristic information to the server. The input is the characteristic information from step 2, and the output is the status indicating that the transmission to the server is complete. A secure protocol (e.g., HTTPS) is used for communication. Specifically, the data is converted into packets on the terminal side and sent to the server via the network.

[0621] Step 4:

[0622] The server compares the received characteristic information with data stored in a pre-stored database. The input consists of the characteristic information from step 3 and existing data in the database. The server analyzes these using a matching algorithm and determines the degree of match. Specifically, a database query is executed, and a high-speed comparison process is performed.

[0623] Step 5:

[0624] The server sends an authentication result to the terminal based on the comparison result. The input is the comparison result from step 4, and the output is an access control signal. If there is a match, an access approval signal is sent to the terminal; otherwise, a rejection signal is sent. Specifically, a signal is sent from the server to the terminal, and the terminal performs an action according to the received signal.

[0625] Step 6:

[0626] The terminal controls the physical door based on the authentication result from the server. The input is the authentication result from step 5, and the output is the opening and closing of the door. If access is approved, the terminal issues a door unlock command; if denied, it sounds an access denied sound. Specifically, the control circuit inside the terminal activates, and the door lock system physically operates.

[0627] (Application Example 1)

[0628] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0629] In modern society, while there is a demand for improved security in facilities and residences, the challenge lies in realizing access control systems that enhance authentication accuracy without compromising convenience. Furthermore, there is a lack of mechanisms for quickly and remotely verifying authentication results. Additionally, technologies that utilize facial recognition to safely and smoothly control physical barriers are also necessary.

[0630] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0631] In this invention, the server includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the acquired facial images, a matching means for comparing the extracted facial features with pre-registered data, and a means for linking the access control system with a physical unlocking device. This enables rapid and accurate access control, and by notifying another terminal of the authentication result using a remote notification means, external security monitoring becomes possible. Furthermore, by controlling the physical barrier based on the authentication result, smooth access is possible while maintaining high security.

[0632] "Image acquisition means" refers to a device or system that has the function of acquiring facial images using a camera or sensor.

[0633] "Feature extraction means" refers to a technical method or device for extracting specific feature data from acquired facial images.

[0634] A "matching device" is a device or system responsible for the process of comparing extracted facial feature data with information in a pre-registered database to determine whether or not there is a match.

[0635] "Access control means" refers to a device or system that has the function of managing permissions within a system or access to specific areas based on verification results.

[0636] A "recording and alarming device" is a device or system that records the results of authentication and has the function of reporting or issuing an alarm in the event of an abnormal situation or when an unregistered face is recognized.

[0637] A "remote notification means" is a device or system that has a mechanism for notifying another terminal of the authentication result via the internet or other communication means.

[0638] An "access control system" is a device or system that has a set of functions to manage people's entry into and exit from a facility or area and to control access based on authentication.

[0639] A "physical unlocking device" is a device or system that has the function of unlocking a physical lock based on authentication and opening and closing barriers or gates.

[0640] The system used to implement this application example employs facial recognition technology for access control and utilizes the following hardware and software.

[0641] The server is equipped with a camera device (e.g., a general-purpose camera or sensor) for acquiring facial images, and uses an image processing library (e.g., OpenCV) to extract facial features from the acquired images.

[0642] The extracted feature data is matched against pre-registered data on a cloud-based database service (e.g., AWS DynamoDB) via the network. This matching is performed using a face recognition algorithm (e.g., a deep learning model using Dlib).

[0643] Based on the verification results, instructions are sent to an access control terminal via the network. This works in conjunction with a physical unlocking device (e.g., a smart lock system) to control the opening and closing of doors and gates. If an abnormality occurs during verification, a security notification is issued using a recording alarm system.

[0644] Furthermore, as a means of remote notification, a real-time notification service (e.g., Firebase Notifications) is used to immediately send authentication results to the user's or administrator's device. This enables rapid response from external sources.

[0645] As a concrete example, when a user approaches their home, the installed camera recognizes their face, notifies their smartphone of the result, and automatically unlocks the front door. This feature allows the user to enter smoothly without having to take out their key. Furthermore, if unexpected authentication occurs, an immediate warning is issued, allowing for corrective action.

[0646] An example of a prompt message is, "Write a process to identify users requesting access to a specific area using facial recognition and automate their entry and exit."

[0647] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0648] Step 1:

[0649] The device acquires a facial image from the camera. The input here is real-time image data from the camera, and the output is a facial image for subsequent processing. The device stores this image in temporary memory.

[0650] Step 2:

[0651] The server receives the face image and performs feature extraction using a face recognition algorithm. The input here is the face image obtained in step 1, and the output is extracted data containing facial feature points. In this process, OpenCV is used to quantify features such as the positions of the eyes and nose from the image.

[0652] Step 3:

[0653] The server sends the extracted feature data to a cloud database and compares it with pre-registered user data. The input is feature data, and the output is the authentication match or mismatch result. A cloud service is used to perform the matching operation with the feature data in the database.

[0654] Step 4:

[0655] The server generates an access instruction to the physical unlocking device based on the verification result. The input here is the verification result, and the output is an unlock or restriction instruction signal. Specifically, upon successful authentication, a smart lock control signal is sent.

[0656] Step 5:

[0657] The server sends the authentication result to the user or administrator's terminal using remote notification methods. The input is the matching result and notification content, and the output is notification data via a real-time notification service. Firebase Notifications are used in this process.

[0658] Step 6:

[0659] When an anomaly is detected, the server activates a recording and alarm system, logs the event, and issues an alarm. Inputs are an anomaly detection flag and the target image, while outputs are the alarm and recorded log. This enables monitoring and tracking of unauthorized access.

[0660] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0661] This invention combines a facial recognition access control system with an emotion engine to recognize the user's emotional state and process accordingly. The system acquires a user's facial image using an image acquisition means and generates facial features from the image using a feature extraction means. This information is transmitted to a server and compared with pre-registered data by a matching means. If the matching is successful, an access control means grants the user access.

[0662] Furthermore, it incorporates an emotion engine that analyzes the user's facial expressions to evaluate their emotions. Emotion analysis is performed in real time, identifying various emotional states, such as whether the user is stressed or happy. This information is transmitted to a server for recording and management. If the emotion data exceeds a certain range or is deemed abnormal, a recording alarm system will issue an alarm and notify the administrator.

[0663] As a concrete example, when a user enters a building, the terminal's camera captures their face. The terminal extracts facial features, analyzes emotions from facial expressions, and sends the data to a server. The server verifies the facial data, grants access, stores the emotion data, and sends an alert to the administrator if any anomalies are detected. This system goes beyond simple entry and exit management, enabling security and environmental adjustments based on user emotions.

[0664] The following describes the processing flow.

[0665] Step 1:

[0666] The device uses a camera to photograph the face of a user attempting to enter the building. Images are acquired from multiple angles to collect sufficient data to improve recognition accuracy.

[0667] Step 2:

[0668] The device extracts features from the acquired facial image. It identifies facial feature points and formats them as digital data. Furthermore, it uses an emotion engine to analyze the user's facial expressions and generate emotion data.

[0669] Step 3:

[0670] The device sends feature data and sentiment data to the server. The data is designed to be processed by the server immediately.

[0671] Step 4:

[0672] The server compares the feature data received from the terminal with pre-registered information in the database. It determines whether the information matches the registered information and decides whether access is permitted.

[0673] Step 5:

[0674] Once the server completes the facial data matching, it sends the authentication result back to the device. If access is permitted, the server sends an authentication success signal.

[0675] Step 6:

[0676] Upon receiving a signal indicating successful authentication, the terminal unlocks the gate or door. The user can then enter the facility.

[0677] Step 7:

[0678] The server records emotional data, and if an anomaly detection system detects changes or abnormalities in emotions, it uses recording alarm mechanisms to alert the administrator. In this way, the user's psychological state is continuously monitored, ensuring safety.

[0679] (Example 2)

[0680] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0681] Conventional access control systems use facial recognition technology for access control, but they have the drawback of not being able to take into account the user's emotional state, making it difficult to respond based on emotions or to detect abnormal conditions early. Furthermore, introducing emotion recognition requires enhanced security and environmental adjustments.

[0682] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0683] In this invention, the server includes an image acquisition means for acquiring facial images, an emotion analysis means for recognizing emotional states, and a recording alarm means for recording and notifying of abnormalities when the emotional state exceeds a specific range. This enables flexible access control and rapid abnormality notification based on the user's emotional state.

[0684] "Image acquisition means" refers to devices and technologies for capturing and acquiring images of a user's face.

[0685] "Feature extraction means" refers to software or technology used to identify specific feature points from acquired facial images and extract them as data.

[0686] "Matching means" refers to a technology or process that compares extracted facial features with pre-registered data to determine whether they match or not.

[0687] "Access control means" refers to systems or technologies for managing the opening and closing of physical or digital entrances and exits based on matching results.

[0688] "Emotional analysis methods" refer to technologies and processes that analyze and recognize various emotional states from a user's facial expressions.

[0689] "Recording and alarming means" refers to technologies and processes that record recognition results such as emotional states and issue alarms to administrators and related systems when an anomaly is detected.

[0690] An "electronic data processing device" refers to a digital system or device that comprehensively utilizes the above-mentioned methods to perform facial recognition and emotion analysis.

[0691] This invention is an advanced access control system that combines facial recognition and emotion analysis. It recognizes the user's face to control access, while simultaneously monitoring, recording, and managing the user's emotional state.

[0692] Specifically, the device uses a high-resolution camera to acquire an image of the user's face. The camera automatically adjusts the exposure and focus to obtain a clear image. First, facial features are extracted from this image using an image processing library such as OpenCV. In this feature extraction, feature points such as the eyes, nose, and mouth are converted into data.

[0693] The extracted feature data is sent from the device to the server. The server uses cloud-based facial recognition technology to compare it with pre-registered data. AWS Rekognition and similar technologies are used for this process, and authentication is considered successful if a match is found. Based on this authentication result, the server sends a signal to control the physical door or gate, granting the user access.

[0694] Furthermore, the device is equipped with an emotion analysis module that uses technologies such as the Azure Face API to analyze the user's emotions from their facial expressions in real time. This emotion data quantifies whether the user is feeling joy, anger, or stress, and sends it to the server.

[0695] The server records this emotional data in a database and notifies the administrator if there are emotional changes that exceed the set criteria. This notification is sent via email or system alert, allowing for a quick response.

[0696] As a concrete example, when a user visits an office building, the device takes a picture of their face and analyzes their emotions in real time. Based on this information, the device can appropriately control the user's access and adjust the environment based on their emotions (for example, by adjusting the room temperature or changing the music).

[0697] The generative AI model uses prompts such as the following:

[0698] "Please analyze the emotions expressed by the person in this photograph."

[0699] "Please output the real-time sentiment analysis results from the building's entry system."

[0700] This invention enables more flexible and sophisticated management that goes beyond simple entry and exit control, and is based on user emotions.

[0701] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0702] Step 1:

[0703] The device acquires the user's facial image. The device is installed at the facility's entrance, and when a user stands in front of the camera, the high-resolution camera activates and captures a facial image. At this time, appropriate exposure and focus are automatically adjusted. The input is the facial image captured by the camera, and the output is high-resolution digital image data.

[0704] Step 2:

[0705] The device extracts features from the acquired facial image. Specifically, it uses the OpenCV library to identify feature points such as the positions of the eyes, nose, and mouth, and extracts them as numerical data. The input is facial image data, and the output is facial feature data.

[0706] Step 3:

[0707] The terminal sends the extracted feature data to the server. The feature data is transferred to the server via a secure protocol. The input is digitized feature data, and the output is the digital data transferred to the server.

[0708] Step 4:

[0709] The server uses the received feature data to compare it with registered data. It utilizes a facial recognition system and cloud-based recognition technology to compare it with pre-registered data. The input is the transmitted feature data, and the output is the result of successful or unsuccessful authentication.

[0710] Step 5:

[0711] If authentication is successful, the server sends an access control signal to the terminal. This is how the physical gate or door opens. The input is the result of successful authentication, and the output is the activation signal of the access control device.

[0712] Step 6:

[0713] The device performs emotion analysis using facial images. It utilizes the Azure Face API to analyze emotions such as joy, anger, and stress in real time. The input is a real-time facial image, and the output is numerical data indicating the emotional state.

[0714] Step 7:

[0715] The server receives emotional data and determines anomalies based on predefined criteria. If the emotional state exceeds the criteria, the server issues an alarm and notifies the administrator. The input is emotional state data, and the output is the issuance of an alarm or the execution of a notification.

[0716] Step 8:

[0717] The server analyzes the accumulated data and uses it to improve the system. Generative AI models are used to improve recognition accuracy and add new features. The input is the accumulated data set, and the output is the specifications of the improved algorithms and new features.

[0718] (Application Example 2)

[0719] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0720] Current facial recognition systems primarily manage user entry and exit, with few systems that understand and utilize users' emotional states. Therefore, it is difficult to respond to visitors' and customers' emotions, limiting the potential for improving satisfaction and preventing potential problems. Consequently, there is a growing need for technology that can identify user emotions in real time and respond appropriately based on that information.

[0721] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0722] In this invention, the server includes an image acquisition means for acquiring facial images, a feature extraction means for extracting features from the acquired facial images, a matching means for comparing the extracted facial features with pre-registered data, an access control means for performing access control based on the matching results, a recording and alarm means for recording the matching results and issuing an alarm in case of an anomaly, an emotion analysis means for analyzing the extracted emotion data and recording the information, and a means for performing corresponding processing based on the analyzed emotion data. This enables flexible responses that reflect the user's emotional state.

[0723] "Image acquisition means" refers to the device or process for acquiring a user's facial image.

[0724] A "feature extraction method" refers to an algorithm or device for analyzing and identifying specific patterns or shapes from acquired facial images.

[0725] A "matching means" is a function or process that compares extracted features with pre-registered data and evaluates the degree of agreement.

[0726] "Access control means" refers to a system for granting or denying a user physical or digital access based on the matching results.

[0727] A "recording and alarming means" is a device or program that has the function of recording information from verification results and abnormal emotions, and issuing an alarm as needed.

[0728] "Emotional analysis methods" refer to technologies and systems that detect a user's emotional state from their facial expressions and record and evaluate it as data.

[0729] "Means of handling response processing" refers to a function or process that implements specific countermeasures for the user based on the analyzed sentiment data.

[0730] In this invention, a user terminal acquires a facial image when it enters a physical store. First, the user terminal uses its internal camera to acquire a facial image of the user and extracts features for facial recognition. A facial recognition library such as OpenCV is used for this process. The extracted feature data is compared with pre-registered data, and the result is sent to the server. If the comparison is successful, the server authenticates the user and notifies the store's corresponding processing system.

[0731] Furthermore, the user terminal uses an emotion analysis engine, such as the Microsoft Azure Emotion API, to analyze the user's emotional state in real time based on the acquired facial image. This emotional data is sent to a server for recording and management. If the emotional data exceeds a certain threshold, for example, if the system determines that the user is experiencing stress, the response processing system sends an alert to store staff, instructing them to provide special assistance to the user.

[0732] As a concrete example, a smartphone application can simultaneously perform facial recognition and sentiment analysis when a user enters a store, enabling it to provide services tailored to the user's state. This can improve the customer experience and prevent potential problems.

[0733] An example of a prompt to input into a generative AI model is: "Please propose an application that combines customer facial recognition and sentiment analysis to provide a personalized shopping experience. Please also provide details on the specific hardware, software, and data processing methods."

[0734] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0735] Step 1:

[0736] The user terminal uses a camera to acquire an image of the user's face. At this stage, the input is the user's face as seen through the camera, and the output is digitized facial image data.

[0737] Step 2:

[0738] The device uses a face recognition library (such as OpenCV) to extract necessary features from the acquired face image. The input to this process is face image data, and the output is extracted feature data. The feature data is identified from the face image using a proprietary algorithm.

[0739] Step 3:

[0740] The extracted feature data is sent to the server and compared with a pre-registered database. The input for this step is the feature data and database information, and the output is the matching result. The server calculates the match rate and determines whether authentication is successful or not.

[0741] Step 4:

[0742] The server determines access rights based on the matching result. The input is the matching result, and the output is the access permission or denial status. The access control system uses this result to manage access to physical gates and networks.

[0743] Step 5:

[0744] The user's device sends the facial image to an emotion analysis engine (such as the Microsoft Azure Emotion API) to evaluate the emotion. The input is facial image data, and the output is data indicating the emotional state. Emotions can be determined based on changes in facial expression and facial features.

[0745] Step 6:

[0746] The server records the analyzed sentiment data, and if it exceeds a certain threshold, the response system is activated. The input is sentiment data, and the output is recording to the management database and sending an alert notification. The response system sends a notification to the store staff in the event of an anomaly and instructs them on the necessary actions.

[0747] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0748] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0749] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0750] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0751] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0752] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0753] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0754] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0755] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0756] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0757] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0758] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0759] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0760] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0761] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0762] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0763] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0764] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0765] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0766] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0767] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0768] The following is further disclosed regarding the embodiments described above.

[0769] (Claim 1)

[0770] Image acquisition method for obtaining facial images,

[0771] A feature extraction means for extracting features from acquired facial images,

[0772] A matching means that compares extracted facial features with pre-registered data,

[0773] Access control means that perform access control based on the matching result,

[0774] It is equipped with a recording and alarm means that records the verification results and issues an alarm in the event of an abnormality.

[0775] A system including an electronic data management system.

[0776] (Claim 2)

[0777] The system according to claim 1, further comprising means for controlling the opening and closing of physical gates or doors based on the authentication result of a matching means.

[0778] (Claim 3)

[0779] The system according to claim 1, comprising means for accessing and matching registered data via a network.

[0780] "Example 1"

[0781] (Claim 1)

[0782] A means for acquiring an image of a person using an image acquisition device,

[0783] A means for extracting characteristic information from acquired images of people,

[0784] A means of comparing extracted characteristic information with previously stored information,

[0785] A means of implementing entry and exit control based on the comparison results,

[0786] It has a means to record comparison results and issue warnings in case of abnormalities.

[0787] A system including a digital information management system.

[0788] (Claim 2)

[0789] The system according to claim 1, further comprising means for controlling the opening and closing of physical entrances and doors based on the authentication results of a comparison means.

[0790] (Claim 3)

[0791] The system according to claim 1, comprising means for accessing and comparing stored information via a communication network.

[0792] "Application Example 1"

[0793] (Claim 1)

[0794] Image acquisition method for obtaining facial images,

[0795] A feature extraction means for extracting features from acquired facial images,

[0796] A matching means that compares extracted facial features with pre-registered data,

[0797] Access control means that perform access control based on the matching result,

[0798] A recording and alarm means that records the verification results and issues an alarm in case of an abnormality,

[0799] A means of notifying another terminal of the authentication result using a remote notification means,

[0800] A system that includes means for linking an access control system with a physical unlocking device.

[0801] (Claim 2)

[0802] The system according to claim 1, further comprising means for controlling the opening and closing of a physical barrier based on the authentication result of a matching means.

[0803] (Claim 3)

[0804] The system according to claim 1, comprising means for accessing and verifying registered data via a communication network.

[0805] "Example 2 of combining an emotion engine"

[0806] (Claim 1)

[0807] Image acquisition method for obtaining facial images,

[0808] A feature extraction means for extracting features from acquired facial images,

[0809] A matching means that compares extracted facial features with pre-registered data,

[0810] Access control means that perform access control based on the matching result,

[0811] An emotion analysis method that analyzes the user's facial expressions to recognize their emotional state,

[0812] It is equipped with a recording alarm means that records when an emotional state exceeds a certain range and notifies of the abnormality.

[0813] A system including an electronic data processing device.

[0814] (Claim 2)

[0815] The system according to claim 1, further comprising means for controlling the opening and closing of physical gates and doors based on the authentication result of a matching means and emotional state.

[0816] (Claim 3)

[0817] The system according to claim 1, comprising means for accessing, matching, and performing sentiment analysis of registered data via a network.

[0818] "Application example 2 when combining with an emotional engine"

[0819] (Claim 1)

[0820] Image acquisition method for obtaining facial images,

[0821] A feature extraction means for extracting features from acquired facial images,

[0822] A matching means that compares extracted facial features with pre-registered data,

[0823] Access control means that perform access control based on the matching result,

[0824] A recording and alarm means that records the verification results and issues an alarm in case of an abnormality,

[0825] A sentiment analysis tool that analyzes extracted sentiment data and records the information,

[0826] A means of processing responses based on analyzed sentiment data,

[0827] A system that includes this.

[0828] (Claim 2)

[0829] The system according to claim 1, further comprising means for controlling the opening and closing of physical gates or doors based on the authentication result of a matching means.

[0830] (Claim 3)

[0831] The system according to claim 1, comprising means for accessing and matching registered data via a network. [Explanation of symbols]

[0832] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Image acquisition method for obtaining facial images, A feature extraction means for extracting features from acquired facial images, A matching means that compares extracted facial features with pre-registered data, Access control means that perform access control based on the matching result, A recording and alarm means that records the verification results and issues an alarm in case of an abnormality, A means of notifying another terminal of the authentication result using a remote notification means, A system that includes means for linking an access control system with a physical unlocking device.

2. The system according to claim 1, further comprising means for controlling the opening and closing of a physical barrier based on the authentication result of a matching means.

3. The system according to claim 1, further comprising means for accessing and verifying registered data via a communication network.

Images