Method and apparatus for identifying actual users in a single-camera-based kiosk environment

Abstract

A method and apparatus for identifying actual users are disclosed, which can more accurately identify actual users by distinguishing between environments where multiple users are gathered, or users with various conditions such as children, the elderly, and wheelchair users. Specifically, a method and apparatus for identifying actual users in a single-camera-based kiosk environment are disclosed, which can stably and accurately identify actual users interacting with the kiosk even when multiple people are included in the image acquired through a single camera in a kiosk environment, improve the reliability of user recognition by effectively removing unnecessary face detection results, and further enable appropriate user selection even in barrier-free environments that consider users with various physical conditions such as wheelchair users, children, and the elderly, thereby improving the accuracy and responsiveness of personalized services such as interface height adjustment and eye tracking.

Description

Technology Field

[0001] The present invention relates to a technology for identifying actual kiosk users in a kiosk environment equipped with a single camera, and more specifically, to a method and apparatus for identifying actual users that can more accurately identify actual users by distinguishing users under various conditions, such as in an environment where multiple users are gathered, or users such as children, the elderly, and wheelchair users. Background Technology

[0002] With the widespread adoption of unmanned ordering systems, kiosks are being widely utilized in various environments, including public facilities, hospitals, restaurants, and transportation facilities, and the importance of interface automation technology to enhance user convenience is steadily increasing. In particular, intelligent kiosk technology is gaining attention for utilizing camera-based image recognition to analyze the user's location, gaze, and physical condition, and accordingly performs functions such as adjusting screen height, modifying UI configurations, and providing accessibility features.

[0003] Meanwhile, the need for barrier-free kiosks that accommodate users with diverse physical conditions, such as the elderly, children, and wheelchair users, is emerging. Accordingly, kiosks are evolving beyond simple input devices to automatically recognize user characteristics and provide adaptive services. However, for these personalized services to be smoothly implemented, technology capable of accurately identifying the actual user utilizing the kiosk within the camera footage must be developed first.

[0004] Conventional kiosk systems primarily rely on touch input or infrared-based proximity sensors to detect user approach; however, this approach has limitations in that it cannot distinguish between actual users and those engaging in real interaction among multiple individuals captured in camera footage. In particular, when multiple people are present around a kiosk, simple distance-based or event-based detection methods struggle to clearly differentiate between actual users and mere passersby, leading to a high likelihood of malfunctions.

[0005] Furthermore, in barrier-free environments, wheelchair users and pedestrians are captured by cameras at different heights and positions, making it difficult to accurately process various user types simultaneously using face detection or user recognition methods based on a single threshold. Moreover, there is a problem where the reliability of the entire system is compromised due to the lack of technology to accurately identify the actual user prior to acquiring information such as user location and physical condition, which is necessary for automatically adjusting the interface.

[0006] Furthermore, conventional face detection technology suffers from the problem of failing to sufficiently eliminate various noise-related detection results occurring in consecutive video frames. In other words, there is a possibility of mistaking objects unrelated to actual kiosk usage—such as temporarily detected faces, distant figures, or profile or non-frontal faces—for actual users. Additionally, in situations where multiple people are lined up in front of a kiosk, the ability to distinguish between the nearest actual user and waiting individuals is inadequate. For these reasons, the accuracy of subsequent processing steps, such as eye tracking and interface adjustment, is degraded.

[0007] Therefore, there is a need to develop preprocessing technology capable of reliably and accurately identifying the actual user among multiple individuals based on camera footage in a kiosk environment. In particular, there is a need for real user identification technology that operates reliably even in barrier-free environments considering various user conditions and can improve the accuracy of subsequent personalized services. The problem to be solved

[0008] According to one embodiment, a method and apparatus for identifying actual users in a single-camera-based kiosk environment are proposed, which can stably and accurately identify actual users interacting with the kiosk even when multiple people are included in the image acquired through a single camera in a kiosk environment, improve the reliability of user recognition by effectively removing unnecessary face detection results, and further enable appropriate user selection even in a barrier-free environment that considers users with various physical conditions such as wheelchair users, children, and the elderly, thereby improving the accuracy and responsiveness of personalized services such as interface height adjustment and eye tracking. means of solving the problem

[0009] A method for identifying an actual user of a kiosk among a plurality of persons included in a camera image in a kiosk environment equipped with a single camera according to one embodiment comprises: a step of activating a trigger by a proximity sensor or touch input provided in the kiosk; a step of detecting one or more face regions and landmark information corresponding to said face regions using a face detection model in an image frame input from the camera after the trigger is activated; a step of performing zone filtering to select only valid faces by dividing the image frame into a plurality of regions in a vertical direction and applying different face size thresholds to each region; and a step of performing inter-frame Intersection over Union (IoU) based tracking on faces that have passed the zone filtering to assign a unique identifier to each face and generating and updating tracking information including a face image buffer, a landmark buffer, and an IoU buffer. The method comprises: a step of classifying each face into an unverified state, a waiting state, or a verified state by performing face stability verification for the aforementioned tracked face, including stability verification using an average IoU value for a preset number of frames, a change amount verification determining whether the amount of change in face area between consecutive frames converges to below a threshold value, and an evaluation using a pose estimation score based on a frontal view calculated from face landmarks; and, if there is one or more faces classified as verified state, a step of calculating a depth value for each face using a monocular depth estimation model and selecting the face closest to the camera based on the depth value, or, if there is only one face in the verified state, determining the actual user based on the pose estimation score of the corresponding face.

[0010] Here, the step of performing zone filtering divides the image frame into an upper region, a middle region, and a lower region, and applies different face area thresholds to each region to remove faces included in distant figures or backgrounds.

[0011] In addition, the step of generating and updating tracking information may assign a new identifier and initialize the tracker if it does not match an existing face based on the IoU-based matching result during tracking, and update the tracking information of the corresponding identifier if it matches.

[0012] In addition, when performing face stability verification, if the average IoU value is 0.9 or higher for a preset number of frames, it can be determined as a stable face.

[0013] In addition, when performing face stability verification, the change in the area of ​​the face region in consecutive frames is analyzed as a time series, and if the state in which the amount of change in area is below a preset threshold persists for a certain number of times or more, the face can be determined to be stabilized.

[0014] In the actual user confirmation step, if there are multiple faces in the verified state, a monocular depth estimation model is used to calculate the depth value of each face, and the face with the largest depth value can be selected as the actual user.

[0015] A device for identifying an actual user using a kiosk among a plurality of persons included in a camera image in a kiosk environment equipped with a single camera according to another embodiment of the present invention comprises: a trigger unit activated by a proximity sensor or touch input; a face detection unit that detects one or more face regions and landmark information corresponding to the face regions using a face detection model in an image frame input from the camera after the trigger is activated; a zone filtering unit that divides the image frame into a plurality of regions in a vertical direction and selects only valid faces by applying different face size thresholds to each region; and a face tracking unit that performs frame-to-frame Intersection over Union (IoU) based tracking on faces that have passed the zone filtering, assigns a unique identifier to each face, and generates and updates tracking information including a face image buffer, a landmark buffer, and an IoU buffer. A face stability verification unit that classifies each face into an unverified state, a waiting state, or a verified state by performing face stability verification for the aforementioned tracked face, including stability verification using an average IoU value for a preset number of frames, a change amount verification determining whether the amount of change in the face region area between consecutive frames converges to below a threshold value, and an evaluation using a pose estimation score based on a frontal view calculated from face landmarks; and a real user selection unit that, if there is one or more faces classified as verified state, calculates a depth value for each face using a monocular depth estimation model and selects the face closest to the camera based on the depth value, or, if there is only one face in the verified state, determines the real user based on the pose estimation score of the corresponding face.

[0016] Here, the zone filtering unit can separate the image frame into an upper region, a middle region, and a lower region, and apply different face area thresholds to each region to remove distant figures or faces included in the background.

[0017] In addition, the face tracking unit can assign a new identifier and initialize the tracker if it does not match an existing face based on the IoU-based matching result, and update the tracking information of the corresponding identifier if it matches.

[0018] In addition, the face stability verification unit determines that a face is stable if the average IoU value for a preset number of frames is greater than or equal to a reference value, and determines that the face is stabilized if the change in the area of ​​the face region between consecutive frames is below a preset threshold value for a certain number of consecutive times.

[0019] And the actual user selection unit can calculate the depth value of each face using a monocular depth estimation model when there are multiple faces in a verified state, and select the face with the largest depth value as the actual user. Effects of the invention

[0020] According to one embodiment, since it is possible to accurately identify an actual user based on camera images in a kiosk environment and analyze various information such as the person's location, age, and gender, the reliability of automatic interface adjustments, such as screen height adjustment or font size change, is improved, and the accuracy of personalized services required in barrier-free kiosks is significantly improved.

[0021] In addition, as the actual user is reliably identified in advance, eye tracking of that individual becomes possible, and the precise analysis of which area of ​​the screen the user is focusing on lays the foundation for guaranteeing the accuracy of subsequent technologies, such as eye-based interface control.

[0022] Furthermore, by utilizing DPT-Large-based monocular depth estimation, it is possible to identify the actual user among multiple individuals without a separate depth sensor, which simplifies system configuration, reduces hardware costs, and enables the implementation of a lightweight kiosk system that can be flexibly applied to various environments.

[0023] Furthermore, by applying zone filtering techniques that utilize face size thresholds for each region, users with different physical conditions—such as wheelchair users, children, and the elderly—can be reliably processed using a single algorithm, thereby improving accessibility.

[0024] Finally, the depth-based queuing mechanism allows for the accurate identification of the nearest actual user and clear distinction from waiting users, even when multiple users are lined up in front of a kiosk, thereby improving the user recognition stability and practicality of the entire system. Brief explanation of the drawing

[0025] FIG. 1 is a diagram illustrating a situation in which users under various conditions exist in a kiosk environment equipped with a single camera according to an embodiment of the present invention. FIG. 2 is a flowchart of a method for identifying actual users in a single-camera-based kiosk environment according to an embodiment of the present invention. FIG. 3 is a diagram illustrating zone filtering according to an embodiment of the present invention, FIG. 4 is a drawing illustrating a face state transition diagram according to an embodiment of the present invention, Figure 5 is a diagram illustrating an example of calculating a pose estimation score in face stability verification, FIG. 6 is a configuration diagram of a user-specific device in a single-camera-based kiosk environment according to one embodiment of the present invention. Specific details for implementing the invention

[0026] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0027] In describing the embodiments of the present invention, if it is determined that a detailed description of known functions or configurations may unnecessarily obscure the essence of the invention, such detailed description will be omitted. Furthermore, the terms described below are defined considering the functions in the embodiments of the present invention, and these may vary depending on the intentions or conventions of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0028] Combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions (execution engine), and since these computer program instructions may be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, the instructions executed through the processor of the computer or other programmable data processing device create a means to perform the functions described in each block of the block diagram or each step of the flowchart.

[0029] Since these computer program instructions may be stored in computer-available or computer-readable memory that can be directed toward a computer or other programmable data processing device to implement a function in a specific way, the instructions stored in said computer-available or computer-readable memory may also be used to produce a manufactured item containing instruction means that perform the function described in each block of a block diagram or each step of a flowchart.

[0030] And since computer program instructions can be loaded onto a computer or other programmable data processing device, instructions that perform a series of operation steps on a computer or other programmable data processing device to create a process executed by a computer and that execute the computer or other programmable data processing device can also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

[0031] Additionally, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing specific logical functions, and it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps described in succession may actually be performed substantially simultaneously, and the blocks or steps may also be performed in the reverse order of the corresponding functions as needed.

[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, the embodiments of the present invention exemplified below may be modified in various different forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art to which this invention pertains.

[0033] FIG. 1 is a diagram illustrating a situation in which users under various conditions exist in a kiosk environment equipped with a single camera according to an embodiment of the present invention.

[0034] Referring to FIG. 1, the kiosk (100) is configured to include a display unit and an input unit for receiving order input from a user, and a camera (110) for capturing the surrounding environment may be provided at the top of the kiosk (100). The camera (110) is configured to acquire images including a plurality of people located in front of the kiosk (100) in real time.

[0035] Additionally, multiple users with different physical conditions may be positioned in front of the kiosk (100), and as an example, a situation is illustrated in which an adult male (10), a child (20), and a user in a wheelchair (30) are present together. In this case, the adult male (10) may be an actual user who operates the kiosk (100), and the child (20) and the wheelchair user (30) may be individuals waiting to use the kiosk or simply located nearby.

[0036] As such, the image acquired by a single camera (110) may simultaneously include multiple individuals at different positions and heights, and in particular, a face may be detected at a height different from that of a normal pedestrian, such as a wheelchair user (30), or an individual with a relatively small face size, such as a child (20), may be included. Accordingly, a problem may arise in that it is difficult to accurately identify the actual user using the kiosk (100) based on simple face detection or distance-based judgment alone.

[0037] Accordingly, in this embodiment, by applying an algorithm to accurately identify the actual user among a plurality of people from an image obtained through a camera (110) in such an environment, the actual user can be reliably identified even in a situation where users of various physical conditions are mixed.

[0038] FIG. 2 is a flowchart of a method for identifying actual users in a single-camera-based kiosk environment according to one embodiment of the present invention.

[0039] Referring to FIG. 2, a step (210) in which a trigger is activated is performed first. The trigger can be activated by a proximity sensor provided in the kiosk or by a touch input by a user, and unnecessary consumption of computational resources can be prevented by not performing face detection and subsequent processing on the camera image before the trigger is activated.

[0040] Next, a step (220) of detecting one or more face regions and landmark information corresponding to the face regions in a video frame input from a camera is performed. At this time, landmark information including major feature points such as eyes, nose, and mouth, along with a bounding box indicating the location of the face, is extracted using a face detection model, and this is used as input data for a subsequent step.

[0041] Subsequently, a zone filtering step (230) is performed in which the video frame is divided into multiple regions in the vertical direction, and different face size thresholds are applied to each region to select only valid faces. In the zone filtering step, different minimum face area criteria are applied to the upper, middle, and lower regions of the frame, respectively, thereby effectively detecting users with various physical conditions, such as wheelchair users and children, while removing small faces included in distant figures or backgrounds. A more detailed explanation of the zone filtering will be provided later with reference to FIG. 3.

[0042] Next, a step (240) of performing inter-frame Intersection over Union (IoU) based tracking on faces that have passed zone filtering is performed. In this step, a unique identifier is assigned to each face, tracking is maintained by determining whether the same person exists between consecutive frames, and tracking information including a face image buffer, a landmark buffer, and an IoU buffer is generated and updated. Additionally, if a face that does not match an existing tracking target is detected, tracking is initialized by assigning a new identifier, and if it matches, the information of the corresponding identifier is updated.

[0043] Next, a step (250) of performing stability verification on the tracked face is performed. The stability verification may include an IoU-based verification that determines the movement stability of the face using an average IoU value for a preset number of frames, and a change amount verification that determines whether the user has stopped in front of the kiosk by analyzing the change in the area of ​​the face region between consecutive frames. Additionally, a face that is close to the front may be evaluated first using a pose estimation score based on the front view calculated from face landmarks. Based on these verification results, each face is classified into an unverified state, a waiting state, or a verified state, and a face in the waiting state may wait to transition to the verified state for a preset number of frames.

[0044] The step (250) of performing stability verification is described in more detail as follows. First, multi-stage stability verification is performed for each face being tracked to determine whether the face is a user actually using the kiosk, and based on the result, it is classified into an unverified state, a waiting state, or a verified state.

[0045] First, IoU buffer verification is performed. In IoU buffer verification, the average of IoU values ​​calculated over a pre-set number of frames, for example, 30 frames, is calculated, and the face is determined to be a stable face only if the average value is greater than or equal to a threshold value, for example, 0.9. This allows for the effective exclusion of individuals who are simply passing in front of the camera or individuals with significant movement.

[0046] Next, a change verification is performed. In this verification, the area of ​​the face region detected in consecutive frames is tracked in a time series to determine whether the change in area between frames gradually decreases and converges below a preset threshold. Generally, when a user approaches a kiosk, the face region gradually increases until it reaches a maximum value when stopping in front of the kiosk, after which almost no change occurs; therefore, stability can be determined using this change pattern. In particular, a more precise assessment of stability is possible by additionally filtering out cases where the area change persists, such as for users who are still moving, even if the IoU value remains high.

[0047] Additionally, a pose estimation score can be calculated. The pose estimation score is calculated using the ratio of horizontal and vertical distances extracted from facial landmarks, and has a higher value as the face faces forward. The pose estimation score can subsequently be used as a criterion to prioritize the selection of frames with high frontality during the final user confirmation stage.

[0048] Each face is classified into a state based on the results of IoU buffer verification and change amount verification. Specifically, faces that fail to pass either IoU buffer verification or change amount verification are classified as unverified; faces in the unverified state may wait to transition to the verified state or be excluded from processing until certain conditions are met. Meanwhile, if a face in the unverified state is determined to be closer to a face currently in the verified state, it may transition to a waiting state. In the waiting state, additional verification is performed for a preset number of frames, e.g., 30 frames, to determine whether to transition to the verified state. Finally, faces that pass both IoU buffer verification and change amount verification are classified as verified and are subsequently utilized as candidates in the actual user confirmation stage.

[0049] Finally, a step (260) for determining the actual user is performed. In this step, if there is one or more faces classified as having completed verification, a depth value for each face is calculated using a monocular depth estimation model, and the face closest to the camera can be selected as the actual user based on this depth value. Additionally, if there is only one face with completed verification, the final actual user can be determined by considering the pose estimation score of that face.

[0050] To explain the step (260) for determining actual users more specifically, first, there are faces being tracked, and the selection of final actual users is performed only if there is at least one face among them that is classified as having completed verification. At this time, actual users can be determined in different ways depending on the number of faces that have completed verification.

[0051] First, if there is only one verified face, that face is set as a candidate for the actual user. The actual user can then be finally identified by referring to the pose estimation score buffer accumulated for the face and selecting the latest frame that satisfies a pose score above a preset threshold. Through this process, the actual user can be determined based on a stable face image with high frontality, even in a single-user scenario.

[0052] On the other hand, if there are two or more verified faces, depth information can be additionally utilized to distinguish the actual user operating the kiosk among the multiple candidates. Specifically, a monocular depth estimation model, such as the DPT-Large model, can be used to calculate depth values ​​for each face region, and based on these depth values, the face closest to the camera—that is, the face with the highest depth value—can be selected as the actual user. In this case, the depth value refers to relative distance information from the camera and reflects the fact that the user actually operating the kiosk is generally located furthest forward.

[0053] In this way, by utilizing a monocular depth estimation model, the relative distance relationship between multiple individuals can be estimated using only a single RGB camera without the need for an additional separate depth camera, and based on this, the actual user can be identified more accurately. Therefore, through this embodiment, the accuracy of identifying the actual user in a multi-user environment can be improved while simplifying the hardware configuration.

[0054] Through this series of steps, actual users using the kiosk can be reliably identified even when multiple individuals are included in the video acquired from a single camera, and the accuracy of subsequent personalized services can be improved.

[0055] FIG. 3 is a diagram illustrating zone filtering according to an embodiment of the present invention.

[0056] Referring to FIG. 3, the input image frame can be divided into multiple regions along the vertical direction, and as an example, can be divided into a first region (310) located at the top, a second region (320) located in the middle, and a third region (330) located at the bottom. Each region can be set according to the proportion it occupies in the entire frame, for example, the first region (310) can be set to an area corresponding to about 60% of the height of the entire frame, the second region (320) can be set to about 25%, and the third region (330) can be set to an area corresponding to about 15%.

[0057] In this embodiment, zone filtering is performed to select valid faces by applying different face size thresholds to each region. Specifically, in the first region (310), a face can be determined to be valid only if the area of ​​the face region is greater than or equal to a preset first threshold relative to the entire frame, and as an example, the first threshold can be set to 0.055. In the second region (320), a face can be determined to be valid if the face area is greater than or equal to a second threshold, and as an example, the second threshold can be set to 0.03. Additionally, in the third region (330), a face can be determined to be valid if the face area is greater than or equal to a third threshold, and as an example, the third threshold can be set to 0.025.

[0058] The reason for applying different threshold values ​​to each area is to reflect the characteristic that users located in front of the kiosk are captured by the camera at different heights. In other words, while typical pedestrians are detected in the relatively upper or middle areas, users in lower positions, such as wheelchair users or children, are more likely to be detected in the lower area. Accordingly, the threshold value in the lower area can be set low so that even faces with relatively small facial areas can be recognized as valid faces.

[0059] Furthermore, since there is a high probability of detecting distant figures or small faces embedded in the background in the upper area, a relatively high threshold can be applied to effectively eliminate these unnecessary detection results. On the other hand, because the lower area contains individuals likely to be actual kiosk users, a more relaxed threshold can be applied to reliably detect users with various physical characteristics.

[0060] Thus, unlike simple single threshold-based filtering, the zone filtering technique according to one embodiment performs location-based adaptive filtering, thereby enabling the simultaneous consideration of various user types while removing distant persons and noisy detection results.

[0061] FIG. 4 is a diagram illustrating a face state transition diagram according to one embodiment of the present invention.

[0062] Referring to FIG. 4, the present invention is configured to classify each face detected and tracked in a camera image into a plurality of states according to the results of stability verification, and to ultimately select an actual user through transitions between each state. Specifically, the face can be classified into an unverified state (410), a standby state (420), and a verified state (430).

[0063] First, the unverified state (410) refers to an unstable face that has not passed IoU buffer verification or change amount verification. That is, if a face is not maintained stably for a certain frame or if there is a continuous change in the area of ​​the face region, it is classified as an unverified state (410) because it is highly likely that the face is not an actual user. Faces in this unverified state may wait to transition to a verified state until stability conditions are satisfied, or may be excluded from processing under certain conditions.

[0064] Next, the waiting state (420) is a state that is set when a face in the unverified state is determined to be closer to the camera than a face in the currently verified state. That is, if there is a face with a higher Depth value according to the monocular depth estimation result, that face can be considered a potential real user, so it is transitioned to the waiting state (420) to perform additional verification for a certain period of time. At this time, the waiting state can be maintained for a preset number of frames, for example, 30 frames, and if both IoU buffer verification and change amount verification are satisfied during this period, it can be switched to the verified state. Through this Depth Wait mechanism, a new user approaching a location closer than the existing real user can be reliably identified.

[0065] Finally, the verified state (430) refers to a stable face that has passed both IoU buffer verification and change amount verification, and the face is selected as a candidate for actual user in a subsequent stage. Since the face in the verified state is judged to have almost no change in position and size for a certain period of time and maintains a stationary state in front of the camera, it is considered highly likely to be a user who actually uses the kiosk.

[0066] Additionally, as illustrated in FIG. 4, if a face in the unverified state (410) passes the stability verification, it can be directly transferred to the verified state (430), and if a face in the unverified state satisfies the depth condition, it can be transferred to the verified state (430) via the waiting state (420). Conversely, if a face in the waiting state (420) fails to pass the stability verification within a preset time, it can be returned to the unverified state (410) or removed.

[0067] Through such a state-based face classification and transition structure, the present invention can perform highly reliable identification of actual users considering temporal stability rather than simple single-frame-based judgment, and enables more accurate and stable user selection even in environments with multiple users.

[0068] Figure 5 is a diagram illustrating an example of calculating a pose estimation score in face stability verification.

[0069] Referring to FIG. 5, the geometric structure of the face can be defined using the ends of both eyes, the nose, and the mouth among the face landmarks obtained in the face detection step. Specifically, a first horizontal distance (B1) and a second horizontal distance (B2), which are horizontal distances, can be calculated using the distance between both eyes and the distance between both ends of the mouth, and a first vertical distance (A1) and a second vertical distance (A2), which are vertical distances, can be calculated using the distance between each eye and the corresponding mouth end point.

[0070] In this embodiment, a pose estimation score representing the frontal view of the face is calculated using the relationship between horizontal distances (B1, B2) and vertical distances (A1, A2). The pose estimation score is calculated by comparing the sum of vertical distances (A1+A2) and the sum of horizontal distances (B1+B2), and the value can be defined by the following formula.

[0071] PS = min(A1 + A2, B1 + B2) / max(A1 + A2, B1 + B2)

[0072] According to this formula, when the face is facing the camera directly, the sum of the vertical distances and the sum of the horizontal distances have similar values, so the pose estimation score approaches 1. On the other hand, when the face is tilted left or right or up or down, the ratio between the vertical distances and the horizontal distances changes significantly, causing the pose estimation score to decrease.

[0073] Therefore, the present invention can improve the accuracy of face recognition and subsequent processing by preferentially selecting faces that are close to the front using pose estimation scores. For example, if there are multiple faces classified as having completed verification, faces with higher pose estimation scores can be considered preferentially, or frames satisfying pose estimation scores above a certain threshold can be selected and utilized for determining actual users.

[0074] As such, the pose estimation score calculated based on simple geometric relationships enables effective determination of whether a face is facing forward without additional complex calculations, and can be efficiently applied even in real-time kiosk environments.

[0075] FIG. 6 is a configuration diagram of a user-specific device in a single-camera-based kiosk environment according to one embodiment of the present invention.

[0076] Referring to FIG. 6, the actual user specific device according to the present invention includes a trigger unit (610), a face detection unit (620), a zone filtering unit (630), a face tracking unit (640), a face stability verification unit (650), and an actual user selection unit (660), and each component operates in correspondence with each step of the actual user specific method described in FIG. 2.

[0077] First, the trigger unit (610) is activated by a proximity sensor or touch input provided in the kiosk, and controls subsequent image processing operations to be performed only after the trigger is activated. This prevents unnecessary consumption of computational resources.

[0078] The face detection unit (620) receives an image frame input from a camera after the activation of the trigger unit (610) and detects one or more face regions and landmark information corresponding to the face regions using a face detection model. At this time, the bounding box and landmark information of the detected face regions are utilized in subsequent processing.

[0079] The zone filtering unit (630) divides the input image frame into multiple regions in the vertical direction and selects valid faces by applying different face size thresholds to each region. For example, by applying different minimum face area criteria to the upper, middle, and lower regions, users with various physical conditions, such as wheelchair users and children, can be effectively detected while removing distant figures or small faces in the background.

[0080] The face tracking unit (640) performs Intersection over Union (IoU) based tracking on the face that has passed through the zone filtering unit (630) and assigns a unique identifier to each face. Additionally, depending on the IoU matching result, if it does not match with an existing face, a new identifier is assigned and tracking is initialized; if it matches, the tracking information of the corresponding identifier is updated. Along with this, tracking information including a face image buffer, a landmark buffer, and an IoU buffer is generated and managed.

[0081] The face stability verification unit (650) performs multi-stage stability verification on the face being tracked. Specifically, it determines the movement stability of the face using the average IoU value for a preset number of frames, and determines whether the user has stopped in front of the kiosk by analyzing whether the change in the area of ​​the face region between consecutive frames converges to a preset threshold. In addition, it can evaluate the front view by calculating a pose estimation score based on face landmarks. Based on these verification results, each face is classified into an unverified state, a waiting state, or a verified state.

[0082] The actual user selection unit (660) selects the final actual user when there is one or more faces classified as having completed verification. Specifically, if there is only one face with completed verification, the actual user can be determined based on the pose estimation score of the face, and if there are multiple faces with completed verification, the depth value of each face can be calculated using a monocular depth estimation model, and the face with the largest depth value can be selected as the actual user.

[0083] The present invention has been described above with reference to its embodiments. Those skilled in the art will understand that the present invention may be implemented in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

Claims

Claim 1 A method for identifying an actual user of a kiosk among multiple individuals included in a camera image in a kiosk environment equipped with a single camera, comprising: a step of activating a trigger by a proximity sensor or touch input provided in the kiosk; a step of detecting one or more face regions and landmark information corresponding to the face regions using a face detection model in a video frame input from the camera after the trigger is activated; a step of performing zone filtering to select only valid faces by dividing the video frame into multiple regions in the vertical direction and applying different face size thresholds to each region; a step of performing inter-frame Intersection over Union (IoU) based tracking on faces that have passed the zone filtering to assign a unique identifier to each face, and generating and updating tracking information including a face image buffer, a landmark buffer, and an IoU buffer; and a step of performing face stability verification on the faces being tracked, the verification of stability using an average IoU value for a preset number of frames, a change amount verification determining whether the amount of change in face region area between consecutive frames converges to below a threshold, and an evaluation using a pose estimation score based on a frontal view calculated from face landmarks, thereby classifying each face into an unverified state, a waiting state, or a verified state. A method for identifying a real user in a single-camera-based kiosk environment, comprising: a step of classifying into states; and, if there is one or more faces classified into the verified state, calculating a depth value for each face using a monocular depth estimation model and selecting the face closest to the camera based on the depth value, or, if there is only one face in the verified state, determining the real user based on the pose estimation score of the face. Claim 2 A method for identifying actual users in a single-camera-based kiosk environment, wherein the step of performing zone filtering in claim 1 is characterized by dividing an image frame into an upper area, a middle area, and a lower area, and applying different face area thresholds to each area to remove faces included in distant people or backgrounds. Claim 3 A method for identifying an actual user in a single-camera-based kiosk environment, wherein, in claim 1, the step of generating and updating tracking information is characterized by assigning a new identifier and initializing a tracker when the existing face is not matched according to the IoU-based matching result during tracking, and updating the tracking information of the corresponding identifier when a match is made. Claim 4 A method for identifying a real user in a single-camera-based kiosk environment, characterized in that, in claim 1, when performing the face stability verification, the average IoU value is 0.9 or higher for a preset number of frames and is determined to be a stable face. Claim 5 A method for identifying a real user in a single-camera-based kiosk environment, characterized in that, in the first aspect, when performing the face stability verification, the area change of the face region in a continuous frame is analyzed as a time series, and if the state in which the amount of area change is below a preset threshold value persists for a certain number of times or more, the face is determined to be stabilized. Claim 6 A method for identifying an actual user in a single-camera-based kiosk environment, wherein, in claim 1, the actual user determination step is characterized by calculating a depth value for each face using a monocular depth estimation model when there are multiple faces in a verified state, and selecting the face with the largest depth value as the actual user. Claim 7 A device for identifying an actual user of a kiosk among multiple individuals included in a camera image in a kiosk environment equipped with a single camera, comprising: a trigger unit activated by a proximity sensor or touch input; a face detection unit that detects one or more face regions and landmark information corresponding to the face regions using a face detection model in a video frame input from the camera after the trigger is activated; a zone filtering unit that divides the video frame into multiple regions in a vertical direction and selects only valid faces by applying different face size thresholds to each region; a face tracking unit that performs frame-to-frame Intersection over Union (IoU)-based tracking on faces that have passed the zone filtering, assigns a unique identifier to each face, and generates and updates tracking information including a face image buffer, a landmark buffer, and an IoU buffer; and a face stability verification for the faces being tracked, which includes stability verification using an average IoU value for a preset number of frames, change amount verification determining whether the amount of change in face region area between consecutive frames converges to below a threshold, and evaluation using a pose estimation score based on a frontal view calculated from face landmarks, thereby classifying each face into an unverified state, a waiting state, or a verified state. A device for identifying a real user in a single-camera-based kiosk environment, characterized by including: a stability verification unit; and a real user selection unit that, when there is one or more faces classified as having completed verification, calculates a depth value for each face using a monocular depth estimation model and selects the face closest to the camera based on the depth value, or, when there is only one face in the completed verification state, determines the real user based on the pose estimation score of the face. Claim 8 A real user-specific device in a single-camera-based kiosk environment, wherein, in claim 7, the zone filtering unit divides the image frame into an upper area, a middle area, and a lower area, and applies different face area thresholds for each area to remove faces included in distant people or backgrounds. Claim 9 A device for identifying actual users in a single-camera-based kiosk environment, wherein, in claim 7, the face tracking unit assigns a new identifier and initializes the tracker when it does not match with an existing face according to an IoU-based matching result, and updates the tracking information of the corresponding identifier when it matches. Claim 10 A device for identifying actual users in a single-camera-based kiosk environment, characterized in that, in claim 7, the face stability verification unit determines a face to be stable when the average IoU value for a preset number of frames is greater than or equal to a reference value, and determines that the face is stabilized when the amount of change in the face region between consecutive frames is less than or equal to a preset threshold value for a certain number of consecutive times. Claim 11 In claim 7, the actual user selection unit is characterized by calculating a depth value for each face using a monocular depth estimation model when there are multiple faces in a verified state, and selecting the face with the largest depth value as the actual user, in a device for identifying an actual user in a single camera-based kiosk environment.

Images