Apparatus for recognizing face and method thereof

By combining facial information and character information, a facial character matching table is established and the weight is adjusted, which solves the problems of low facial recognition performance and high error recognition rate in the prior art, and achieves more accurate and efficient facial recognition.

CN118942129APending Publication Date: 2024-11-12HYUNDAI MOTOR CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311534296.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2023-11-16
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing facial recognition technology does not use human facial features, resulting in low facial recognition performance and difficulty in achieving accurate facial recognition and minimizing misidentification.

Method used

By using facial information and character information to distinguish, a continuously updated facial character matching table is established, and facial weights and character weights are adjusted to improve the accuracy and performance of facial recognition.

Benefits of technology

Continuous tracking and recognition is realized when facial information and character information are lost, reducing facial error recognition rate and improving the performance of re-identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942129A_ABST
    Figure CN118942129A_ABST
Patent Text Reader

Abstract

Various embodiments relate to an apparatus for recognizing a face and a method thereof. A face recognition apparatus of an exemplary embodiment includes: a processor configured to extract face information and person information from image data and to recognize a face using the face information and the person information; and a memory configured to store data and an algorithm driven by the processor.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2023-0060088, filed on May 9, 2023, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a facial recognition device and a method thereof. Background Art

[0004] Recently, facial recognition technology has been commercialized by being applied to various services such as unlocking smartphones, banking, public transportation, convenience stores, and security devices.

[0005] Existing facial recognition technology compares a facial image taken on the spot with an already registered image and only outputs a result of whether it is the same person.

[0006] That is, in the related art, since information related to human facial features is not used during facial recognition, facial recognition performance is low. Summary of the invention

[0007] The present disclosure relates to a facial recognition device and method thereof. Detailed description of the invention The present disclosure relates to a technology for improving the accuracy of facial recognition technology.

[0008] Exemplary embodiments of the present disclosure provide a facial recognition apparatus and method thereof, which can continuously track to accurately recognize a face by recognizing a face using the facial information and the person information in response to loss of one of the facial information and the person information.

[0009] Exemplary embodiments of the present disclosure also provide a facial recognition device and method thereof, which are capable of recognizing a face based on a continuously updated face-person matching table without similarity comparison of expressor vectors to minimize misrecognition of a face.

[0010] Furthermore, exemplary embodiments of the present disclosure provide a face recognition apparatus and method thereof, which are capable of improving re-recognition performance by adjusting a face weight and a person weight according to situations of the face weight and the person weight during re-recognition.

[0011] The technical objectives of the embodiments of the present disclosure are not limited to the objectives described above, and those skilled in the art can clearly understand other technical objectives not mentioned from the description of the claims.

[0012] An exemplary embodiment of the present disclosure provides a facial recognition device, including: a processor configured to extract facial information and person information from image data and recognize a face using the facial information and person information; and a memory configured to store data and algorithms driven by the processor.

[0013] In an exemplary embodiment of the present disclosure, the processor may be configured to detect a person region and a face region from image data, create a bounding box in the person region and assign a person ID for tracking, and create a bounding box in the face region and assign a face ID for tracking.

[0014] In an exemplary embodiment of the present disclosure, the processor may be configured to determine whether the person ID and the face ID are registered in the face-person matching table.

[0015] In an exemplary embodiment of the present disclosure, the memory may be configured to store a face-person matching table, and the face-person matching table is configured to store face IDs, face expressions, person IDs, person expressions, and comprehensive IDs by matching them.

[0016] In an exemplary embodiment of the present disclosure, the processor may be configured to recognize a face using an ID registered in the face-person matching table in response to a situation in which the person ID and the face ID are registered in the face-person matching table.

[0017] In an exemplary embodiment of the present disclosure, the processor may be configured to, in response to a situation where both a facial region and a person region are detected in a first frame of image data but no facial region is detected in a second frame of image data, query whether a person ID of the person region detected in the second frame is registered in a facial person matching table, and in response to a situation where the person ID of the person region detected in the second frame is registered in the facial person matching table, identify a face with a corresponding ID.

[0018] In an exemplary embodiment of the present disclosure, the processor may be configured to assign a new face ID to the face area detected again in the third frame of the image data, and update the face ID registered in the face person matching table to the new face ID.

[0019] In an exemplary embodiment of the present disclosure, the processor may be configured to, in response to a situation where both a facial region and a person region are detected in a first frame of image data but no person region is detected in a second frame of image data, query whether a facial ID of the facial region detected in the second frame is registered in a facial-person matching table, and in response to a situation where the facial ID of the facial region detected in the second frame is registered in the facial-person matching table, identify a face with a corresponding ID.

[0020] In an exemplary embodiment of the present disclosure, the processor may be configured to assign a new person ID to the person region detected again in the third frame of the image data, and update the person ID registered in the face person matching table to the new person ID.

[0021] In an exemplary embodiment of the present disclosure, the processor may be configured to extract person information from the person area and extract facial information from the facial area to determine whether the person information and the facial information are information about the same person in response to a situation where the person ID and the facial ID are not registered in the facial person matching table.

[0022] In an exemplary embodiment of the present disclosure, the processor may be configured to determine whether the information belongs to the same person by determining whether the pixel distance difference between the facial position extracted from the person information and the facial position extracted from the person area is less than a predetermined first threshold.

[0023] In an exemplary embodiment of the present disclosure, the processor may be configured to determine whether the information belongs to the same person by determining whether the difference between the distance from the camera to the person area and the distance from the camera to the face area is less than a predetermined second threshold.

[0024] In an exemplary embodiment of the present disclosure, the processor may be configured to determine whether the information belongs to the same person by determining whether the size of the overlapping area between the bounding box of the person area and the bounding box of the face area is greater than a predetermined third threshold.

[0025] In an exemplary embodiment of the present disclosure, the processor may be configured to perform a similarity comparison between the extracted character information and the extracted facial information and the character information and the facial information registered in a face-character matching table in response to determining that the character information and the facial information are information of the same person.

[0026] In an exemplary embodiment of the present disclosure, the processor may be configured to perform a similarity comparison by using the similarity between the character expressors of the extracted character information and the character expressors of the character information registered in the facial character matching table, the similarity between the facial expressors of the extracted facial information and the facial expressors of the facial information registered in the facial character matching table, and a predetermined facial weight and a predetermined character weight.

[0027] In an exemplary embodiment of the present disclosure, the processor may be configured to increase the face weight and decrease the person weight as the difference between the input time of the facial expression symbol of the extracted facial information and the registration time of the facial expression symbol of the facial information registered in the face-person matching table increases.

[0028] In an exemplary embodiment of the present disclosure, the processor may be configured to increase the person weight and decrease the face weight as the distance from the camera to the person area increases.

[0029] In an exemplary embodiment of the present disclosure, the processor may be configured to determine whether the similarity between the extracted person information and the extracted facial information and the person information and facial information registered in the face-person matching table is greater than a predetermined similarity threshold.

[0030] In an exemplary embodiment of the present disclosure, the processor may be configured to, in response to a situation where the similarity between the extracted character information and the extracted facial information and the character information and facial information registered in the facial character matching table is greater than a predetermined similarity threshold, determine that the extracted character information and the extracted facial information are information of the same person as the character information and facial information registered in the facial character matching table, and in response to a situation where the similarity between the extracted character information and the extracted facial information and the character information and facial information registered in the facial character matching table is equal to or less than a predetermined similarity threshold, newly register the extracted character information and the extracted facial information in the facial character matching table.

[0031] An exemplary embodiment of the present disclosure provides a facial recognition method, including extracting facial information and person information from image data, and recognizing a face using the facial information and person information.

[0032] According to the present embodiment, by using both facial information and person information, in response to loss of one information, continuous tracking can be performed to accurately recognize the face.

[0033] According to the present embodiment, it is possible to minimize misidentification of faces by performing face recognition based on a continuously updated face-person matching table without similarity comparison of expression vectors.

[0034] Furthermore, according to the present embodiment, by adjusting the face weight and the person weight according to the circumstances of the face weight and the person weight during re-recognition, the re-recognition performance can be improved.

[0035] Furthermore, various effects that can be directly or indirectly recognized through this document can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A block diagram illustrating an example configuration of a facial recognition device.

[0037] Figure 2 A diagram for describing an example flow of a face recognition process is shown.

[0038] Figure 3 A view for describing a process of determining the same person using character information and face information is shown.

[0039] Figure 4 An example view of a face person matching table is shown.

[0040] Figure 5 A diagram for describing a facial recognition process in response to a case where facial information is lost is shown.

[0041] Figure 6 A diagram for describing a face recognition process in response to a case where personal information is lost is shown.

[0042] Figure 7 A diagram for describing a face recognition process in response to a case where both person information and face information are lost is shown.

[0043] Figure 8 A flow chart showing an example facial recognition method is shown.

[0044] Fig. 9 An example computing system is shown. DETAILED DESCRIPTION

[0045] Hereinafter, some exemplary embodiments of the present disclosure will be described in detail with reference to the exemplary drawings. It should be noted that when reference numerals are added to the constituent elements of the various drawings, even if the same constituent elements are shown in different drawings, they have the same reference numerals as much as possible. When describing the exemplary embodiments, when it is determined that the detailed description of the well-known configuration or function associated with the exemplary embodiments may obscure the main points of the exemplary embodiments of the present disclosure, its description will be omitted.

[0046] When describing the constituent elements according to the exemplary embodiments, terms such as first, second, A, B, (a) and (b) may be used. These terms are only used to distinguish constituent elements from other constituent elements, and the properties, sequence or order of the constituent elements are not limited by these terms. In addition, unless defined differently, all terms (including technical scientific terms) used in this article have the same meaning as the meanings commonly understood by technicians in the technical field (technical personnel in the art) of the exemplary embodiments of the present disclosure. The terms defined in commonly used dictionaries should be interpreted as having meanings that match the meanings in the context of the relevant technology, and should not be interpreted as having idealized or overly formal meanings unless they are clearly defined in this specification.

[0047] In the following, reference will be made to Figures 1 to 9 Various exemplary embodiments of the present disclosure are described in detail.

[0048] Figure 1 A block diagram showing an example configuration of a facial recognition device is shown, and Figure 2A diagram for describing an example flow of a face recognition process is shown.

[0049] The facial recognition apparatus 100 according to an exemplary embodiment of the present disclosure may recognize a user's face from image data input through the sensing device 200 .

[0050] The facial recognition device 100 may be implemented inside a system (eg, a vehicle, a robot, etc.) that requires facial recognition or separately from the system. In addition, the facial recognition device 100 may be used in a surveillance system using CCTV and a robot system.

[0051] The facial recognition device 100 may be formed integrally with an internal control unit of a system using the device 100, or it may be implemented as a separate hardware device connected to a control unit of a vehicle via a connection device. For example, the facial recognition device 100 may be implemented integrally with the system, or it may be installed or attached as a separate element of the system.

[0052] The facial recognition device 100 may extract facial information and person information from the image data, and the facial recognition device may recognize a face using the facial information and person information.

[0053] The sensing device 200 may include a camera 210 for photographing a user's face.

[0054] Reference Figure 1 , the facial recognition device 100 may include a communication device 110 , a memory (ie, storage) 120 , an interface device 130 , and a processor 140 .

[0055] The communication device 110 is a hardware device implemented by various electronic circuits, which sends and receives signals through wireless or wired connections, and the communication device 110 can send and receive information based on the intra-system device and network communication technology. For example, the network communication technology may include controller area network (CAN) communication, local interconnect network (LIN) communication, FlexRay communication, etc.

[0056] In addition, the communication device 110 can communicate with external servers, infrastructure, other robots, etc. through wireless Internet access or short-range communication technology. Here, the wireless communication technology may include wireless LAN (WLAN), wireless broadband (Wibro), Wi-Fi, World Interoperability for Microwave Access (Wimax), etc. In addition, the short-range communication technology may include ZigBee, Ultra-Wideband (UWB), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), etc.

[0057] As an example, the communication device 110 may receive image data by communicating with the sensing device 200, and the communication device 110 may transmit facial recognition information recognized by the processor 140 to another device using the same device. In addition, the communication device 110 may receive camera depth information from the camera 210. In this case, the camera depth information may include a distance from the camera 210 to a person area, a distance from the camera 210 to a face area, and the like.

[0058] The memory 120 may store sensing results of the sensing device 200 and data and / or algorithms required for the operation of the processor 140 .

[0059] As an example, the memory 120 may store Figure 4 The face person matching table shown in , etc. The face person matching table (FPMT) may include the facial features, the face tracking ID for face tracking, the person features, the person tracking ID for person tracking, and the comprehensive ID by matching the facial features, the face tracking ID for face tracking, the person features, and the person tracking ID for person tracking.

[0060] The memory 120 may include at least one type of storage medium among the following types of memories: such as flash memory, hard disk, micro memory, card (e.g., secure digital (SD) card or extreme digital (XD) card), random access memory (RAM), static RAM (SRAM), read-only memory (ROM), programmable ROM (PROM), electrically erasable PROM (EEPROM), magnetic memory (MRAM), magnetic disk, and optical disk.

[0061] The interface device 130 may include an input device for receiving a control command from a user and an output device for outputting an operation state of the device 100 and a result thereof. Herein, the input device may include a key, and the input device may include a mouse, a joystick, a jog shuttle, a stylus pen, etc. In addition, the input device may include a soft key implemented on a display.

[0062] The output device may include a display, and the output device may also include a voice output device such as a speaker. In this case, in response to a touch sensor formed by a touch film, a touch sheet, or a touch pad being provided on the display, the display may operate as a touch screen, and it may be implemented in a form in which an input device and an output device are integrated. In an embodiment of the present disclosure, the output device may display a face recognition result and a screen for face registration.

[0063] In this case, the display may include at least one of a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFTLCD), an organic light emitting diode display (OLED display), a flexible display, a field emission display (FED), a 3D display, or any combination thereof.

[0064] The processor 140 may be electrically connected to the communication device 110 , the memory 120 , the interface device 130 , etc., may electrically control each component, and may be a circuit that executes software commands, thereby performing various data processing and calculations described below.

[0065] The processor 140 may process signals transmitted between components in flight of the facial recognition device 100 to perform overall control so that each component can perform its function normally. The processor 140 may be implemented in the form of hardware, software, or a combination of hardware and software. For example, the processor 140 may be implemented as a microprocessor, but the embodiments of the present disclosure are not limited thereto.

[0066] The processor 140 can detect the person area 30 and the face area 20 from the input image data 10, create a bounding box in the person area 30, give it a person ID for tracking, create a bounding box in the face area 20, and give it a face ID for tracking.

[0067] The person region 30 indicates a region including a human body shape and is handled as a bounding box from the human body shape to a predetermined pixel range, and the face region 20 indicates a region including a face shape and is handled as a bounding box from the face shape to a predetermined pixel range.

[0068] The person ID and the face ID are IDs used for tracking, and assuming that the first frame, the second frame, and the third frame are input among the input image data, they can be assigned according to the position of the bounding box of the person area 30 or the face area 20. For example, in response to the case where the bounding box of person A is located at the center of the image in the first frame and person ID1 is assigned to the bounding box of person A, in the second frame, person ID1 can also be assigned to the bounding box located at the center of the image. As a result, person ID1 can be continuously tracked.

[0069] Thus, according to an embodiment of the present disclosure, a tracking algorithm is used for object tracking, and the tracking algorithm may include algorithms such as an IOU tracker, Simple Online and Real-time Tracking (SORT), and DeepSORT.

[0070] The processor 140 may determine whether the person ID and the face ID are registered in a face person matching table (FPMT). In this case, the face person matching table may store the face ID, the face expression symbol, the person ID, the person expression symbol, and the comprehensive ID by matching the face ID, the face expression symbol, the person ID, the person expression symbol, and the comprehensive ID (see later). Figure 4 ).

[0071] The processor 140 may recognize the face using the ID registered in the face-person matching table in response to the person ID and the face ID being registered in the face-person matching table.

[0072] The processor 140 detects both the facial area 20 and the person area 30 in the first frame of the image data, but in response to the fact that the facial area 20 is not detected in the second frame of the image data, it can query whether the person ID of the person area 30 detected in the second frame is registered in the facial person matching table, and in response to the fact that the person ID of the person area 30 detected in the second frame is registered in the facial person matching table, the face can be identified by using the corresponding ID.

[0073] In response to the facial region 20 being detected again in the third frame of the image data, the processor 140 may again assign a new facial ID to the detected facial region 20 to update the facial ID registered in the facial person matching table to the new facial ID.

[0074] The processor 140 detects both the facial area 20 and the person area 30 in the first frame of the image data, but in response to the fact that the person area 30 is not detected in the second frame of the image data, it can query whether the facial ID of the facial area 20 detected in the second frame is registered in the facial person matching table, and in response to the fact that the facial ID of the facial area 20 detected in the second frame is registered in the facial person matching table, the face can be identified by using the corresponding ID.

[0075] In response to the person region 30 being detected again in the third frame of the image data, the processor 140 may assign a new person ID to the detected person region 30 again to update the person ID registered in the facial person matching table to the new person ID.

[0076] In response to the fact that the person ID and the face ID are not registered in the face-person matching table, the processor 140 may extract person information from the person area 30 and face information from the face area 20 to determine whether the person information and the face information are information of the same person.

[0077] The person information includes person expressors (feature vectors) in the person region 30, and the face information includes face expressors (feature vectors) in the face region 20. In this case, the expressor means that image data is represented as a vector based on a deep learning model.

[0078] The processor 140 may determine whether the information belongs to the same person by determining whether a pixel distance difference between a facial position extracted from the person information and a facial position extracted from the person region 30 is less than a predetermined first threshold.

[0079] The processor 140 may determine whether the information belongs to the same person by determining whether the difference between the distance from the camera to the person area 30 and the distance from the camera to the face area 20 is less than a predetermined second threshold.

[0080] In addition, the processor 140 may determine whether the information belongs to the same person by determining whether the size of the overlapping area between the bounding box of the person area 30 and the bounding box of the face area 20 is greater than a predetermined third threshold.

[0081] In response to determining that the person information and the facial information are information of the same person, the processor 140 may perform a similarity comparison between the extracted person information and the extracted facial information and the person information and facial information registered in the face-person matching table.

[0082] The processor 140 can perform a similarity comparison by using the similarity between the character expressors of the extracted character information and the character expressors of the character information registered in the facial character matching table, the similarity between the facial expressors of the extracted facial information and the facial expressors of the facial information registered in the facial character matching table, and a predetermined facial weight and a predetermined character weight.

[0083] As the difference between the input time of the facial expressor of the extracted facial information and the registration time of the facial expressor of the facial information registered in the face-person matching table increases, the processor 140 may increase the face weight and decrease the person weight.

[0084] The processor 140 may increase the person weight and decrease the face weight as the distance from the camera to the person region 30 increases.

[0085] The processor 140 may determine whether the similarity between the extracted person information and the extracted facial information and the person information and facial information registered in the face-person matching table is greater than a predetermined similarity threshold.

[0086] In response to the situation where the similarity between the extracted character information and the extracted facial information and the character information and facial information registered in the facial character matching table is greater than a predetermined similarity threshold, the processor 140 can determine that the extracted character information and the extracted facial information are information about the same person as the character information and the facial information registered in the facial character matching table.

[0087] In response to the situation where the similarity between the extracted person information and the extracted facial information and the person information and facial information registered in the facial person matching table is equal to or less than a predetermined similarity threshold, the processor 140 may newly register the extracted person information and the extracted facial information in the facial person matching table.

[0088] To this end, the processor 140 may include a detector 141 , a facial information extractor 142 , a person information extractor 143 , a same person determiner 144 , and a re-identifier 145 .

[0089] refer to Figure 2 , the detector 141 detects the face region 20 and the person region 30 in the image data 10 obtained from the sensing device 200 to create a bounding box in the detected region, and the detector 141 tracks the bounding box information to assign a face ID and a person ID to the face region 20 and the person region 30, respectively. In this case, tracking of the face region 20 and tracking of the person region 30 may be performed separately. In addition, the detector 141 may receive the distance from the camera 210 to the face region 20 (face depth) and the distance from the camera 210 to the person region 30 (person depth) through the communication device 110. In addition, the detector 141 may estimate the distance from the camera 210 to the face region 20 (face depth) and the distance from the camera 210 to the person region 30 (person depth) using the size of the bounding box and the distance to the camera 210.

[0090] According to an embodiment of the present disclosure, Figure 1 and Figure 2 In the present invention, in one detector 141, the person region 30 and the face region 20 may be classified into different categories to be detected, and the detector 141 may be divided into a person detector and a face detector.

[0091] The facial information extractor 142 extracts facial expressions (feature vectors) (facial information) 40 from the facial region 20 detected by the detector 141. Thus, the facial information extractor 142 may provide the facial bounding box, facial tracking ID, facial depth information, and facial expressions (facial feature vectors) to the same person determiner 144.

[0092] The person information extractor 143 extracts a person expression (person feature vector) (person information) 50 and person posture information (person landmark) 60 from the person area 30 detected by the detector 141. Thus, the person information extractor 143 can provide the person bounding box, person tracking ID, person depth information, person expression (person feature vector) and person landmark to the same person determiner 144.

[0093] In this case, faces or people are recognized by representing the RGB image as vectors using a deep learning model. In the deep learning model, the same people are closely clustered in the vector space, while other people are trained to be located far away from these areas, and these feature vectors are called expressions (feature vectors). The character posture information 60 may include the positions of the ears, eyes, nose, and various joints of the human body on the image data.

[0094] The same person determiner 144 may determine whether the facial information extracted from the face extractor 142 and the person information extracted from the person information extractor 143 indicate the same person. Figure 3 As shown, in response to the situation where there are n pieces of face information and m pieces of detected person information, a process of determining whether all the n faces and all the m persons indicate the same person is performed. Figure 3 A view for describing a process of determining the same person using character information and face information is shown.

[0095] The same person determiner 144 may determine whether the following three conditions are satisfied in order to determine whether the person information and the facial information indicate the same person.

[0096] First, the same person determiner 144 determines whether the pixel distance between the face position extracted by the person information extractor 143 and the face position extracted by the detector 141 is less than a predetermined threshold Thr_1 to determine whether they are the same (condition 1). In this case, the face position extracted by the person information extractor 143 includes the average position of the face area (ears, eyes, and nose) among the person position information (person landmark), and the face position extracted by the detector 141 may include the center position of the face bounding box of the face area 20.

[0097] In addition, the same person determiner 144 can determine whether the difference between the distance from the camera 210 to the person (person depth) and the distance from the camera 210 to the face (face depth) is less than a predetermined threshold Thr_2 to determine whether they are the same (condition 2).

[0098] In this case, the difference between the distance from the camera 210 to the person and the distance from the camera 210 to the face may be obtained from the camera 210 or may be estimated based on a deep learning algorithm using the size of the bounding box.

[0099] In addition, the same person determiner 144 may determine whether the size of the overlapping area of ​​the bounding box of the person area 30 and the bounding box of the face area 20 is greater than a predetermined threshold Thr_3 to determine whether they are the same (condition 3). In this case, as will be described later Figure 5 As shown, the bounding box 31 of the person region 30 and the bounding box 21 of the face region 20 may overlap for the same person, and may not overlap for different people. In addition, the same person determiner 144 may determine the size of the overlapping area of ​​the bounding box of the person region 30 and the bounding box of the face region 20, and may compare the size with the threshold Thr_3 to determine whether the same person is indicated. In this case, the threshold Thr_1, the threshold Thr_2, and the threshold Thr_3 may be predetermined by experimental values ​​to be stored in the memory 120.

[0100] The same person determiner 144 can determine that the corresponding facial information and person information indicate the same person in response to the situation that all three conditions are satisfied. In this case, the priority of the conditions can be determined in the order of condition 1, condition 2, and condition 3, and the priority can be changed according to the situation, or some of the three conditions can be used.

[0101] In response to the situation that the corresponding facial information and person information indicate the same person, the same person determiner 144 registers the facial information expressor (facial features), the facial tracking ID (e.g., 1), the person information expressor (person features), and the person tracking ID (e.g., 5) by matching the facial information expressor (facial features), the facial tracking ID, the person information expressor (person features), and the person tracking ID with the comprehensive ID (e.g., 1). Thus, continuous tracking and re-identification are possible by utilizing the information registered in the facial person matching table.

[0102] The re-identifier 145 may recognize faces from the image data using a face person matching table.

[0103] Figure 5 A view for describing a facial recognition process in response to a situation where facial information is lost is shown. In this case, the situation where facial information is lost may include a situation where a person turns around and the person is detected but the face is not detected, etc.

[0104] refer to Figure 5In view 501, the facial recognition device 100 maps the face ID=1 and the person ID=5 determined to indicate the same person, and assigns a comprehensive ID (face-person ID=1) to register it in the face person matching table (FPMT). In this case, in response to the person information and the facial information indicating the same person, the bounding box 31 of the person area and the bounding box 21 of the face area can be overlapped and displayed.

[0105] Subsequently, in response to the situation where facial information is lost for face ID = 1 as in view 502, the facial recognition device 100 loses the face ID, but has a person ID (person = 5) as in view 503, and it can therefore be seen that based on the information registered in the face-person matching table (FPMT), the comprehensive ID is face-person ID = 1.

[0106] Furthermore, as shown in view 504, in response to a situation where a face is recognized again later (eg, a situation where a face is not initially detected, but the face turns toward the camera 210 and is detected again), the face ID is changed (eg, face ID=9).

[0107] Thus, as shown in view 505, the facial recognition device 100 determines whether the new facial information (face ID = 9) and the person information (person ID = 5) indicate the same person, and in response to determining that the new facial information (face ID = 9) and the person information (person ID = 5) indicate the same person, the changed facial information (face ID = 9) is matched with the person information (person ID = 5) to assign a comprehensive ID (face-person ID = 1), and the face person matching table (FPMT) is updated. That is, the facial recognition device 100 updates the new facial information (face ID = 9) to the person information (person ID = 5).

[0108] Figure 6 A view for describing a facial recognition process in response to a situation where person information is lost is shown. In this case, the situation where person information is lost may include a situation where only the face is shown because an obstacle covers the person.

[0109] Reference Figure 6 601 , the facial recognition device 100 maps the face ID=1 and the person ID=5 determined to indicate the same person, and assigns a comprehensive ID (face-person ID=1) to register it in the face-person matching table (FPMT).

[0110] Subsequently, in response to the situation where the person information is lost for person ID = 5 as in view 602, the facial recognition device 100 loses the person ID, but has the face ID (face ID = 1) as in view 603, and it can therefore be seen that based on the information registered in the face-person matching table (FPMT), the integrated ID is face-person ID = 1.

[0111] In response to a situation where a person is recognized later as in view 604 (for example, a situation where an obstacle is passed and the person is detected again), the facial recognition device 100 determines whether the newly detected person information (person ID=20) and the facial information (face ID=1) indicate the same person, and in response to determining that the newly detected person information (person ID=20) and the facial information (face ID=1) indicate the same person, the changed person information (person ID=20) is matched with the existing facial information (face ID=1) to assign a comprehensive ID (face-person ID=1), and is updated in the facial person matching table (FPMT).

[0112] Figure 7 A view for describing a facial recognition process in response to a situation where both person information and facial information are lost is shown. In this case, the situation where both person information and facial information are lost may include a situation where the person information and facial information are not obtained due to the person temporarily moving out of the angle of the camera 210 when the person information and facial information are continuously acquired.

[0113] refer to Figure 7 In response to detecting new person information and new facial information, the facial recognition device 100 uses the above three conditions to determine whether the new person information and the new facial information indicate the same person. As in view 701, for example, assuming that the face ID of the new facial information is 15 and the person ID of the new person information is 10.

[0114] In response to the situation where the new facial information (face ID=15) and the new person information (person ID=10) indicate the same person as in view 702, the facial recognition device 100 determines whether the corresponding ID information (face ID=15, person ID=10) is registered in the face-person matching table.

[0115] In response to the fact that the corresponding ID information (face ID=15 and person ID=10) has been registered in the face-person matching table, the facial recognition device 100 performs facial recognition using the registered IDs.

[0116] On the other hand, in response to the situation where the ID information (face ID=15 and person ID=10) is not registered in the facial person matching table, as in view 703, the facial recognition device 100 performs a similarity comparison between the expression symbols of the new person information and the expression symbols of the new facial information and the expression symbols of the person information and the expression symbols of the facial information registered in the facial person matching table.

[0117] That is, the facial recognition device 100 performs similarity comparison between the expressive symbols of the new person information and the expressive symbols of the new facial information and the expressive symbols of all person information and all facial information registered in the facial person matching table.

[0118] Equation 1:

[0119] Similarity = face weight * face similarity + person weight * person similarity

[0120] The facial similarity and the person similarity may be calculated as in Equation 2 below.

[0121] Equation 2:

[0122]

[0123] S c (A, B) is the cosine similarity of A and B.

[0124] In face similarity, A indicates a face expression previously registered in the face person matching table, and B indicates a face expression extracted from new face information. In person similarity, A indicates a person expression previously registered in the face person matching table, and B indicates a person expression extracted from new person information.

[0125] In this case, the sum of the face weight and the person weight may be 1, and the discrimination performance may be improved by adjusting the weight according to its situation.

[0126] For example, as the difference between the registration time of the facial information already registered in the face-person matching table and the input time of the newly input facial information increases, the face weight may increase and the person weight may decrease. This is because the appearance of a person may change over time.

[0127] In addition, as the distance from the person increases, the person weight increases and the face weight decreases. This is because as the distance increases, the face becomes smaller and difficult to recognize.

[0128] Referring to view 704, in response to the presence of an ID among the IDs registered in the facial person matching table whose similarity with the new facial information and the new person information is greater than a threshold, the facial recognition device 100 determines that the new person information and the new facial information are information of the same person as the corresponding IDs registered in the facial person matching table, and matches the new person information and the new facial information with the corresponding IDs determined to be the same person to update the facial person matching table (FPMT).

[0129] On the other hand, referring to view 705, in response to the fact that there is no ID greater than the similarity threshold among the IDs registered in the face person matching table, the facial recognition device 100 assigns a new comprehensive ID to the new facial information and the new person information to register them in the face person matching table (FPMT).

[0130] In this case, the similarity threshold value may be predetermined through experimental values ​​to be stored in the memory 120 .

[0131] In the following, reference will be made to Figure 8 A facial recognition method according to an exemplary embodiment of the present disclosure is described. Figure 8 A flow chart showing an example facial recognition method is shown.

[0132] In the following, it is assumed Figure 1 The facial recognition device 100 performs Figure 8 In addition, Figure 8 In the description, the operations described as being performed by the device can be understood as being controlled by the processor 140 of the facial recognition device 100.

[0133] refer to Figure 8 In response to the case where image data is input from the camera 210 or the like (S101), the facial recognition device 100 detects the person region 30 and the face region 20 from the image data (S102). In this case, the facial recognition device 100 creates bounding boxes in the person region 30 and the face region 20, respectively, and assigns a person ID and a face ID for tracking.

[0134] The facial recognition device 100 determines whether the person ID of the person area 30 and the person ID of the face area 20 belong to the person ID and the face ID registered in the face person matching table (S103). In response to the fact that the IDs of the person area 30 and the face area 20 are registered in the face person matching table in step S103, the corresponding IDs are identified (S104).

[0135] Subsequently, the facial recognition device 100 determines whether the person ID or face ID has changed (S105). That is, in response to the situation where the face area or person area is not detected temporarily and then the face area or person area is detected again, the facial recognition device 100 determines that the person ID and face ID of the re-detected person area or face area have changed.

[0136] In response to a change in the person ID or face ID in this manner, the existing information stored in the face person matching table is updated (S106). For example, in response to a change in the person ID to 9 when the person ID, face ID, and integrated ID are stored in the face person matching table as 1, 2, and 3, respectively, the person ID, face ID, and integrated ID are updated to person ID = 9, face ID = 2, and integrated ID = 3.

[0137] On the other hand, in response to the situation in step S103 where the IDs of the person area 30 and the facial area 20 are not registered in the facial person matching table, the facial recognition device 100 extracts person information (including posture information) and facial information from the detected person area 30 and facial area 20, respectively (S107 and S108).

[0138] Then, the facial recognition device 100 determines whether the unregistered person information and the facial information are information of the same person (S109). The facial recognition device 100 determines that the unregistered person information and the facial information are information of the same person in response to satisfying all three conditions for determining the same person.

[0139] In response to the fact that these information do not belong to the same person, the facial recognition device 100 may terminate the current process and execute the process starting from step S101 again.

[0140] On the other hand, in response to determining that the information belongs to the same person, the facial recognition device 100 performs a similarity comparison between the expression of the new person information and the expression of the new facial information and the expression of the person information and the expression of the facial information stored in the facial person matching table. In this case, the similarity comparison can be determined as in the above equations 1 and 2.

[0141] The facial recognition device 100 determines whether the similarity is greater than a predetermined similarity threshold (S110), and in response to the similarity being greater than the predetermined similarity threshold, identifies the integrated ID whose similarity is greater than the predetermined similarity threshold among the IDs registered in the face person matching table as the corresponding integrated ID (S112). That is, the facial recognition device 100 identifies the new person information and the new face information as the person (face) of the corresponding integrated ID.

[0142] On the other hand, in response to the absence of an ID whose similarity is greater than the similarity threshold, the facial recognition device 100 determines that the same person does not exist, and newly assigns new person information and new facial information as new information and registers them as a comprehensive ID in the face-person matching table (S111).

[0143] Therefore, according to an embodiment of the present disclosure, before extracting person information and facial information, it is determined whether the person has been registered in the facial person matching table by using the person ID and facial ID assigned to the person area and the facial area, and in response to the fact that the person has been registered, the face can be quickly identified without extracting person information and facial information.

[0144] In addition, according to an embodiment of the present disclosure, in response to a situation where the person has not yet been registered in the facial person matching table using the person ID and face ID previously assigned to the person area and the face area, whether the person information and the face information are information about the same person is determined by extracting the person information and the face information, and in response to the person information and the face information being information about the same person, even if the person has not yet been registered in the facial person matching table, it is determined again whether the person has been registered in the facial person matching table by performing a similarity comparison between the facial expression symbol and the person expression symbol.

[0145] Therefore, in response to determining through similarity that there is an ID matching the new facial information and the new person information among the persons registered in the face-person matching table, the facial recognition accuracy can be further improved by identifying the person (face) corresponding to the ID.

[0146] Steps S107 to S112 describe an implementation method in which the facial region and the person region are not registered in the facial person matching table (FPMT), but even if the facial region and the person region are registered in the facial person matching table (FPMT), the face can be recognized through the above steps S107 to S112 in response to the situation where the facial region and the person region are not temporarily extracted.

[0147] Therefore, when facial recognition is performed using facial information or person information, tracking may be lost in response to the loss of facial information or person information, but according to an embodiment of the present disclosure, even if one of the facial information and person information is lost, continuous tracking can be performed by performing facial recognition using both facial information and person information, thereby improving facial recognition accuracy.

[0148] Furthermore, according to the present embodiment, in response to the comparison similarity, the face re-recognition performance can be improved by adjusting the face weight and the person weight according to their circumstances.

[0149] Furthermore, according to an embodiment of the present disclosure, facial recognition performance may be improved by continuously updating a face person matching table (FPMT).

[0150] Furthermore, according to an embodiment of the present disclosure, in response to a situation where a detector detects a face region and a person region, the positions of the face region and the person region can be grasped, thereby improving the accuracy of determining whether they indicate the same person.

[0151] Fig. 9 An example computing system is shown.

[0152] refer to Fig. 9 The computing system 1000 includes at least one processor 1100 , a storage 1300 , a user interface input device 1400 , a user interface output device 1500 , a memory (ie, storage) 1600 , and a network interface 1700 connected via a bus 1200 .

[0153] The processor 1100 may be a central processing unit (CPU) or a semiconductor device that performs processing on commands stored in the storage 1300 and / or the memory 1600. The storage 1300 and the memory 1600 may include various types of volatile or non-volatile storage media. For example, the storage 1300 may include a read-only memory (ROM) 1310 and a random access memory (RAM) 1320.

[0154] Thus, the steps of the method or algorithm described in conjunction with the exemplary embodiments disclosed herein may be directly implemented by hardware, software modules, or a combination of both, and executed by the processor 1100. The software module may reside in a storage medium (i.e., storage 1300 and / or memory 1600), such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, and CD-ROM.

[0155] An exemplary storage medium is coupled to the processor 1100, and the processor 1100 can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium can be integrated with the processor 1100. The processor and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a user terminal. Alternatively, the processor and the storage medium can reside in a user terminal as separate components.

[0156] The above description merely illustrates the technical concept of the embodiments of the present disclosure, and those skilled in the art to which the present disclosure belongs may make various modifications and changes without departing from the essential features of the embodiments of the present disclosure.

[0157] Therefore, the exemplary embodiments disclosed in the present disclosure are not intended to limit the technical concepts of the embodiments of the present disclosure, but to explain them, and the scope of the technical concepts of the embodiments of the present disclosure is not limited to these exemplary embodiments. The protection scope of the embodiments of the present disclosure should be interpreted by the attached claims, and all technical concepts within the equivalent scope should be interpreted as included in the scope of the embodiments of the present disclosure.

Claims

1. A facial recognition device, include: a processor configured to extract facial information and person information from the image data and recognize a face using the facial information and the person information; as well as A memory is configured to store data and algorithms driven by the processor.

2. The device according to claim 1, in, The processor is configured to detect a person region and a face region from the image data, create a first bounding box in the person region and assign a person ID for tracking, and create a second bounding box in the face region and assign a face ID for tracking.

3. The device according to claim 2, in, The processor is configured to determine whether the person ID and the face ID are registered in a face-person matching table.

4. The device according to claim 3, in: The memory is configured to store the face-person matching table; and The face-person matching table is configured to store a face ID, a face expression symbol, a person ID, a person expression symbol, and a comprehensive ID.

5. The device according to claim 3, in, In response to the person ID and the face ID being registered in the face-person matching table, the processor is configured to recognize the face using the ID registered in the face-person matching table.

6. The device according to claim 5, in, The processor is configured to: In response to a situation where both a face region and a person region are detected in a first frame of the image data but the face region is not detected in a second frame of the image data, querying whether a person ID of the person region detected in the second frame is registered in the face-person matching table; as well as In response to a situation in which a person ID of the person area detected in the second frame is registered in the face-person matching table, a face having a corresponding ID is identified.

7. The device according to claim 6, in, The processor is configured to: assigning a new face ID to the face area detected again in a third frame of the image data; and The face ID registered in the face-person matching table is updated to the new face ID.

8. The device according to claim 5, in, The processor is configured to: In response to a situation where both a face region and a person region are detected in a first frame of the image data but the person region is not detected in a second frame of the image data, querying whether a face ID of the face region detected in the second frame is registered in the face-person matching table; as well as In response to the face ID of the face area detected in the second frame being registered in the face person matching table, a face having a corresponding ID is identified.

9. The device according to claim 8, in, The processor is configured to: assigning a new person ID to the person region detected again in the third frame of the image data; and The person ID registered in the face-person matching table is updated to the new person ID.

10. A facial recognition device, the device include: The processor is configured as: Detecting person regions and facial regions from image data; Creating a first bounding box in the person region and assigning a person ID for tracking; creating a second bounding box in the facial region and assigning a facial ID for tracking; Determining whether the person ID and the face ID are registered in a face-person matching table; as well as In response to determining that the person ID and the face ID are not registered in the face-person matching table, extracting person information from the person area and extracting face information from the face area to determine whether the person information and the face information are information of the same person; as well as A memory is configured to store data and algorithms driven by the processor.

11. The device according to claim 10, in, The processor is configured to determine whether the character information and the facial information belong to the same person by determining whether a pixel distance difference between a facial position extracted from the character information and a facial position extracted from the character area is less than a predetermined first threshold.

12. The device according to claim 11, in, The processor is configured to determine whether the person information and the facial information belong to the same person by determining whether a difference between a distance from a camera to the person area and a distance from the camera to the facial area is less than a predetermined second threshold.

13. The device according to claim 12, in, The processor is configured to determine whether the person information and the facial information belong to the same person by determining whether a size of an overlapping area between the first bounding box of the person area and the second bounding box of the facial area is greater than a predetermined third threshold.

14. The device according to claim 10, in, In response to determining that the person information and the facial information are information of the same person, the processor is configured to perform a similarity comparison between the extracted person information and the extracted facial information and the person information and facial information registered in the face-person matching table.

15. The device according to claim 14, in, The processor is configured to perform the similarity comparison by using the similarity between the character expressors of the character information extracted and the character expressors of the character information registered in the face-character matching table, the similarity between the facial expressors of the facial information extracted and the facial expressors of the facial information registered in the face-character matching table, and a predetermined face weight and a predetermined character weight.

16. The device according to claim 15, in, The processor is configured to increase the face weight and decrease the person weight as a difference between an input time of the extracted facial expression symbol of the facial information and a registration time of the facial expression symbol of the facial information registered in the face-person matching table increases.

17. The device according to claim 15, in, The processor is configured to increase the person weight and decrease the face weight as the distance from the camera to the person area increases.

18. The device according to claim 15, in, The processor is configured to determine whether similarity between the extracted person information and the extracted facial information and the person information and facial information registered in the face-person matching table is greater than a predetermined similarity threshold.

19. The device according to claim 18, in, The processor is configured to: In response to the situation that the similarity between the extracted person information and the extracted facial information and the person information and facial information registered in the face-person matching table is greater than the predetermined similarity threshold, determining that the extracted person information and the extracted facial information and the person information and facial information registered in the face-person matching table are information of the same person; and In response to the situation where the similarity between the extracted person information and the extracted facial information and the person information and the facial information registered in the facial person matching table is equal to or less than the predetermined similarity threshold, the extracted person information and the extracted facial information are newly registered in the facial person matching table.

20. A facial recognition method, the method include: Extracting facial information and person information from image data; as well as A face is recognized using the facial information, the person information, and information stored in a face-person matching table and accessed by a processor.