Customer identification method and device for identifying same person in store

Abstract

Provided is a customer identification method which monitors a real-time status inside a store and identifies the same person in the store. The customer identification method comprises the steps of: receiving, from an Internet of Things (IoT) device installed in a store, a first face image vector obtained by encoding a face image of a customer detected inside the store; searching for a plurality of second face image vectors corresponding to a preset search range among data stored in advance in a vector database; calculating, as a value, a first potential that for each of the plurality of second face image vectors, the person corresponding to the first face image vector is the same as the person corresponding to the second face image vectors, and selecting, among the plurality of second face image vectors, a face image vector determined to have the highest first potential as a third face image vector on the basis of the calculated value; and determining that the person corresponding to the first face image vector is the same as the person corresponding to the third face image vector when the number of times the third face image vector has been selected as the face image vector determined to have the highest first potential reaches a predetermined first reference value.

Description

Customer identification method and device for identifying the same person within a store

[0001] Cross-citation with related applications

[0002] This application claims the benefit of priority from Korean Patent Application No. 10-2024-0056297, filed April 26, 2024, the entire contents of which are incorporated herein by reference.

[0003] The disclosure relates to a customer identification method and device for identifying the same person within a store.

[0004] In modern society, the importance of identity verification technology is growing rapidly across diverse fields, including retail, security, and marketing. This is because personalizing user experiences and providing tailored services has become a crucial competitive advantage. For example, in services that recommend stores based on a customer's preferences and provide real-time store status updates, identifying the same individual is essential for store analysis. However, when applied offline, this technology can encounter legal and technical constraints. Specifically, under the Personal Information Protection Act, collecting and utilizing facial data without the user's explicit consent can carry significant legal risks. Furthermore, situations requiring real-time processing of large volumes of facial data require high-performance computing resources, necessitating an efficient method for processing facial data and identifying individuals.

[0005] The task to be solved is to provide a customer identification method and device that can efficiently and effectively identify the same person in a store by providing the results of facial recognition performed in offline spaces, including stores, while implementing privacy protection.

[0006] According to one embodiment, a customer identification method for monitoring real-time status inside a store and identifying the same person inside the store may include the steps of: receiving a first face image vector in which a face image of a customer detected inside the store is encoded from an IoT (Internet of Things) device installed in the store; searching for a plurality of second face image vectors corresponding to a preset search range among data previously stored in a vector database; calculating a first probability that a person corresponding to the first face image vector is the same person as a person corresponding to the second face image vector, for each of the plurality of second face image vectors, as a value, and selecting a face image vector determined to have the highest first probability based on the calculated value from among the plurality of second face image vectors as a third face image vector; and determining that the person corresponding to the first face image vector is the same person as the person corresponding to the third face image vector when the number of times the third face image vector is selected as the face image vector determined to have the highest first probability reaches a first reference value.

[0007] In some embodiments, the step of selecting the face image vector determined to have the highest first probability based on the calculated value as the third face image vector may include the step of calculating, for each of the plurality of second face image vectors, a cosine distance with the first face image vector; and the step of selecting, among the plurality of second face image vectors, the face image vector having the lowest cosine distance with the first face image vector as the third face image vector.

[0008] In some embodiments, the step of selecting the face image vector determined to have the highest first probability based on the calculated value as the third face image vector may include the step of calculating, for each of the plurality of second face image vectors, a cosine distance with the first face image vector; and the step of selecting, among the plurality of second face image vectors, a face image vector having the lowest cosine distance with the first face image vector and having a cosine distance less than or equal to a predetermined threshold value as the third face image vector.

[0009] In some embodiments, the customer identification method may further include: obtaining a plurality of fourth face image vectors that are not stored in the vector database and in which a person is not identified; calculating, for each of the plurality of fourth face image vectors, a second likelihood that the person corresponding to the first face image vector is the same person as the person corresponding to the fourth face image vector, and selecting, from among the plurality of fourth face image vectors, a face image vector determined to have the highest second likelihood based on the calculated value as a fifth face image vector; and determining, when the number of times the fifth face image vector is selected as the face image vector determined to have the highest second likelihood reaches a second reference value, that the person corresponding to the first face image vector is the same person as the person corresponding to the fifth face image vector.

[0010] In some embodiments, the customer identification method may further include the step of counting the number of times that the person corresponding to the first face image vector is determined to be the same person as the person corresponding to the fifth face image vector; and, if the counted value reaches a predetermined third reference value, the step of assigning a new identifier to the first face image vector and storing the first face image vector together with the new identifier in the vector database.

[0011] In some embodiments, the second reference value may be set to a value greater than the first reference value.

[0012] In some embodiments, the customer identification method may further include a step of determining, based on a predetermined criterion, that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector, if the number of times the third face image vector is selected as the face image vector determined to have the highest first probability does not reach the first reference value.

[0013] In some embodiments, the step of determining that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector may include the step of comparing a first value obtained by multiplying a value representing the first possibility by a predetermined constant with a second value representing the second possibility; and the step of determining that the person corresponding to the first face image vector is the same person as the person corresponding to the third face image vector when the first value is greater than the second value.

[0014] In some embodiments, the method for customer identification, wherein the step of determining that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector may further include the step of determining that the person corresponding to the first face image vector is the same person as the person corresponding to the fifth face image vector if the first value is not greater than the second value.

[0015] In some embodiments, the customer identification method further comprises a step of adding or updating a new face image vector or a previously stored face image vector to the vector database, wherein the step of adding or updating may include a step of calculating an average of a plurality of candidate face image vectors based on a cosine distance; and a step of adding or updating a face image vector selected in order of a short distance based on a candidate face image vector corresponding to the average to the vector database.

[0016] According to one embodiment, a customer identification device is a customer identification device that monitors the real-time status inside a store and identifies the same person inside the store by executing a program code loaded on one or more memory devices through one or more processors, wherein the program code is executed to receive a first face image vector in which a face image of a customer detected inside the store is encoded from an IoT (Internet of Things) device installed in the store, search for a plurality of second face image vectors corresponding to a preset search range among data previously stored in a vector database, and, for each of the plurality of second face image vectors, calculate a first probability that a person corresponding to the first face image vector is the same person as the person corresponding to the second face image vector as a value, and, among the plurality of second face image vectors, select a face image vector determined to have the highest first probability based on the calculated value as a third face image vector, and, when the number of times the third face image vector is selected as the face image vector determined to have the highest first probability reaches a first reference value, the person corresponding to the first face image vector is selected as the same person as the person corresponding to the third face image vector. It can be determined that they are the same person.

[0017] In some embodiments, selecting the face image vector determined to have the highest first probability based on the calculated value as the third face image vector may include calculating a cosine distance with respect to the first face image vector for each of the plurality of second face image vectors, and selecting, among the plurality of second face image vectors, the face image vector having the lowest cosine distance with respect to the first face image vector as the third face image vector.

[0018] In some embodiments, selecting the face image vector determined to have the highest first probability based on the calculated value as the third face image vector may include calculating a cosine distance with respect to each of the plurality of second face image vectors and selecting, among the plurality of second face image vectors, a face image vector having the lowest cosine distance with respect to the first face image vector and having a cosine distance less than or equal to a predetermined threshold value as the third face image vector.

[0019] In some embodiments, the program code is executed to obtain a plurality of fourth face image vectors that are not stored in the vector database and in which the person is not identified, and for each of the plurality of fourth face image vectors, calculate as a value a second likelihood that the person corresponding to the first face image vector is the same person as the person corresponding to the fourth face image vector, and select, from among the plurality of fourth face image vectors, a face image vector determined to have the highest second likelihood based on the calculated value as a fifth face image vector, and when the number of times the fifth face image vector is selected as the face image vector determined to have the highest second likelihood reaches a second reference value, it is determined that the person corresponding to the first face image vector is the same person as the person corresponding to the fifth face image vector.

[0020] In some embodiments, the program code may be executed to count the number of times a person corresponding to the first face image vector is determined to be the same person as the person corresponding to the fifth face image vector, and when the counted value reaches a predetermined third reference value, assign a new identifier to the first face image vector and store the first face image vector together with the new identifier in the vector database.

[0021] In some embodiments, the second reference value may be set to a value greater than the first reference value.

[0022] In some embodiments, the program code may be executed so that, if the number of times the third face image vector is selected as the face image vector determined to have the highest first probability does not reach the first reference value, it may be determined that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector, based on a predetermined criterion.

[0023] In some embodiments, determining that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector may include comparing a first value obtained by multiplying a value representing the first possibility by a predetermined constant with a second value representing the second possibility, and determining that the person corresponding to the first face image vector is the same person as the person corresponding to the third face image vector if the first value is greater than the second value.

[0024] In some embodiments, determining that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector may further include determining that the person corresponding to the first face image vector is the same person as the person corresponding to the fifth face image vector if the first value is not greater than the second value.

[0025] In some embodiments, the program code is executed to add or update a new face image vector or a previously stored face image vector to the vector database, wherein the adding or updating may include calculating an average of a plurality of candidate face image vectors based on a cosine distance, and adding or updating a face image vector selected in order of a short distance based on a candidate face image vector corresponding to the average to the vector database.

[0026] According to embodiments, it is possible to provide facial recognition results obtained in offline spaces including stores while implementing privacy protection by using only a form in which a facial image is converted into a high-dimensional vector based on only a few unique features included in the face rather than using the entire image of the face detected in the store, and to efficiently and effectively perform same-person identification by utilizing a vector database and based on similarity between facial image vectors.

[0027] FIG. 1 is a block diagram illustrating a store curation system according to one embodiment.

[0028] FIG. 2 is a block diagram illustrating a customer identification device according to one embodiment.

[0029] Figure 3 is a flowchart illustrating a customer identification method according to one embodiment.

[0030] Figure 4 is a flowchart illustrating a customer identification method according to one embodiment.

[0031] FIGS. 5 to 7 are drawings showing an implementation example of a customer identification method and device according to one embodiment.

[0032] FIG. 8 is a drawing showing an example of a screen displayed on a user terminal according to one embodiment.

[0033] Figure 9 is a block diagram illustrating a computing device according to one embodiment.

[0034] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description, and similar parts are designated with similar reference numerals throughout the specification.

[0035] Throughout the specification and claims, whenever a part is referred to as "comprising" a component, this does not exclude other components, but rather includes other components, unless otherwise stated. Terms including ordinal numbers, such as "first," "second," etc., may be used to describe various components, but these components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0036] Terms such as "...part," "...device," and "module" described in the specification may refer to a unit capable of processing at least one function or operation described in the specification, which may be implemented by hardware or a circuit, software, or a combination of hardware or a circuit and software. In addition, at least some of the components or functions of the customer identification method and device for identifying the same person in a store according to the embodiments described below may be implemented by a program or software, and the program or software may be stored on a computer-readable medium.

[0037] FIG. 1 is a block diagram illustrating a store curation system according to one embodiment.

[0038] Referring to FIG. 1, a store curation system (1) according to one embodiment may include a store monitoring device (10, 11, 12), a store curation server (2), and a user terminal (30, 31). The store monitoring device (10, 11, 12), the store curation server (2), and the user terminal (30, 31) may exchange data with each other via a network (40).

[0039] The store monitoring device (10, 11, 12) may be an IoT (Internet of Things) device installed in a store. IoT is a technology that connects objects to the Internet by incorporating sensing and communication functions into them. The store monitoring device (10, 11, 12) may be installed inside a store to monitor the inside of the store and transmit the monitoring results to another device, such as a store curation server (2), via a network (40). The store monitoring device (10, 11, 12) may be equipped with an artificial intelligence model (e.g., a deep learning model) to monitor the inside of the store, and may be installed inside multiple stores to monitor the real-time status inside the store and predict future status. For example, the store monitoring device (10, 11) may be installed inside a store (A) to monitor and predict the status inside the store (A), and the store monitoring device (12) may be installed inside another store (B) to monitor and predict the status inside the other store (B). The monitoring and prediction results that can be obtained from the store monitoring device (10, 11) installed in the store (A) and the monitoring and prediction results that can be obtained from the monitoring device (12) installed in the store (B) can be transmitted to the store curation server (2) via the network (40).

[0040] The store curation server (2) can present stores that match the values ​​that customers consider important when selecting a store to customers through user terminals (30, 31), based on store monitoring data for stores (A, B) collected through store monitoring devices (10, 11, 12). In some embodiments, the store monitoring data collected through store monitoring devices (10, 11, 12) can be stored and managed in a database accessible to the store curation server (2).

[0041] In some embodiments, store monitoring data may include at least one of current status data and predicted data regarding the interior of the store. The current status data may include information regarding the interior of the store collected by the store monitoring device (10, 11, 12) using sensors such as cameras and microphones and artificial intelligence technologies such as object detection, pose estimation, and semantic segmentation. Specifically, the current status data may include data regarding at least one of table or seat arrangement status, table or seat occupancy status, noise status, status of music played in the store, status of in-store music playback history, gender status of visiting customers, age status of visiting customers, behavior status of visiting customers, length of stay status of visiting customers, order status by menu, and interior status. Meanwhile, the predicted data may include data inferred, derived, or predicted based on information regarding the interior of the store collected using sensors and an artificial intelligence inference model, or data estimated from accumulated current status data. Specifically, the predictive data may include data regarding at least one of, for example, prediction of changes in table or seat occupancy, prediction of seating atmosphere, prediction of waiting time, prediction of noise changes, prediction of music to be played in the store, prediction of gender of visiting customers, prediction of age group of visiting customers, prediction of behavioral patterns of visiting customers, prediction of purpose of visit, prediction of menu ranking, prediction of recommended menu, sales prediction, inventory prediction, and prediction of atmosphere inside the store.

[0042] For example, sales prediction data can be estimated from status data on the order status by menu, and prediction data on visitor behavior patterns (e.g., 30% computer work, 20% conversation, 50% rest, etc.), visit purpose prediction, or waiting time prediction can be inferred from status data on visitor behavior and visitor length of stay. As another example, prediction data on seat atmosphere prediction, visitor gender prediction, or visitor age prediction can be inferred from status data on visitor gender or visitor age. As yet another example, prediction data on noise change prediction, prediction of music to be played in the store, or prediction of the atmosphere inside the store can be inferred from status data on noise status, music played in the store, or the history of music playback in the store.

[0043] In some embodiments, the current status data and the predicted data may be implemented so that they are generated in the store monitoring device (10, 11, 12) and only transmitted to the store curation server (2), and in some other embodiments, some of the current status data and the predicted data may be generated in the store monitoring device (10, 11, 12) and transmitted to the store curation server (2), while other parts of the predicted data may be implemented so that they are generated by the store curation server (2) from the store monitoring data transmitted to the store curation server (2).

[0044] In some embodiments, store monitoring data may be implemented as text data following a certain format. For example, store monitoring data may be organized as "3 people in table 1, 1 person in table 3, 2 lattes, 3 iced Americanos, 3 women, 2 men," and abstracted into text data with a certain format, such as JSON.

[0045] The user terminal (30, 31) can receive a curated store list from the store curation server (2) and provide a store curation service to the customer. To this end, the user terminal (30, 31) can provide customer data related to the customer using the user terminal (30, 31) to the store curation server (2), so that the store curation server (2) can use the customer data together with the store monitoring data when providing the store curation service. Accordingly, the store curation server (2) can analyze the correlation between the store monitoring data and the customer data, and can discover and recommend stores that can actually provide customer satisfaction, rather than simply selecting stores close to the current location or famous franchise stores.

[0046] In some embodiments, the customer data may include data regarding at least one of customer gender, customer age, customer location, service usage history, customer tastes, preferred menu, preferred music genre, store visit history, and predicted purpose of visit.

[0047] In some embodiments, the store curation server (2) and user terminals (30, 31) may be computing devices such as smartphones, tablet computers, wearable devices, laptop computers, desktop computers, etc. Users may utilize the store curation service through applications running on the user terminals (30, 31). The network (40) may include a wireless network such as a Wi-Fi network, a Bluetooth network, a cellular network, etc., a wired network such as a Local Area Network (LAN), or a network in the form of a mixture of wireless and wired networks.

[0048] In some embodiments, store monitoring data, including current status data and predicted data, may be anonymized within the store monitoring device (10, 11, 12). That is, the store monitoring data generated by the store monitoring device (10, 11, 12) may be anonymized, for example, during the process of generating the various data described above, or after generation is complete, so that data that could identify individuals is not transmitted to the store curation server (2). As an example implementation, if data on gender and number of visitors are extracted from captured data of visitors inside the store, only the data on gender and number of visitors may be transmitted to the store curation server (2), and the captured data may be destroyed within the store monitoring device (10, 11, 12). Accordingly, it is possible to understand and predict the current status of the store while protecting personal information.

[0049] In particular, the store monitoring device (10, 11, 12) can perform face detection, face alignment, and face encoding to implement same-person identification within the store while implementing privacy protection.

[0050] The store monitoring device (10, 11, 12) can perform face detection using a face detection algorithm that can consistently detect faces despite factors such as various backgrounds, lighting conditions, facial orientation, changes in facial expressions, and partial occlusion. In some embodiments, the store monitoring device (10, 11, 12) may adopt a face detection algorithm based on a deep learning model based on a convolutional neural network (CNN). In particular, high-performance computing resources are required to process a large amount of face data in real time, but there are limitations such as cost and technical complexity in deploying a store monitoring device comprised of high-performance computing devices in each store. Therefore, a lightweight store monitoring device that guarantees processing speed is required. To this end, the store monitoring device (10, 11, 12) may adopt an algorithm based on a one-stage detection model as a face detection algorithm. Single-step detection models directly identify faces within an image in a single step. Because they combine the extraction and classification of candidate regions into a single process, they offer fast processing speeds while being relatively simple to implement and optimize. Furthermore, because they determine face locations by processing the entire image only once, they consume minimal computing resources and are suitable for real-time environments. Examples of single-step detection model-based methods include YOLO (You Only Look Once) and SSD (Single Shot Multibox Detector).

[0051] Facial alignment can be a crucial step for improving the accuracy of person identification. This is because model accuracy improves when faces are aligned frontally and vertically. Therefore, if there is a process for aligning faces collected offline to the front, the accuracy of person identification will increase. This process can be performed by detecting facial landmarks and performing an affine transformation to align them.

[0052] An affine transformation can be a linear transformation based on mathematical operations that map points in an image to new locations in two-dimensional or three-dimensional space. In two-dimensional space, an affine transformation can be expressed as the following operations:

[0053] x' = ax + by + t x

[0054] y' = cx + dy + t y

[0055] Here, (x, y) are the coordinates of the original image, (x', y') are the coordinates of the transformed image, a, b, c, d are the elements of the transformation matrix, and (t x , t y ) represents the translation vector.

[0056] The store monitoring device (10, 11, 12) can perform face alignment based on affine transformation as follows. The store monitoring device (10, 11, 12) can first detect facial landmarks (main feature points such as eyes, nose, and mouth). These landmarks can be used as reference points for transformation. Next, the store monitoring device (10, 11, 12) can calculate an affine transformation matrix based on the relationship between the detected landmarks and the target landmark (standardized facial pose). Finally, the store monitoring device (10, 11, 12) can map each pixel of the original image to a new position using the calculated matrix. Accordingly, rotation and scaling are performed simultaneously to adjust the facial image to a standardized pose.

[0057] The store monitoring device (10, 11, 12) can perform face encoding, which analyzes the image of the detected face and converts the unique features of the face into a numerical form. For example, the store monitoring device (10, 11, 12) can convert the image of the face into a high-dimensional vector based on a deep learning model. In the embodiments described herein, the high-dimensional vector thus converted can be transmitted to the store curation server (2) and used to determine whether or not it matches in identifying the same person. In addition, the store monitoring device (10, 11, 12) can implement privacy protection because it transmits an encoded vector instead of directly transmitting the face image to the store curation server (2).

[0058] FIG. 2 is a block diagram illustrating a customer identification device according to one embodiment.

[0059] Referring to FIG. 2, a customer identification device (20) according to one embodiment can execute program code loaded on one or more memory devices via one or more processors. For example, the customer identification device (20) can be implemented as a computing device (50) as described below with reference to FIG. 9 . In this case, one or more processors may correspond to the processor (510) of the computing device (50), and one or more memory devices may correspond to the memory (520) of the computing device (50). The program code, executed by one or more processors, can monitor the real-time status within the store and identify the same person within the store. In this specification, the term "module" is used to logically distinguish these functions performed by the program code.

[0060] The customer identification device (20) may include a facial image vector receiving module (200), a vector database access module (201), an identical person possibility judgment module (202), and an identical person judgment module (203) to identify the same person within the store.

[0061] The face image vector receiving module (200) can receive, from an IoT device installed in the store, a first face image vector (FV1) encoded with a face image of a customer detected inside the store. That is, the face image vector receiving module (200) does not receive the entire image of the face detected in the store, but receives a form in which the face image is converted into a high-dimensional vector based only on a few unique features included in the face, so that personal information protection can be implemented. Here, the IoT device installed in the store may be the store monitoring device (10) described above with reference to FIG. 1, and the face image vector receiving module (200) can receive the first face image vector (FV1) via the network (40) of FIG. 1. The first face image vector (FV1) may be a vector in which a face is detected and encoded by the store monitoring device (10), and may be a face image vector whose face has not yet been identified.

[0062] The vector database access module (201) can search for a plurality of second face image vectors (FV2) corresponding to a preset search range among data previously stored in the vector database (21). The plurality of second face image vectors (FV2) may be face image vectors that identify a person's face. However, the customer identification device (20) according to one embodiment does not process information about identity such as customer name, gender, and age corresponding to the face, but can only process whether a certain face image vector is identical to another face image vector. For this purpose, each of the plurality of second face image vectors (FV2) may be assigned an identifier consisting of a string that does not include information about identity. Accordingly, among the plurality of second face image vectors (FV2) assigned with identifiers, the second face image vectors (FV2) having the same identifier can be understood as vectors converted from the face of the same person.

[0063] The preset search range here may be the range of all data stored in the vector database (21) regardless of store distinction, or may be the range of data corresponding to a specific store or multiple stores among all data stored in the vector database (21).

[0064] In some embodiments, the preset search range may be changed so that, in the operation of the same person possibility determination module (202) described below, if a same person candidate is not derived from a certain search range, a same person candidate can be derived from another search range.

[0065] Specifically, the preset search range may include a first search range set according to a first criterion, and a second search range set according to a second criterion different from the first criterion. The same person possibility determination module (202) attempts to derive a same person candidate using the second face image vector (FV2) searched by the vector database access module (201) in the first search range, and if successful, the same person determination operation of the same person determination module (203) may be performed. In contrast, if the derivation of a same person candidate fails in the first search range, the vector database access module (201) re-searches the second face image vector (FV2) in the second search range, and the same person possibility determination module (202) attempts to derive a same person candidate using the second face image vector (FV2) re-searched in the second search range, and if successful, the same person determination operation of the same person determination module (203) may be performed.

[0066] For example, the vector database access module (201) may search for a second face image vector (FV2) in a first search range where the visit time is set to a range within the past week, and the same person possibility determination module (202) may attempt to derive a same person candidate using the second face image vector (FV2) searched in the first search range. If successful, the same person determination operation of the same person determination module (203) may then be performed.

[0067] If unsuccessful, the vector database access module (201) searches for a second face image vector (FV2) in a second search range where the gender is set to male or female, and the same person possibility determination module (202) may attempt to derive a same person candidate using the second face image vector (FV2) searched in the second search range. If successful, the same person determination operation of the same person determination module (203) may then be performed.

[0068] In case of failure, the vector database access module (201) searches for a second face image vector (FV2) in a third search range set to a range of a predetermined distance from the current store, and the same person possibility determination module (202) may attempt to derive a same person candidate using the second face image vector (FV2) searched in the third search range. In case of success, the same person determination operation of the same person determination module (203) may then be performed.

[0069] In case of failure, the vector database access module (201) searches for a second face image vector (FV2) in a fourth search range where the visit time is set to a range within the last month, and the same person possibility determination module (202) may attempt to derive a same person candidate using the second face image vector (FV2) searched in the fourth search range. In case of success, the same person determination operation of the same person determination module (203) may then be performed.

[0070] Meanwhile, in some embodiments, the customer identification device (20) may further include a cache and may utilize both the cache and the vector database (21).

[0071] Specifically, the preset search range may include a first search range set according to a first criterion, and a second search range set according to a second criterion different from the first criterion. The same person possibility determination module (202) attempts to derive a same person candidate using the second face image vector (FV2) searched for in the first search range from the cache, and if successful, the same person determination operation of the same person determination module (203) may be performed. In contrast, if the derivation of a same person candidate fails in the first search range, the vector database access module (201) searches for a second face image vector (FV2) in the vector database (21) as a second search range, and the same person possibility determination module (202) attempts to derive a same person candidate using the second face image vector (FV2) searched for in the second search range of the vector database (21), and if successful, the same person determination operation of the same person determination module (203) may be performed.

[0072] For example, the second face image vector (FV2) may be searched in the cache with a first search range in which the visit time is set to within the last week, and the same-person possibility determination module (202) may attempt to derive a same-person candidate using the second face image vector (FV2) searched in the cache. If successful, the same-person determination operation of the same-person determination module (203) may then be performed. If unsuccessful, the second face image vector (FV2) may be searched in the vector database (210) with a second search range in which the gender is set to male or female, and the same-person possibility determination module (202) may attempt to derive a same-person candidate using the second face image vector (FV2) searched in the vector database (210). If successful, the same-person determination operation of the same-person determination module (203) may then be performed.

[0073] The vector database (21) may be a database specialized in storing, retrieving, and processing high-dimensional vector data. Encoded facial encoding values ​​in face recognition take the form of high-dimensional vectors, and a database structure capable of efficiently managing and quickly accessing these vectors is essential. The vector database (21) provides an efficient indexing mechanism for data points in a high-dimensional space, ensuring fast search times for similarity searches between vectors, and can search for the most similar vector or vector set using a similarity measurement metric such as Euclidean distance or cosine similarity. Furthermore, the vector database (21) may apply dimensionality reduction techniques to alleviate the curse of dimensionality of high-dimensional data, which refers to a problem that occurs as the dimensionality of data increases. For example, algorithms such as PCA, t-SNE, and UMAP may be used for this. Furthermore, the vector database (21) is designed to be scalable to large-scale data sets, and can operate without performance degradation even in distributed systems.

[0074] By storing the facial encoding values ​​in a vector database, the high-dimensional indexing capabilities of the vector database can be utilized during the person identification process, thereby improving the search speed for the face encoding data set, and thus enabling real-time face recognition and person identification. Furthermore, similarity search can be used to find the most accurate match between the input face vector and the existing vectors in the database, thereby reducing false positives and improving the reliability of the overall system. Furthermore, by applying techniques such as dimensionality reduction for person identification, it is possible to expect advantageous effects such as efficient use of storage space and reduced computational burden during search, scalability that allows for easy expansion of the system capacity as the user base expands, and the ability to distribute data processing by operating the vector database in a large-scale distributed system, thereby distributing the load on the entire network.

[0075] The identical person possibility judgment module (202) can calculate, for each of a plurality of second face image vectors (FV2), a first possibility that the person corresponding to the first face image vector (FV1) is the same person as the person corresponding to the second face image vector (FV2). Then, the identical person possibility judgment module (202) can select, from among the plurality of second face image vectors (FV2), a face image vector determined to have the highest first possibility based on the calculated value as a third face image vector. That is, the person corresponding to the third face image vector selected by the identical person possibility judgment module (202) can be treated as an identical person candidate who is likely to be the same person as the person corresponding to the first face image vector (FV1).

[0076] In some embodiments, the same-person likelihood determination module (202) may calculate a cosine distance between each of a plurality of second face image vectors (FV2) and the first face image vector (FV1), and may take the cosine distance value as the value of the first likelihood. In addition, the same-person likelihood determination module (202) may select, among the plurality of second face image vectors (FV2), the face image vector having the lowest cosine distance from the first face image vector (FV1) as the third face image vector.

[0077] In some other embodiments, the same-person likelihood determination module (202) may calculate a cosine distance between each of a plurality of second face image vectors (FV2) and the first face image vector (FV1), and may take the cosine distance value as the value of the first likelihood. In addition, the same-person likelihood determination module (202) may select, among the plurality of second face image vectors (FV2), a face image vector having the lowest cosine distance from the first face image vector (FV1) and having a cosine distance equal to or less than a predetermined value as the third face image vector.

[0078] The same person determination module (203) can determine that the person corresponding to the first face image vector (FV1) is the same person as the person corresponding to the third face image vector if the number of times the third face image vector is selected as the face image vector determined to have the highest first possibility reaches a first reference value. That is, when the same person possibility determination module (202) calculates the similarity with the first face image vector (FV1) for each of the plurality of second face image vectors (FV2) and repeats the process of selecting the face image vector with the highest similarity with the first face image vector (FV1) from among the plurality of second face image vectors (FV2), if the number of times a specific face image vector is selected from among the plurality of second face image vectors (FV2) reaches a predetermined n value (n is an integer greater than or equal to 2), the person corresponding to the specific face image vector can be determined to be the same person as the person corresponding to the first face image vector (FV1).

[0079] Thereafter, the same person determination module (203) can assign the same identifier to the first face image vector (FV1) as the identifier assigned to the third face image vector, and add the first face image vector (FV1) to the vector database (21) through the vector database access module (201).

[0080] According to this embodiment, by using only a form in which a face image is converted into a high-dimensional vector based on only a few unique features included in the face rather than using the entire image of the face detected in the store, it is possible to provide the results of face recognition performed in an offline space including a store while implementing privacy protection, and to efficiently and effectively perform same-person identification by utilizing a vector database and based on similarity between face image vectors.

[0081] Meanwhile, the face image vector receiving module (200) may receive a first face image vector (FV1) encoded with a face image of a customer detected inside the store from an IoT device installed in the store, and may obtain a plurality of fourth face image vectors (FV3) not stored in the vector database (21) from the vector temporary storage module (22). That is, the plurality of fourth face image vectors (FV3) may correspond to face image vectors in which a person is not identified.

[0082] The same person possibility judgment module (202) can calculate, for each of the plurality of fourth face image vectors (FV3), a second possibility that the person corresponding to the first face image vector (FV1) is the same person as the person corresponding to the fourth face image vector (FV3). Then, the same person possibility judgment module (202) can select, from among the plurality of fourth face image vectors (FV3), a face image vector determined to have the highest second possibility based on the calculated value as the fifth face image vector. In this regard, in order to determine the second possibility, a method may be used in which the face image vector having the lowest cosine distance is selected as the fifth face image vector as described above in relation to the first possibility, or the face image vector having the lowest cosine distance and a cosine distance equal to or less than a predetermined value is selected as the fifth face image vector.

[0083] The same person determination module (203) can determine that the person corresponding to the first face image vector (FV1) is the same person as the person corresponding to the fifth face image vector if the number of times the fifth face image vector is selected as the face image vector determined to have the highest second possibility reaches a second reference value. That is, when the same person possibility determination module (202) calculates the similarity with the first face image vector (FV1) for each of the plurality of fourth face image vectors (FV3) and repeats the process of selecting the face image vector with the highest similarity with the first face image vector (FV1) from among the plurality of fourth face image vectors (FV3), if the number of times a specific face image vector is selected from among the plurality of fourth face image vectors (FV3) reaches a predetermined m value (n is an integer greater than or equal to 2), the person corresponding to the specific face image vector can be determined to be the same person as the person corresponding to the first face image vector (FV1).

[0084] In some embodiments, the second reference value may be set to a value greater than the first reference value. That is, the m value for determining the number of times a specific face image vector is selected from among the plurality of fourth face image vectors (FV3) may be set to a value greater than the n value for determining the number of times a specific face image vector is selected from among the plurality of second face image vectors (FV2). By setting the first and second reference values ​​in this way, in an offline situation, the number of face image vectors that are not yet identified and not stored in the vector database (21) is greater than the number of face image vectors that are already identified and stored in the vector database (21), thereby improving the performance of identifying the same person by conservatively setting the first reference value more than the second reference value.

[0085] In some embodiments, the same-person determination module (203) may count the number of times that the person corresponding to the first face image vector (FV1) is determined to be the same person as the person corresponding to the fifth face image vector, and if the counted value reaches a predetermined third reference value (e.g., 5), assign a new identifier to the first face image vector (FV1) and store the first face image vector (FV1) together with the new identifier in the vector database (21). Alternatively, if the counted value does not reach the third reference value, the first face image vector (FV1) may be treated as noise.

[0086] Meanwhile, if the number of times the third face image vector is selected as the face image vector determined to have the highest first possibility does not reach the first reference value, the same person determination module (203) can determine, based on a predetermined criterion, that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector (FV1).

[0087] In some embodiments, the same-person determination module (203) may compare a first value, which is obtained by multiplying a value representing a first possibility by a predetermined constant (e.g., 2), with a second value representing a second possibility, and if the first value is greater than the second value, the same-person determination module (203) may determine that the person corresponding to the first facial image vector (FV1) is the same person as the person corresponding to the third facial image vector. Alternatively, if the first value is not greater than the second value, the same-person determination module (203) may determine that the person corresponding to the first facial image vector (FV1) is the same person as the person corresponding to the fifth facial image vector.

[0088] In some embodiments, the same-person determination module (203) may add or update a new face image vector or a previously stored face image vector to the vector database (21). In some embodiments, the same-person determination module (203) may calculate an average of a plurality of candidate face image vectors based on a cosine distance, and add or update a face image vector selected in order of a short distance based on a candidate face image vector corresponding to the average to the vector database.

[0089] Figure 3 is a flowchart illustrating a customer identification method according to one embodiment.

[0090] Referring to FIG. 3, a customer identification method according to an embodiment may include a step (S301) of receiving a first face image vector in which a face image of a customer detected inside a store is encoded, a step (S302) of searching for a plurality of second face image vectors corresponding to a preset search range among data previously stored in a vector database, a step (S303) of calculating a first likelihood that a person corresponding to the first face image vector is the same person as the person corresponding to the second face image vector, for each of the plurality of second face image vectors, as a value, a step (S304) of selecting a face image vector determined to have the highest first likelihood based on the calculated value among the plurality of second face image vectors as a third face image vector, and a step (S305) of determining whether the number of times the third face image vector is selected as the face image vector determined to have the highest first likelihood reaches a preset first reference value.

[0091] If it is determined that the first reference value has been reached (S305, Y), the method may proceed to a step (S306) of determining that the person corresponding to the first face image vector is the same person as the person corresponding to the third face image vector. Alternatively, if it is determined that the first reference value has not been reached (S305, N), the method may proceed to a step (S307) of determining that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector, based on a predetermined criterion.

[0092] For further details on the above method, reference may be made to the description of the embodiments described herein, and thus redundant descriptions will be omitted.

[0093] Figure 4 is a flowchart illustrating a customer identification method according to one embodiment.

[0094] Referring to FIG. 4, a customer identification method according to an embodiment may include a step (S401) of receiving a first face image vector in which a face image of a customer detected inside a store is encoded, a step (S402) of obtaining a plurality of fourth face image vectors that are not stored in a vector database and in which a person is not identified, a step (S403) of calculating a second possibility that a person corresponding to the first face image vector is the same person as the person corresponding to the fourth face image vector, for each of the plurality of fourth face image vectors, a step (S404) of selecting a face image vector determined to have the highest second possibility based on the calculated value among the plurality of fourth face image vectors as a fifth face image vector, and a step (S405) of determining whether the number of times the fifth face image vector is selected as the face image vector determined to have the highest second possibility reaches a predetermined second reference value.

[0095] If it is determined that the second reference value has been reached (S405, Y), the method may proceed to a step (S406) of determining that the person corresponding to the first face image vector is the same person as the person corresponding to the fifth face image vector.

[0096] For further details on the above method, reference may be made to the description of the embodiments described herein, and thus redundant descriptions will be omitted.

[0097] FIGS. 5 to 7 are drawings showing an implementation example of a customer identification method and device according to one embodiment.

[0098] Referring to FIG. 5, in one implementation example, the face image vector receiving module (200) may receive, from an IoT device installed in the store, a first face image vector encoded with a face image of a customer detected inside the store. Here, the first face image vector may be a vector for a face image of a customer indicated as an "object."

[0099] The vector database access module (201) can search for a plurality of second face image vectors corresponding to a preset search range among data previously stored in the vector database (21). Accordingly, "User 1", "User 2", "User 3", and "User 4" can be searched as a plurality of second face image vectors.

[0100] The same person possibility judgment module (202) can calculate, for each of a plurality of second face image vectors called "User 1", "User 2", "User 3", and "User 4", a first likelihood that the person corresponding to the first face image vector corresponding to the "object" is the same person as the person corresponding to the second face image vector, as a value. As a result, the values ​​of the first likelihood for "User 1", "User 2", "User 3", and "User 4" can be calculated as values ​​corresponding to the cosine distances "0.45", "0.24", "0.57", and "0.81", respectively. The same person possibility judgment module (202) can select, as a third face image vector, the face image vector "User 2", which is determined to have the highest first likelihood because it has the lowest cosine distance, based on the calculated values, from among the plurality of second face image vectors called "User 1", "User 2", "User 3", and "User 4". Alternatively, the identical person possibility judgment module (202) may select, among a plurality of second face image vectors named "User 1", "User 2", "User 3", and "User 4", the face image vector "User 2", which is judged to have the highest first possibility by having the lowest cosine distance and satisfying the predetermined value of 0.38 or less based on the calculated value, as the third face image vector. In some embodiments, a value calculated for similarity judgment based on face detection confidence, or a value calculated for similarity judgment based on face position such as face angle or the size of a face bounding box, may be used instead of the cosine distance.

[0101] Next, referring to FIG. 6, three situations are illustrated. Situation 1 is when the possibility that the "object" is a "specific user" reaches a threshold value, situation 2 is when the possibility that the "object" is a "specific user" does not reach the threshold value, and situation 3 is when the possibility that the "object" is an "unidentified user" reaches the threshold value. Here, the "specific user" may correspond to a facial image vector that has been identified and already stored in the vector database (21), and the "unidentified user" may correspond to a facial image vector that has not yet been identified and has not yet been stored in the vector database (21). Meanwhile, "Step 1" may refer to operations performed by the customer identification method and device according to one embodiment described with reference to FIG. 5.

[0102] As illustrated in situation 1 in FIG. 6, when "Step 1" is repeatedly performed for a "specific user" and reaches a threshold value N (where N is a natural number), that is, when the number of times the third face image vector is selected as the face image vector judged to have the highest first probability reaches a first predetermined criterion value, the "object" can be matched to the "specific user". That is, the person corresponding to the first face image vector can be determined to be the same person as the person corresponding to the third face image vector. That is, situation 1 is finally identified as the same person only when "Step 1" is repeatedly performed over time and a predetermined criterion (threshold value) is reached.

[0103] As illustrated in situation 2 in FIG. 6, if "Step 1" is repeatedly performed for a "specific user" and fails to reach the threshold value N, i.e., if the number of times the third face image vector is selected as the face image vector determined to have the highest first possibility does not reach the first reference value, the possibility of the "specific user" with the highest possibility and the possibility of the "unidentified user" are compared, and a matching can be performed between the "specific user" and the "unidentified user" according to a predetermined criterion. That is, according to a predetermined criterion, it can be determined that either one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector. Referring to Figure 7 together, if the probability of corresponding to "User 1", "User 2", "User 3", and "User 4" is calculated as "5.16", "24.63", "0.78", and "9.21", respectively, and the probability of corresponding to "Unidentified User" is calculated as "32.74", matching to "User 2" can be performed based on a predetermined criterion, i.e., whether the probability of "specific user" * 2 is higher than the probability of being an "Unidentified User". In this case, if the total number of data is too small (e.g., 5), it may be regarded as noise and not matched.

[0104] As illustrated in situation 3 in FIG. 6, if "Step 1" is repeatedly performed for an "unidentified user" and reaches a threshold value N' (where N' is a natural number), that is, if the number of times the fifth face image vector is selected as the face image vector judged to have the second highest possibility reaches a predetermined second reference value, the "object" can be matched to the "unidentified user". That is, the person corresponding to the first face image vector can be judged to be the same person as the person corresponding to the fifth face image vector. That is, in situation 3, if the possibility of being an unidentified person exceeds the predetermined standard, the person can be identified as a new person and a new identifier can be assigned and stored in the vector database (21).

[0105] In some embodiments, the N and N' values ​​may differ, and the N' value may be set to be larger. This is because in offline situations, there will be more unidentified cases than identified cases, and the thresholds for checking for a "specific user" are conservatively set.

[0106] In some embodiments, it is necessary to add a new "anonymous user" to the vector database and update the information of an existing "specific user." Both processes follow similar procedures. First, when adding a new "anonymous user," encodings for a specific user can be collected throughout the day, compared to each other, cosine distances can be calculated, and then the average can be calculated. For example, if 100 encodings are collected for a specific person throughout the day, the average of the cosine distances can be calculated by comparing the 100 encodings. The top 10 encodings with the shortest distances can be selected and stored in the vector database. A short distance indicates that the encoding is most centrally located and best represents the individual. The process for updating a "specific user" is similar. In this case, the 10 encodings stored in the vector database can also be included in the daily data collection, and the top 10 encodings with the shortest distances can be selected and stored again in the vector database.

[0107] FIG. 8 is a drawing showing an example of a screen displayed on a user terminal according to one embodiment.

[0108] Referring to Figure 8, a screen displayed on a user terminal according to one embodiment may include a curated list of stores. For example, the screen may display store information for "Store A" and "Store B." Specifically, information such as a representative image for each store, sales performance, store location, distance to the store, and expected discount amount may be displayed.

[0109] Current and predicted data for a store can be displayed on the screen in at least one of the following forms: text, icons, images, and videos. For example, referring to area (A) of the screen, current data such as "Table occupancy: 36%" and "(approximately 30 dB)" may be displayed in text, while predicted data such as "Quiet" and "Good for chatting" may be displayed in text. Additionally, to the right of area (A), predicted data such as "Expected wait of approximately 12 minutes" may be displayed in text.

[0110] As exemplified above, in processing current status data and predicted data for a store, the embodiments described in this specification enable personal information protection while providing the results of facial recognition performed in the store to efficiently and effectively identify the same person.

[0111] Figure 9 is a block diagram illustrating a computing device according to one embodiment.

[0112] Referring to FIG. 9, a customer identification method and device for identifying the same person in a store according to embodiments can be implemented using a computing device (50).

[0113] The computing device (50) may include at least one of a processor (501), a memory (502), a storage device (503), a display device (504) communicating via a bus (509), a network interface device (505) providing a connection to a network (40) for communication with other entities, and an input / output interface device (506) providing a user input interface or a user output interface. Of course, the computing device (50) may additionally include any electronic device necessary to implement the technical ideas described herein, although not illustrated in FIG. 9.

[0114] The processor (501) may be implemented in various types such as an MCU (Micro Controller Unit), an AP (Application Processor), a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an NPU (Neural Processing Unit), etc., and may be any electronic device that executes a program or command stored in a memory (502) or a storage device (503). In particular, the processor (501) may be configured to implement the functions or methods described above with respect to FIGS. 1 to 8.

[0115] The memory (502) and the storage device (503) may include various types of volatile or non-volatile storage media. For example, the memory (502) may include a read-only memory (ROM) or a random access memory (RAM), and the memory (502) may be located inside or outside the processor (501) and may be connected to the processor (501) through various known means. Meanwhile, examples of the storage device (503) include a hard disk drive (HDD) or a solid state drive (SSD), and the scope of the present invention is not limited to the elements listed above for explanation.

[0116] In some embodiments, at least some components or functions of the customer identification method and device for identifying the same person in a store according to the embodiments may be implemented as a program or software running on a computing device (50), and the program or software may be stored on a computer-readable medium. Specifically, the computer-readable medium according to one embodiment may be a program recorded thereon for causing a computer including a processor (501) that executes a program or instructions stored in a memory (502) or a storage device (503) to execute steps included in the customer identification method and device for identifying the same person in a store according to the embodiments.

[0117] In some embodiments, at least some components or functions of the customer identification method and device for identifying the same person within a store according to the embodiments may be implemented using hardware or circuitry of the computing device (50), or may be implemented as separate hardware or circuitry that can be electrically connected to the computing device (50).

[0118] According to the embodiments described so far, it is possible to provide facial recognition results obtained in offline spaces including stores while implementing privacy protection by using only a form in which the image of the face is converted into a high-dimensional vector based on only a few unique features included in the face rather than using the entire image of the face detected in the store, and to efficiently and effectively perform identification of the same person by utilizing a vector database and based on the similarity between facial image vectors.

[0119] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art to which the present invention pertains using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. A customer identification method that monitors the real-time status inside the store and identifies the same person within the store. A step of receiving a first facial image vector encoded with a facial image of a customer detected inside the store from an IoT (Internet of Things) device installed in the store; A step of searching for a plurality of second face image vectors corresponding to a preset search range among data already stored in a vector database; For each of the plurality of second face image vectors, calculating a first probability that the person corresponding to the first face image vector is the same person as the person corresponding to the second face image vector, and selecting a face image vector among the plurality of second face image vectors, which is determined to have the highest first probability based on the calculated value, as a third face image vector; and A step of determining that a person corresponding to the first face image vector is the same person as the person corresponding to the third face image vector when the number of times the third face image vector is selected as the face image vector determined to have the highest first possibility reaches a first reference value How to identify customers.

2. In paragraph 1, The step of selecting the face image vector judged to have the highest first possibility based on the above calculated value as the third face image vector is as follows: For each of the plurality of second face image vectors, calculating a cosine distance with the first face image vector; and A customer identification method comprising a step of selecting, among the plurality of second face image vectors, a face image vector having the lowest cosine distance from the first face image vector as the third face image vector.

3. In paragraph 1, The step of selecting the face image vector judged to have the highest first possibility based on the above calculated value as the third face image vector is as follows: For each of the plurality of second face image vectors, calculating a cosine distance with the first face image vector; and A customer identification method comprising a step of selecting, among the plurality of second face image vectors, a face image vector having the lowest cosine distance from the first face image vector and having a cosine distance less than or equal to a predetermined threshold value as the third face image vector.

4. In paragraph 1, A step of obtaining a plurality of fourth face image vectors that are not stored in the above vector database and in which the person is not identified; For each of the plurality of fourth face image vectors, calculating a second possibility that the person corresponding to the first face image vector is the same person as the person corresponding to the fourth face image vector, and selecting, among the plurality of fourth face image vectors, a face image vector determined to have the highest second possibility based on the calculated value as a fifth face image vector; and A customer identification method further comprising a step of determining that a person corresponding to the first facial image vector is the same person as the person corresponding to the fifth facial image vector when the number of times the fifth facial image vector is selected as the facial image vector determined to have the highest second possibility reaches a predetermined second reference value.

5. In paragraph 4, A step of counting the number of times that the person corresponding to the first face image vector is determined to be the same person as the person corresponding to the fifth face image vector; and A customer identification method further comprising the step of assigning a new identifier to the first face image vector and storing the first face image vector together with the new identifier in the vector database when the counted value reaches a predetermined third reference value.

6. In paragraph 4, A customer identification method wherein the second reference value is set to a value greater than the first reference value.

7. In paragraph 4, A customer identification method further comprising a step of determining, based on a predetermined criterion, that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector, if the number of times the third face image vector is selected as the face image vector determined to have the highest first possibility does not reach the first reference value.

8. In paragraph 7, The step of determining that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector is, A step of comparing a first value obtained by multiplying a value representing the first possibility by a predetermined constant with a second value representing the second possibility; and A customer identification method, comprising a step of determining that a person corresponding to the first face image vector is the same person as a person corresponding to the third face image vector if the first value is greater than the second value.

9. In paragraph 8, The step of determining that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector is, A customer identification method further comprising a step of determining that a person corresponding to the first facial image vector is the same person as a person corresponding to the fifth facial image vector, if the first value is not greater than the second value.

10. In paragraph 1, Further comprising a step of adding or updating a new face image vector or a previously stored face image vector to the above vector database, The steps to add or update above are: A step of calculating an average value of multiple candidate face image vectors based on a cosine distance; and A customer identification method comprising a step of adding or updating, to the vector database, face image vectors selected in order of shortest distance based on candidate face image vectors corresponding to the above average value.

11. A customer identification device that monitors the real-time status inside the store and identifies the same person inside the store by executing program code loaded in one or more memory devices through one or more processors. The above program code is executed, Receive a first facial image vector encoded with a facial image of a customer detected inside the store from an IoT (Internet of Things) device installed in the store, Among the data already stored in the vector database, multiple second face image vectors corresponding to a preset search range are searched, For each of the plurality of second face image vectors, a first probability that the person corresponding to the first face image vector is the same person as the person corresponding to the second face image vector is calculated as a value, and among the plurality of second face image vectors, a face image vector determined to have the highest first probability based on the calculated value is selected as a third face image vector. If the number of times the third face image vector is selected as the face image vector judged to have the highest first possibility reaches a first predetermined reference value, it is determined that the person corresponding to the first face image vector is the same person as the person corresponding to the third face image vector. Customer identification device.

12. In paragraph 11, Based on the above calculated value, selecting the face image vector judged to have the highest first possibility as the third face image vector is: For each of the plurality of second face image vectors, a cosine distance is calculated with respect to the first face image vector, A customer identification device comprising selecting, among the plurality of second face image vectors, a face image vector having the lowest cosine distance from the first face image vector as the third face image vector.

13. In paragraph 11, Based on the above calculated value, selecting the face image vector judged to have the highest first possibility as the third face image vector is: For each of the plurality of second face image vectors, calculate the cosine distance with the first face image vector, A customer identification device comprising selecting, among the plurality of second face image vectors, a face image vector having the lowest cosine distance from the first face image vector and having a cosine distance less than or equal to a predetermined threshold value as the third face image vector.

14. In paragraph 11, The above program code is executed, Obtaining a plurality of fourth face image vectors that are not stored in the above vector database and whose persons are not identified, For each of the plurality of fourth face image vectors, a second possibility that the person corresponding to the first face image vector is the same person as the person corresponding to the fourth face image vector is calculated as a value, and among the plurality of fourth face image vectors, a face image vector determined to have the highest second possibility based on the calculated value is selected as a fifth face image vector. A customer identification device that determines that the person corresponding to the first facial image vector is the same person as the person corresponding to the fifth facial image vector when the number of times the fifth facial image vector is selected as the facial image vector determined to have the highest second possibility reaches a predetermined second reference value.

15. In paragraph 14, The above program code is executed, Count the number of times that the person corresponding to the first face image vector is determined to be the same person as the person corresponding to the fifth face image vector, A customer identification device that assigns a new identifier to the first face image vector when the counted value reaches a predetermined third reference value, and stores the first face image vector together with the new identifier in the vector database.

16. In paragraph 14, A customer identification device, wherein the second reference value is set to a value greater than the first reference value.

17. In paragraph 14, The above program code is executed, A customer identification device that determines, based on a predetermined criterion, that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector, if the number of times the third face image vector is selected as the face image vector determined to have the highest first possibility does not reach the first reference value.

18. In paragraph 17, It is determined that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector. Comparing a first value that represents the first possibility multiplied by a predetermined constant with a second value that represents the second possibility, A customer identification device, comprising: determining that a person corresponding to the first face image vector is the same person as a person corresponding to the third face image vector when the first value is greater than the second value.

19. In paragraph 18, It is determined that one of the third face image vector and the fifth face image vector is the same person as the person corresponding to the first face image vector. A customer identification device further comprising determining that a person corresponding to the first facial image vector is the same person as a person corresponding to the fifth facial image vector, if the first value is not greater than the second value.

20. In paragraph 11, The above program code is executed, Add or update a new face image vector or a previously stored face image vector to the above vector database, Adding or updating the above, Calculate the average of multiple candidate face image vectors based on the cosine distance, A customer identification device comprising adding or updating a face image vector selected in order of shortest distance based on a candidate face image vector corresponding to the above average value to the vector database.

Images