Electronic device and method for recognizing face of user, and computer-readable storage medium

Abstract

This electronic device may comprise: a memory for storing instructions; and a processor. When executed by the processor, the instructions can cause the electronic device to: acquire an image that includes the face of a user; extract feature values from the acquired image by using a face recognition model to which the weight of a masked auto encoder (MAE) trained for faces including the face is transferred; identify, from among reference embedding vectors stored in the memory, a reference embedding vector corresponding to an embedding vector generated on the basis of the extracted feature values; and recognize the user corresponding to the reference embedding vector identified from the image. The face recognition model can be trained through a loss function calculated on the basis of a reference margin value and a rotation margin value, which is based on at least one rotation angle value indicating the orientation of the face.

Description

Electronic device, method, and computer-readable storage medium for recognizing a user's face

[0001] Embodiments disclosed in this disclosure relate to an electronic device, a method, and a computer-readable storage medium for recognizing a user's face.

[0002] Recently, the proliferation of various types of electronic devices, such as smartphones, tablet PCs, wireless earphones, and / or smartwatches, has increased. These electronic devices can provide functions for user interaction based on human-machine interfaces (HMIs). These electronic devices can provide this functionality through face recognition (FR) services.

[0003] In one embodiment, an electronic device can recognize a user's face. For example, the electronic device may utilize a model for recognizing the face. The electronic device may require a method for accurately recognizing the user's face even when the facial orientation changes.

[0004] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0005] An electronic device may include a memory storing instructions. The electronic device may include a processor operatively coupled to the memory. The instructions, when executed by the processor, may cause the electronic device to obtain an image including a face of a user. The instructions, when executed by the processor, may cause the electronic device to extract feature values ​​from the obtained image using a face recognition model in which weights of a masked auto encoder (MAE) learned for faces including the face are transferred. The instructions, when executed by the processor, may cause the electronic device to identify a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values ​​among reference embedding vectors stored in the memory. The instructions, when executed by the processor, may cause the electronic device to recognize the user corresponding to the reference embedding vector identified from the image. The above face recognition model can be trained through a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating the orientation of the face.

[0006] A method performed by an electronic device may include an operation of acquiring an image including a face of a user. The method may include an operation of extracting feature values ​​from the acquired image using a face recognition model in which weights of a masked auto encoder (MAE) learned for faces including the face are transferred. The method may include an operation of identifying a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values ​​among stored reference embedding vectors. The method may include an operation of recognizing the user corresponding to the reference embedding vector identified from the image. The face recognition model may be trained through a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating an orientation of the face.

[0007] A computer-readable storage medium having one or more programs stored thereon may include instructions that, when executed by a processor of an electronic device, cause an image including a face of a user to be acquired. The one or more programs may include instructions that, when executed by the processor, cause a face recognition model in which weights of a masked auto encoder (MAE) learned for faces including the face are transferred to extract feature values ​​from the acquired image. The one or more programs may include instructions that, when executed by the processor, cause a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values ​​among stored reference embedding vectors. The one or more programs may include instructions that, when executed by the processor, cause a user corresponding to the reference embedding vector identified from the image to be recognized. The above face recognition model can be trained through a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating the orientation of the face.

[0008] A method for training a model for recognizing a user's face may include obtaining rotation angle values ​​representing an orientation of the face within an image of the user. The rotation angle values ​​may include a roll for the face, a pitch for the face, and a yaw for the face. The method may include generating an activation function for a sum of a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw. The method may include calculating a rotation margin value from a product between the generated activation function and an additional margin value. The method may include calculating a loss function based on a sum of the rotation margin value and a reference margin value. The method may include applying the loss function to the model.

[0009] In one embodiment, an electronic device can recognize a user's face. For example, the electronic device can utilize a model for recognizing the face. The electronic device can train the model by considering information about the orientation of the face. Using the trained model, the electronic device can accurately recognize the user's face.

[0010] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0011] FIG. 1 illustrates an example block diagram of an electronic device according to one embodiment.

[0012] FIG. 2 illustrates an example for describing a neural network obtained by an electronic device from a set of parameters stored in a memory, according to one embodiment.

[0013] FIG. 3A illustrates an example of a facial recognition model that an electronic device uses to recognize a user's face, according to one embodiment.

[0014] FIG. 3b illustrates an example of a method for recognizing a user based on a facial recognition model, according to one embodiment.

[0015] FIG. 4 illustrates an example of a method for recognizing a user based on a facial recognition model, according to one embodiment.

[0016] FIG. 5a illustrates an example of a loss function that is used for training a face recognition model according to one embodiment and that changes depending on the direction of the face.

[0017] FIGS. 5b and 5c illustrate examples of a method for recognizing a user's face based on a face recognition model learned using a loss function that changes depending on the direction of the face, according to one embodiment.

[0018] FIG. 6 illustrates an example of an operational flow for a method of recognizing a user's face based on a face recognition model learned using a loss function that changes depending on the direction of the face, according to one embodiment.

[0019] FIG. 7 illustrates an example of a face recognition model having weights transferred from weights of a model learned using an image reconstructed from an image representing a portion of a user's face, according to one embodiment.

[0020] FIG. 8 illustrates examples of an image representing a portion of a user's face, a restored image, and an image representing the user's face, according to one embodiment.

[0021] Electronic devices (or external electronic devices) according to various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, servers, or home appliances. Electronic devices (or external electronic devices) according to embodiments of this document are not limited to the aforementioned devices.

[0022] The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components, and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0023] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0024] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium that can be read by a machine (e.g., an electronic device (101)). For example, a processor of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (e.g., an electromagnetic wave), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0025] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™ or App Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0026] According to various embodiments, each component (e.g., a module or a program) of the described components may include one or more entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0027] FIG. 1 illustrates an example block diagram of an electronic device according to one embodiment.

[0028] Referring to FIG. 1, an electronic device (101) according to one embodiment may include a terminal owned by a user. The terminal may include a personal computer (PC) such as a laptop or desktop, a smartphone, a smartpad, a tablet PC, a smartwatch, and a smart accessory such as a head-mounted device (HMD).

[0029] Referring to FIG. 1, an electronic device (101) according to one embodiment may include at least one of a processor (110), a memory (120), and a camera (150). The processor (110), the memory (120), and the camera (150) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus. The type and / or number of hardware components included in the electronic device (101) are not limited to those illustrated in FIG. 1. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 1. Alternatively, for example, the electronic device (101) may further include other components (e.g., a display, a microphone, a sensor) in addition to the hardware components illustrated in FIG. 1.

[0030] The elements within the memory described below (e.g., layers and / or face recognition model (130)) may be logically separated, but are not limited thereto.

[0031] A processor (110) of an electronic device (101) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (110) may be one or more. For example, the processor (110) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0032] The memory (120) of the electronic device (101) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (110). The memory (120) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0033] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (110) of the electronic device (101) may be stored in the memory (120) of the electronic device (101). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the electronic device (101) and / or the processor (110) may perform at least one of the operations of FIG. 6 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in an electronic device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (120), and that the one or more applications are stored in a format executable by the processor (110) (e.g., a file having an extension specified by the operating system of the electronic device (101). As an example, an application may include a program and / or a library related to a service provided to a user.

[0034] In one embodiment, a set of parameters related to a facial recognition model (130) may be stored in the memory (120) of an electronic device (101). The facial recognition model (130) is a recognition model implemented in software or hardware that imitates the computational ability of a biological system by using a large number of artificial neurons (or nodes). The facial recognition model (130) may perform human cognitive functions or learning processes through the artificial neurons. The parameters related to the facial recognition model (130) may represent, for example, a plurality of nodes included in the facial recognition model (130) and / or weights assigned to connections between the plurality of nodes. The structure of the facial recognition model (130) represented by the set of parameters stored in the memory (120) of the electronic device (101) according to one embodiment will be described later with reference to FIG. 2.

[0035] According to one embodiment, a camera (150) of an electronic device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. A plurality of optical sensors included in the camera (150) may be arranged in the form of a two-dimensional array. The camera (150) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate an image corresponding to light reaching the optical sensors of the two-dimensional grid, and including a plurality of pixels arranged two-dimensionally. For example, photographic data captured using the camera (150) may mean one image acquired from the camera (150). For example, the image may include a user's face. For example, video data captured using a camera (150) may refer to a sequence of multiple images acquired from the camera (150) according to a specified frame rate. An electronic device (101) according to an embodiment may further include a flash light that is positioned toward the direction in which the camera (150) receives light and outputs light in the direction. In FIG. 1, the electronic device (101) is illustrated as including a camera (150), but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may not include a camera (150).

[0036] In addition, although not illustrated in FIG. 1, the electronic device (101) may further include a sensor for acquiring data including the user's face. Alternatively, the electronic device (101) may further include a communication circuit for acquiring an image or data including the user's face from an external electronic device (e.g., a server). Alternatively, the electronic device (101) may further include a display for visually displaying the results of recognition of the user's face. Alternatively, the electronic device (101) may further include a haptic module for physically notifying the results of the recognition.

[0037] FIG. 2 illustrates an example for describing a neural network obtained by an electronic device from a set of parameters stored in a memory, according to one embodiment.

[0038] For example, a neural network may configure a face recognition model (130). Referring to FIG. 2, at least a portion of the face recognition model (130) may include a plurality of layers. For example, the face recognition model (130) may include an input layer (210), one or more hidden layers (220), and an output layer (230). The input layer (210) may receive a vector representing input data (e.g., a vector having elements corresponding to the number of nodes included in the input layer (210). Signals generated from each of the nodes in the input layer (210) generated by the input data may be transmitted from the input layer (210) to the hidden layers (220). The output layer (230) may generate output data of the face recognition model (130) based on one or more signals received from the hidden layers (220). Here, the output data may include, for example, a vector having elements corresponding to the number of nodes included in the output layer (230).

[0039] Referring to FIG. 2, one or more hidden layers (220) may be positioned between the input layer (210) and the output layer (230), and may convert input data transmitted through the input layer (210) into values ​​that are easy to predict. The input layer (210), one or more hidden layers (220), and the output layer (230) may include a plurality of nodes. The one or more hidden layers (220) are not limited to the illustrated feedforward-based topology, and may be, for example, a convolution filter or a fully connected layer in a convolutional neural network (CNN), or various types of filters or layers grouped based on special functions or characteristics. In one embodiment, the one or more hidden layers (220) may be a layer based on a recurrent neural network (RNN) in which the output value is input again to the hidden layer of the current time. For example, the input layer (210), one or more hidden layers (220), and / or the output layer (230) may be some layers of a transformer model. The face recognition model (130) according to one embodiment may include a number of hidden layers (220) to form a deep neural network. Training a deep neural network is called deep learning. Among the nodes of the face recognition model (130), a node included in the hidden layers (220) is referred to as a hidden node.

[0040] Nodes included in the input layer (210) and one or more hidden layers (220) may be connected to each other through connecting lines having connecting weights, and nodes included in the hidden layer and the output layer may also be connected to each other through connecting lines having connecting weights. Tuning and / or training the face recognition model (130) may mean changing the connecting weights between nodes included in each of the layers included in the face recognition model (130) (e.g., the input layer (210), one or more hidden layers (220), and the output layer (230)). Tuning the face recognition model (130) may be performed based on, for example, supervised learning and / or unsupervised learning.

[0041] An electronic device according to one embodiment can tune a facial recognition model (130) based on reinforcement learning in unsupervised learning. For example, the electronic device can change the policy information that the facial recognition model (130) uses to control the agent based on the interaction between the agent and the environment. The policy information is a rule that the electronic device uses to determine the agent's actions within the environment using a neural network. The electronic device can train the neural network based on the interaction between the agent and the environment to change the policy information of the neural network. For example, the policy information can be changed to determine the optimal action and / or sequence of actions for the agent to achieve an obtainable reward and / or goal. In one embodiment, the electronic device can cause the facial recognition model (130) to change the policy information in order to maximize the agent's goal and / or reward through the interaction.

[0042] An electronic device (101) may provide a facial recognition function (or facial recognition service). For example, the electronic device (101) may recognize a user using an image or data containing the user's face. The electronic device (101) may input the image or data into a model for facial recognition (hereinafter, "facial recognition model") and generate identification information (or identifier) ​​indicating the user using the output from the facial recognition model. Based on the identification information, the electronic device (101) may recognize the user.

[0043] The above facial recognition model can extract feature values ​​from an image (or the data) and compare the similarity between a vector (or an embedding vector) generated from the extracted feature values ​​and a reference vector (or a reference embedding vector) stored in the electronic device (101). Based on the result of the comparison, the identification information can be generated. At this time, the facial recognition model can be trained through a loss function that induces (or causes) a set of vectors for the face of the same person to be located (or distributed) close to each other and a set of vectors for the faces of different people to be located (or distributed) far from each other. For example, the loss function can include a softmax loss function, a Euclidean distance based loss function, or an angular based (or cosine margin based) loss function.

[0044] For example, an angle-based loss function may include the ArcFace loss function (hereinafter referred to as ArcFace). For example, ArcFace may be a function modified from the Softmax loss function, as an example of an angle-based loss function. An example of ArcFace may be referred to in the following mathematical equation.

[0045]

[0046] The above L may represent the ArcFace, the N may represent the batch (or mini-batch) size (or the number of samples), the x may represent an input vector, the b may represent a vias vector (or bias), the W may represent a weight matrix for classification between embedding vectors (or identifiers), the y may represent a correct class (or label), and the n may represent the number of correct classes. The correct class may correspond to the number of identifiers corresponding to users classified based on the ArcFace.

[0047] Even if the electronic device (101) uses the facial recognition model learned through the loss function, the facial recognition performance may be reduced depending on the orientation of the face in the image, the brightness of the image, or the degree of occlusion of the face in the image. For example, if the face in the image is not facing straight ahead, but is facing in a direction rotated in a different direction from the straight ahead, the facial recognition performance may be reduced. In addition, for example, if the brightness in the image is excessively dark or bright, the facial recognition performance may be reduced. In addition, if a portion of the face is occluded, such as by a mask or sunglasses, the facial recognition performance may be reduced. In other words, the ArcFace may apply the same margin value to feature values ​​(or vectors) extracted from the face in the input image, without considering the orientation, brightness, or degree of occlusion of the user's face.

[0048] Hereinafter, the electronic device, method, and computer-readable storage medium according to embodiments of the present disclosure may use a face recognition model (130) learned through a loss function (hereinafter, referred to as an angle-aware loss function) determined based on information about the direction of the face. The electronic device, method, and computer-readable storage medium according to embodiments of the present disclosure may improve face recognition performance for an image including a face facing a rotated direction based on the face recognition model (130) using the angle-aware loss function. In addition, the electronic device, method, and computer-readable storage medium according to embodiments of the present disclosure may use a face recognition model (130) initialized through weights transferred from weights of a pre-learned model. The pre-learned model may be learned through an image in which a part of the face is occluded or removed. In addition, the electronic device, method, and computer-readable storage medium according to embodiments of the present disclosure may improve face recognition performance for an image including a part of the face based on the initialized face recognition model (130).

[0049] FIG. 3A illustrates an example of a facial recognition model that an electronic device uses to recognize a user's face, according to one embodiment.

[0050] FIG. 3A illustrates an example of a facial recognition model (130) that an electronic device (101) uses to acquire an image including (or representing) a user's face and to recognize the user from the acquired image. The facial recognition model (130) of FIG. 3A may represent an example of the facial recognition model (130) of FIG. 1 and the facial recognition model (130) of FIG. 2 . For example, the facial recognition model (130) may be referred to as a facial recognition pipeline, a facial recognition system, or a facial recognition system pipeline. Hereinafter, operations performed by the facial recognition model (130) are described, but the embodiments of the present disclosure are not limited thereto. For example, the operations may be performed by the electronic device (101) utilizing the facial recognition model (130) or by the facial recognition model (130) being controlled by the processor (110) of the electronic device (101).

[0051] Referring to FIG. 3A, the face recognition model (130) may include a feature extractor (300). According to one embodiment, the face recognition model (130) may extract (or generate) an embedding vector (302) from an image (301) including a face of a first user using the feature extractor (300). For example, the embedding vector (302) may be referenced as feature values ​​or a query. For example, the size of the image (301) and the size of the embedding vector (302) may be determined based on the feature extractor (300).

[0052] According to one embodiment, the face recognition model (130) can compare the extracted embedding vector (302) with the reference embedding vectors (310). For example, the face recognition model (130) can calculate the similarity between the embedding vector (302) and the reference embedding vectors (310). For example, the face recognition model (130) can identify, based on the result of the calculation (or comparison), a reference embedding vector corresponding to the embedding vector (302) among the reference embedding vectors (310). For example, the reference embedding vector corresponding to the embedding vector (302) can represent a vector (e.g., a first reference embedding vector (311)) having a value most similar to the embedding vector (302) among the reference embedding vectors (310).

[0053] For example, the reference embedding vectors (310) may include a first reference embedding vector (311) indicating a first user, a second reference embedding vector (312) indicating a second user, and a third reference embedding vector (313) indicating a third user. However, the embodiments of the present disclosure are not limited thereto. For example, the number of reference embedding vectors (310) may be changed. The reference embedding vectors (310) may be referenced as a gallery that stores information about the user. For example, the reference embedding vectors (310) may be stored in the memory (120) of the electronic device (101).

[0054] According to one embodiment, the face recognition model (130) may output identification information (303) indicated by the first reference embedding vector (311). For example, the identification information (303) may represent identification information (or identifier) ​​indicating a first user.

[0055] Referring to the above, the electronic device (101) can extract feature values ​​from an input image (301) using a facial recognition model (130), compare an embedding vector (302) generated from the extracted feature values ​​with stored reference embedding vectors, and recognize a specific user based on the result of the comparison. At this time, the feature extractor (300) of the facial recognition model (130) may be in a pre-trained state. For details related to the training of the feature extractor (300), reference may be made to the example (320) of FIG. 3A.

[0056] Referring to FIG. 3A, the feature extractor (300) can be trained using training data (321). For example, the training data (321) can include images of faces of multiple users (or identification information of multiple users). For example, the training data (321) can be referred to as a data set. For example, the feature extractor (300) can be trained based on a loss function (325) based on an output embedding vector (322) and a result embedding vector (323) obtained from each of the images of the training data (321). For example, the loss function (325) can be used to train the feature extractor (300) such that within the embedding space (or the space defined by the embedding vectors), vectors having the same identification information are positioned close together, and vectors having different identification information are positioned far apart. In other words, the feature extractor (300) can be trained based on a loss function (325) so that the output embedding vector (322) for a specific user has a similar value to the result embedding vector (323) representing the specific user. For example, the result embedding vector (323) can be referred to as a one-hot vector. For example, the loss function (325) can be used to train the backbone (or weights, backbone weights) of the feature extractor (300) through backpropagation.

[0057] According to one embodiment, the loss function (325) used for learning the feature extractor (300) may be changed according to the orientation of the user's face. For example, the loss function (325) may be referred to as an angle-aware loss function. For example, the loss function (325) may be defined by a rotation margin value and a reference margin value generated based on angle values ​​(or rotation angle values) indicating the orientation. For example, the reference margin value may be used to distinguish between multiple users that can be identified by the feature extractor (300). For example, the reference margin value may have the same value for multiple users (or multiple images including faces of multiple users).

[0058] Specific details of the loss function (325) generated using the above angle values ​​indicating the direction of the face are described in FIGS. 5A and 6. Specific details of a method for distinguishing (or recognizing) a user's face through a loss function (325) having a margin that changes depending on the direction of the user's face according to embodiments of the present disclosure are described below in FIGS. 5B and 6.

[0059] Although not illustrated in FIG. 3A, according to one embodiment, the backbone (or weights) of the feature extractor (300) may be initialized. For example, the weights of the feature extractor (300) may be initialized based on random weights. Alternatively, for example, the weights of the feature extractor (300) may be initialized based on specific weights for effectively performing face recognition. Specific details related thereto are described below in FIGS. 7 and 8.

[0060] FIG. 3b illustrates an example of a method for recognizing a user based on a facial recognition model, according to one embodiment.

[0061] FIG. 3b illustrates an example (350) of a method for recognizing (or identifying) a user (or a user's face) based on the face recognition model (130) of FIG. 3a.

[0062] Referring to example (360), the electronic device (101) can obtain a first image (361) and a second image (362). In FIG. 3B, for convenience of explanation, an example (360) is illustrated in which the electronic device (101) obtains the first image (361) and the second image (362), which are images, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may also use a first embedding vector (or first identification information) extracted from the first image (361) and a second embedding vector (or second identification information) extracted from the second image (362). For example, the first image (361) may include the face of the first user. For example, the second image (362) may include the face of the second user.

[0063] According to one embodiment, the electronic device (101) can compare the first image (361) and the second image (362) with a third image (363). For example, the third image (363) may include a face of a second user. For example, the electronic device (101) can compare the first embedding vector (or first identification information) with a third embedding vector (or third identification information) extracted from the third image (363). For example, the electronic device (101) can compare the second embedding vector (or second identification information) with a third embedding vector (or third identification information) extracted from the third image (363).

[0064] Referring to example (370), the electronic device (101) can distinguish (or recognize) a first cluster (371) and a second cluster (372). For example, the electronic device (101) can classify a first image (361) into a first cluster (371), which is a set of data about the face of the first user, based on the facial recognition model (130). In addition, the electronic device (101) can classify a second image (362) into a second cluster (372), which is a set of data about the face of the second user, based on the facial recognition model (130).

[0065] As described above, the electronic device (101) can learn the face recognition model (130) by classifying the input data (e.g., the first image (361) and the second image (362)). Thereafter, the electronic device (101) can recognize the user's face by comparing the embedding vector (e.g., the embedding vector (302) of FIG. 3A) extracted from the newly input data (e.g., the image (301) of FIG. 3A) with the reference embedding vector (e.g., the reference embedding vectors (310) of FIG. 3A).

[0066] FIG. 4 illustrates an example of a method for recognizing a user based on a facial recognition model, according to one embodiment.

[0067] FIG. 4 illustrates an example of a method for an electronic device (101) to recognize users based on a facial recognition model (130). FIG. 4 illustrates an example of a method for an electronic device (101) to recognize a first user and a second user based on a facial recognition model (130), but this is merely for convenience of explanation and the embodiments of the present disclosure are not limited thereto.

[0068] According to one embodiment, the electronic device (101) can obtain a plurality of images (411, 412, 421, 422). For example, the electronic device (101) can use the plurality of images (411, 412, 421, 422) as inputs to a face recognition model (130).

[0069] For example, a first image (411) may include a face of a first user. The face of the first user in the first image (411) may have (or indicate, face to) a first direction. For example, the first direction may indicate a direction in which the user's face faces forward. For example, a second image (412) may include a face of the first user. The face of the first user in the second image (412) may have a second direction. For example, the second direction may include a direction in which the user's face faces in a direction different from the front.

[0070] For example, the third image (421) may include the face of the second user. The face of the second user in the third image (421) may have the first pose. For example, the fourth image (422) may include the face of the second user. The face of the second user in the fourth image (422) may have the second pose.

[0071] According to one embodiment, the electronic device (101) can output (or generate) user identification information from an image using a facial recognition model (130).

[0072] For example, the electronic device (101) can output (or generate) first identification information of the first user from the first image (411) using the facial recognition model (130). The electronic device (101) can generate first identification information by extracting a first embedding vector from the first image (411) using the facial recognition model (130) and comparing the extracted first embedding vector with reference embedding vectors.

[0073] For example, the electronic device (101) can output (or generate) first identification information of the first user from the second image (412) using the facial recognition model (130). The electronic device (101) can generate first identification information by extracting a second embedding vector from the second image (412) using the facial recognition model (130) and comparing the extracted second embedding vector with reference embedding vectors.

[0074] For example, the electronic device (101) can output (or generate) second identification information of the second user from the third image (421) using the facial recognition model (130). The electronic device (101) can generate second identification information by extracting a third embedding vector from the third image (421) using the facial recognition model (130) and comparing the extracted third embedding vector with reference embedding vectors.

[0075] For example, the electronic device (101) can output (or generate) second identification information of the second user from the fourth image (422) using the facial recognition model (130). The electronic device (101) can extract a fourth embedding vector from the fourth image (422) using the facial recognition model (130) and generate the second identification information by comparing the extracted fourth embedding vector with reference embedding vectors.

[0076] Referring to the above, the electronic device (101) can recognize the face of the user (or the user) from an image including the user's face using the facial recognition model (130). Referring to example (410), the electronic device (101) can recognize the face of the first user from the first image (411) and the second image (412). Referring to example (420), the electronic device (101) can recognize the face of the second user from the third image (421) and the fourth image (422).

[0077] According to one embodiment, the face recognition model (130) can be trained to recognize a user's face even if the orientation of the user's face within the image changes. For example, the face recognition model (130) can be trained using a loss function that is dependent (or variable) on the orientation of the face. For example, the loss function can be referred to as an angle-aware loss function. For example, the loss function can be used to accurately recognize a user's face from an image even when the orientation of the face within the image is in a second orientation different from the first orientation. In other words, the loss function can assign (or set) a margin differently depending on the degree to which the face is rotated from a direction facing forward. Accordingly, the electronic device (101) can accurately recognize a user from an image including a face even when the orientation of the face is not facing forward, by using the face recognition model (130) trained by the loss function.

[0078] Below, in FIG. 5a, an example of the angle-sensitive loss function that changes depending on the direction of the face is described. Also, in FIG. 5b, an example of a method for recognizing a face based on the angle-sensitive loss function that changes depending on the direction of the face and a loss function (e.g., ArcFace) that is set regardless of the direction of the face is described.

[0079] FIG. 5A illustrates an example of a loss function that is used to train a face recognition model according to one embodiment and that changes depending on the direction of the face. FIGS. 5B and 5C illustrate examples of a method for recognizing a user's face based on a face recognition model trained using a loss function that changes depending on the direction of the face, according to one embodiment.

[0080] FIG. 5a illustrates an example (500) of a method for generating an angle-aware loss function (e.g., loss function (325) of FIG. 3a) that is used for learning the face recognition model (130) of FIG. 3a and that changes according to the pose of the face in the image.

[0081] Referring to example (500), image (510) may include a face of a specific user. For example, the face in image (510) may face a specific direction. For example, the specific direction may be determined based on rotation angle values ​​representing an angle changed from a reference direction of the face. For example, the reference direction may include a first direction in which the face faces forward. For example, the rotation angle values ​​may include a roll with respect to the face, a pitch with respect to the face, and a yaw with respect to the face. For example, the roll may represent a first angle rotated around the x-axis (511), which is a direction in which the face faces. For example, the pitch may represent a second angle rotated around the y-axis (512). For example, the yaw may represent a third angle rotated around the z-axis (513).

[0082] According to one embodiment, the face angle estimation model (520) can calculate (or recognize, identify) rotation angles of a face within an image (510) from an image (510). For example, the face angle estimation model (520) can be included within a face recognition model (130). However, the embodiments of the present disclosure are not limited thereto. For example, the face angle estimation model (520) can be included within the memory (120) of the electronic device (101) as a different model from the face recognition model (130).

[0083] According to one embodiment, the face angle estimation model (520) can calculate the absolute value of each of the rotation angles. For example, the face angle estimation model (520) can calculate a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw. For example, the face angle estimation model (520) can calculate a sum (g(x)) of the first absolute value, the second absolute value, and the third absolute value. For example, the sum (g(x)) can be defined as in the following mathematical equation.

[0084]

[0085] The above x represents the input vector (or the embedding vector of the image (510)), the above g(x) represents a function representing the sum of the absolute values ​​of each rotation angle (hereinafter, the sum function), and the above The absolute value of the silver roll is is the absolute value of the pitch, and the above can represent the absolute value of a quantity.

[0086] In one embodiment, the activation function (σ) (530) may represent a function for the sum function (g(x)). For example, the activation function (σ) (530) may include a binary step activation function. For example, the activation function (σ) (530) may represent a function that applies (or calculates) a specific value to the sum function (g(x)) that exceeds a threshold value.

[0087] In one embodiment, the rotation margin value (540) may be calculated by the product of an activation function (σ) (530) and an additional margin value. For example, the final penalty margin may represent the sum of the rotation margin value (540) and a reference margin value (m). For example, the final penalty margin may be defined as follows:

[0088]

[0089] The above m is a reference margin value, the above λ is an additional margin value, the above μ(x) is an activation function (σ)(530) for the above sum function (g(x)), and the above may represent a rotation margin value (540). For example, the additional margin value may be a specified constant. For example, the additional margin value may be determined according to a face angle estimation model (520). For example, the reference margin value (m) may be used to distinguish between multiple users. For example, the reference margin value (m) may have the same value for multiple users (or multiple images including faces of multiple users).

[0090] For example, the rotation margin value (540)( ) and the sum of the reference margin value (m) can be referred to as the final penalty margin. For example, the rotation margin value (540) can be changed according to the pose of the user's face. The rotation margin value (540) can be used to calculate an angle-aware loss function (e.g., loss function (325)) used to train the feature extractor (300) of the face recognition model (130). The loss function, which changes according to the direction of the face, is described below through FIGS. 5b and 5c.

[0091] FIG. 5b illustrates an example (550) of a method for recognizing a user's face based on a face recognition model trained using a loss function that does not consider the direction of the face (e.g., the ArcFace described above). FIG. 5c illustrates an example (570) of a method for recognizing a user's face based on a face recognition model (130) trained using an angle-sensitive loss function that takes the direction of the face into account. Hereinafter, for convenience of explanation, the loss function that does not consider the direction of the face may be referred to as the first loss function, and the angle-sensitive loss function that takes the direction of the face into account may be referred to as the second loss function.

[0092] Referring to examples (550) and (570), the first reference vector (W1) (551) may represent the class (or identification information) of the first user, and the second reference vector (W2) (561) may represent the class (or identification information) of the second user. In other words, the closer the embedding vector obtained from the image is to the first reference vector (W1) (551), the more likely the face in the image is to indicate the face of the first user. In addition, the closer the embedding vector obtained from the image is to the second reference vector (W2) (561), the more likely the face in the image is to indicate the face of the second user.

[0093] Referring to the example (550) of FIG. 5B, the electronic device (101) can learn a face recognition model using a first loss function so as to distinguish users when obtaining an embedding vector from each of a plurality of images. When using the first loss function, the boundary between the first boundary (555) formed for the first reference vector (W1) (551) and the second boundary (565) formed for the second reference vector (W2) (561) can be defined as a reference margin value (m) (550a). In other words, the first loss function can distinguish users (or classes) using the reference margin value (m) (550a). The first loss function can be defined as the following mathematical equation.

[0094]

[0095] Above is the first loss function that removes the influence of bias (vias) from the ArcFace and utilizes the angle between vectors, N is the batch (or mini-batch) size (or number of samples), s is the scale factor, is the angle between the reference vector of the jth class and the input vector, Is The angle between the reference vector of the th class and the input vector may be represented by y, the class (or label), and m may represent the reference margin value.

[0096] As described above, the first loss function may use the reference margin value (550a) to position embedding vectors for the same user (or class) close together and position embedding vectors for different users (or classes) far apart. However, the first loss function may not consider the direction of faces of the same user. When using the first loss function (or using a face recognition model trained based on the first loss function), the electronic device (101) may have difficulty recognizing the reference vector (e.g., the second reference vector (W2) (561)) that is the learning target of the first embedding vector (567) and the second embedding vector (568). This may be because the first embedding vector (567) and the second embedding vector (568) are positioned close to the second boundary (565). In other words, when the electronic device (101) uses the first loss function, the electronic device (101) can apply the same margin (e.g., the reference margin value (550a)) to images including faces regardless of the direction of the face. Accordingly, when the electronic device (101) uses the first loss function, it may be difficult to recognize a learning target (or user, class) for a face facing a rotated direction. If the learning target is not accurately indicated for a face facing a rotated direction, even if the electronic device (101) uses a face recognition model trained using the first loss function, the recognition rate for a face facing a rotated direction may be low.

[0097] In contrast, referring to the example (570) of FIG. 5c, according to one embodiment, the electronic device (101) may learn the face recognition model (130) using the second loss function so as to be distinguished for each user when obtaining an embedding vector from each of a plurality of images. At this time, the second loss function may use a margin value (or a rotation margin value) that takes into account the pose of the face of each of the plurality of images. When the second loss function is used, the first boundary (555) formed for the first reference vector (W1) (551) and the second boundary (565) formed for the second reference vector (W2) (561) may be defined as a reference margin value (m) (550a). In addition, the first boundary (555) formed for the first reference vector (W1) (551) and the third boundary (585) formed for the second reference vector (W2) (561) may be defined as a rotation margin value ( )(570a) and the reference margin value (m)(550a) (or the final penalty margin). For example, the third boundary (585) may be defined as a weight (e.g., a rotation margin value ( )(570a)) can be used to induce (or cause) learning about a specific user of the face (e.g., the second user).

[0098] In other words, the second loss function distinguishes users (or classes) using the reference margin value (m) (550a) and the rotation margin value ( )(570a)) can be used to induce a face in a rotated direction to be learned for a specific user. The second loss function can be defined as the following mathematical formula.

[0099]

[0100] The above L is the second loss function that induces easy recognition of distorted faces during training of the face recognition model (130) (or feature extractor (300)) by applying a larger margin (or penalty margin) as the change in facial pose increases, the above N is the batch (or mini-batch) size (or number of samples), the above s is the scale factor, and the above is the angle between the reference vector of the jth class and the input vector, Is The angle between the reference vector of the th class and the input vector may be represented by y, the class (or label), m may represent a reference margin value, μ(x) may represent an activation function (σ) for the sum function (g(x)), and λ may represent an additional margin value.

[0101] Referring to the above, the second loss function can use the reference margin value (550a) to position the embedding vectors for the same user (or class) closer together and position the embedding vectors for different users (or classes) further apart. In addition, the second loss function can apply a larger penalty value (e.g., a rotation margin value ()) to the embedding vectors of faces with a misaligned direction (e.g., the second direction). )(570a)) By applying the user's reference vector of the face and the above-mentioned embedding vectors of the face, the relationship can be closely adjusted.

[0102] Referring to examples (550) and (570), the first embedding vector (567) and the second embedding vector (568) can be changed to the third embedding vector (587) and the fourth embedding vector (588) as the second loss function is used. For example, the first embedding vector (567) can be changed to the rotation margin value ( )(570a) can be changed to the third embedding vector (587). For example, the second embedding vector (568) can be changed to the rotation margin value of the second loss function ( )(570a) may be changed to the fourth embedding vector (588). For example, the third embedding vector (587) and the fourth embedding vector (588) may be located within the third boundary (585) and closer to the second reference vector (W2)(561) indicating the second user. As described above, compared to the first loss function, the second loss function may apply a larger margin (or penalty margin) to the embedding vector representing the face in the rotated pose.

[0103] According to one embodiment, the electronic device (101) comprises a rotation margin value ( )(570a), a facial recognition model (130) can be trained using the third embedding vector (587) and the fourth embedding vector (588), which are learning targets of the second reference vector (W2)(561). An electronic device (101) using the trained facial recognition model (130) can accurately recognize a user of the face even if an image including a face in a rotated direction is acquired (or input). In other words, when the electronic device (101) uses the trained facial recognition model (130) based on the second loss function, the recognition rate for a distorted face can be improved.

[0104] FIG. 6 illustrates an example of an operational flow for a method of recognizing a user's face based on a face recognition model learned using a loss function that changes depending on the direction of the face, according to one embodiment.

[0105] The electronic device of FIG. 6 may include the electronic device (101) of FIG. 1. At least one of the operations of FIG. 6 may be performed by the electronic device (101) of FIG. 1. For example, at least one of the operations may be controlled by the processor (110) of FIG. 1. Each of the operations of FIG. 6 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each of the operations may be changed, and at least two operations may be performed in parallel.

[0106] The method of FIG. 6 can be used to learn a facial recognition model for recognizing a user's face. For example, the facial recognition model may include the facial recognition model (130) of FIG. 3A. For example, the facial recognition model (130) may be learned based on an angle-recognition loss function (or the second loss function) based on the direction of the face.

[0107] According to one embodiment, in operation (600), the electronic device (101) may obtain rotation angle values ​​indicating the direction of a face within an image. For example, the electronic device (101) may obtain rotation angle values ​​indicating the direction of a face from an image of a user.

[0108] According to one embodiment, the electronic device (101) may obtain rotation angle values ​​using a face angle estimation model (e.g., the face angle estimation model (520) of FIG. 5A). For example, the face angle estimation model (520) may be included in a face recognition model (130). However, the embodiments of the present disclosure are not limited thereto. For example, the face angle estimation model (520) may be included in the memory (120) of the electronic device (101).

[0109] For example, the rotation angle values ​​may include roll, pitch, and yaw relative to the face. For example, the roll may represent a first angle of rotation around the x-axis, which is the direction in which the face is facing. For example, the pitch may represent a second angle of rotation around the y-axis. For example, the yaw may represent a third angle of rotation around the z-axis.

[0110] According to one embodiment, the electronic device (101) can calculate the absolute value of each of the rotation angles using the face angle estimation model (520). For example, the electronic device (101) can calculate a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw using the face angle estimation model (520). For example, the electronic device (101) can calculate a sum (g(x)) of the first absolute value, the second absolute value, and the third absolute value using the face angle estimation model (520).

[0111] According to one embodiment, in operation (620), the electronic device (101) may generate an activation function (σ) for the sum (g(x)) of the first absolute value for the roll, the second absolute value for the pitch, and the third absolute value for the yaw. For example, the activation function (σ) may include a binary step activation function. For example, the activation function (σ) may represent a function that applies (or calculates) a specific value to a sum function (g(x))) that exceeds a threshold value.

[0112] According to one embodiment, in operation (630), the electronic device (101) derives a rotation margin value (σ) from the product between the activation function (σ) and the additional margin value (λ). ) can be calculated. For example, the rotation margin value ( ) can be changed depending on the direction of the user's face. Rotation margin value ( ) can be used to calculate a loss function used to learn a feature extractor (300) of a face recognition model (130).

[0113] According to one embodiment, in operation (640), the electronic device (101) determines a rotation margin value ( ) and the reference margin value (m) can be used to distinguish between multiple users. For example, the reference margin value (m) can have the same value for multiple users (or multiple images including faces of multiple users). The rotation margin value ( ) and the sum of the reference margin value (m), the above mathematical expression 5 may be referred to.

[0114] For example, the loss function can learn embedding vectors for the same user (or class) to be close together and learn embedding vectors for different users (or classes) to be far apart by using a reference margin value (m). In addition, the loss function can apply a larger penalty value (e.g., a rotation margin value ( )) By applying the user's reference vector of the face and the face embedding vectors, the relationship can be closely adjusted.

[0115] According to one embodiment, in operation (650), the electronic device (101) may apply the loss function to the model. For example, the electronic device (101) may use the loss function to train the face recognition model (130). By training the face recognition model (130) using the loss function, the electronic device (101) may accurately recognize the user of the face even if an image including a face in a misaligned direction is acquired (or input).

[0116] FIG. 7 illustrates an example of a face recognition model having weights transferred from weights of a model learned using an image reconstructed from an image representing a portion of a user's face, according to one embodiment.

[0117] FIG. 7 illustrates an example (700) of a method for transferring the weights of a pre-trained model to the face recognition model. For example, the face recognition model of FIG. 7 may include the face recognition model (130) of FIG. 3A. The pre-trained model may include a masked auto encoder (MAE) (710). For example, the feature extractor (300) of FIG. 7 may be included in the face recognition model (130).

[0118] According to one embodiment, the electronic device (101) may initialize the feature extractor (300). For example, the electronic device (101) may initialize the feature extractor (300) to perform face recognition more effectively. At this time, the electronic device (101) may initialize the weights of the feature extractor (300) by transferring the weights of the pre-learned MAE (710).

[0119] According to one embodiment, the electronic device (101) can learn the MAE (710) before initializing the weights of the feature extractor (300). For example, the electronic device (101) can use a first reference image (703) including a user's face to learn the MAE (710). For example, the electronic device (101) can generate a second reference image (701) from the first reference image (703). For example, the second reference image (701) can include a portion of the user's face. For example, the second reference image (701) can represent an image including a portion of the first reference image (703). In this case, the portion included in the second reference image (701) can have a designated value determined according to the feature extractor (300) (or the face recognition model (130)). For example, the electronic device (101) can input a second reference image (701) into the MAE (710). For example, the electronic device (101) can generate an output image (702) restored from the second reference image (701) using the MAE (710).

[0120] According to one embodiment, the electronic device (101) measures the difference (L) between the output image (702) and the first reference image (703). MSE ) can be calculated. For example, the difference (L MSE ) can be calculated using the difference between the vector value of the output image (702) and the vector value of the first reference image (703). The difference (L MSE ) can be referenced to the mathematical formula below for specific details.

[0121]

[0122] Above L MSE is the difference between the restored (or predicted) image (e.g., output image (702)) and the correct image (e.g., first reference image (703)), and N is the batch (or mini-batch) size (or number of samples), is the vector value of the correct image, and can represent the vector values ​​of the restored image.

[0123] Referring to the above, the electronic device (101) has the above difference (L MSE ) and calculate the calculated difference (L MSE ) can be used to learn MAE (710). For example, the electronic device (101) can learn the difference (L MSE ) can be learned through backpropagation. Learning the MAE (710) may include updating the weights of the MAE (710).

[0124] According to one embodiment, the electronic device (101) can transfer the updated weights of the MAE (710) to the feature extractor (300). For example, the electronic device (101) can change (or update, adjust, learn, transfer) the weights of the feature extractor (300) to the updated weights of the MAE (710).

[0125] Referring to the above, the feature extractor (300) (or face recognition model) can have weights that are robust to distortions such as changes in brightness of an image including a face or changes in direction of a face, by transferring the weights of the MAE (710) learned using the intentionally removed second reference image (701).

[0126] FIG. 8 illustrates examples of an image representing a portion of a user's face, a restored image, and an image representing the user's face, according to one embodiment.

[0127] Fig. 8 illustrates examples (810, 820, 830, 840, 850, 860) of images used for training the MAE (710) of Fig. 7. For example, an image representing a part of a face may represent an input image (e.g., the second reference image (701) of Fig. 7). For example, the restored image may represent a restored image (e.g., the output image (702) of Fig. 7). For example, the image representing the face may represent a correct image (e.g., the first reference image (703) of Fig. 7).

[0128] Referring to example (810), the electronic device (101) can use images (811, 812, 813) including a woman's face for training the MAE (710). For example, the electronic device (101) can generate an input image (811) from which a portion is removed from the correct image (813) and input the input image (811) into the MAE (710). Accordingly, the electronic device (101) can generate a restored image (812) restored from the input image (811). The restored image (812) may be similar to the correct image (813), but may not be completely identical.

[0129] Referring to example (820), the electronic device (101) can use images (821, 822, 823) including a male face for training the MAE (710). For example, the electronic device (101) can generate an input image (821) from which a portion is removed from the correct image (823) and input the input image (821) into the MAE (710). Accordingly, the electronic device (101) can generate a restored image (822) restored from the input image (821). The restored image (822) may be similar to the correct image (823), but may not be completely identical.

[0130] Referring to example (830), the electronic device (101) may utilize images (831, 832, 833) containing faces and having relatively dark brightness for training the MAE (710). For example, the electronic device (101) may generate an input image (831) from which a portion is removed from the correct image (833) and input the input image (831) into the MAE (710). Accordingly, the electronic device (101) may generate a restored image (832) restored from the input image (831). The restored image (832) may be similar to the correct image (833), but may not be completely identical.

[0131] Referring to example (840), the electronic device (101) may utilize images (841, 842, 843) that include faces and have relatively bright brightness for training the MAE (710). For example, the electronic device (101) may generate an input image (841) from which a portion is removed from the correct image (843) and input the input image (841) into the MAE (710). Accordingly, the electronic device (101) may generate a restored image (842) that is restored from the input image (841). The restored image (842) may be similar to the correct image (843), but may not be completely identical.

[0132] Referring to example (850), the electronic device (101) can use images (851, 852, 853) including faces that are oriented in a first direction (e.g., right) for training the MAE (710). For example, the electronic device (101) can generate an input image (851) from which a portion is removed from a correct image (853) and input the input image (851) into the MAE (710). Accordingly, the electronic device (101) can generate a restored image (852) that is restored from the input image (851). The restored image (852) may be similar to, but not completely identical to, the correct image (853).

[0133] Referring to example (860), the electronic device (101) can use images (861, 862, 863) including faces that are oriented in a second direction (e.g., left) for training the MAE (710). For example, the electronic device (101) can generate an input image (861) from which a portion is removed from the correct image (863) and input the input image (861) to the MAE (710). Accordingly, the electronic device (101) can generate a restored image (862) that is restored from the input image (861). The restored image (862) may be similar to, but not completely identical to, the correct image (863).

[0134] Referring to the above, the electronic device (101) can learn the MAE (710) by using various images as illustrated in FIG. 8. The electronic device (101) can transfer (or initialize) the weights of the MAE (710) learned based on the various images to the weights of the face recognition model (130) (or feature extractor (300)). By using the face recognition model (130) (or feature extractor (300)) having the transferred weights, the electronic device (101) can accurately recognize the user's face even in an environment where the face is not clearly visible or difficult to recognize in the image.

[0135] As described above, an electronic device may include a memory storing instructions. The electronic device may include a processor operatively coupled to the memory. The instructions, when executed by the processor, may cause the electronic device to obtain an image including a face of a user. The instructions, when executed by the processor, may cause the electronic device to extract feature values ​​from the obtained image using a face recognition model in which weights of a masked auto encoder (MAE) trained on faces including the face are transferred. The instructions, when executed by the processor, may cause the electronic device to identify a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values ​​from among reference embedding vectors stored in the memory. The instructions, when executed by the processor, may cause the electronic device to recognize the user corresponding to the reference embedding vector identified from the image. The face recognition model may be trained using a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating the orientation of the face.

[0136] In one embodiment, the at least one rotation angle value may include a roll relative to the face, a pitch relative to the face, and a yaw relative to the face.

[0137] In one embodiment, the rotation margin value may be calculated based on an activation function for the sum of a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw.

[0138] In one embodiment, the activation function may include a binary step activation function. The rotation margin value may be calculated based on the product between the activation function and an additional margin value for the sum.

[0139] In one embodiment, the reference margin value may be used to distinguish between multiple users. The reference margin value may be applied equally to multiple images containing the face of the user.

[0140] In one embodiment, when the direction is a first direction in which the face faces the front, the rotation margin value may have a first value. When the direction is a second direction in which the face faces a different direction from the front, the rotation margin value may have a second value different from the first value.

[0141] According to one embodiment, when the first direction is the first direction, the face recognition model may be further trained using the embedding vector to which the loss function calculated based on the reference margin value among the rotation margin value and the reference margin value is applied. When the direction is the second direction, the face recognition model may be further trained using the embedding vector to which the loss function calculated based on the sum between the rotation margin value and the reference margin value is applied.

[0142] In one embodiment, the weights of the face recognition model may be changed as the weights of the MAE are transferred. The MAE may be learned based on the difference between an output image reconstructed from a first reference image including the face of the user among the faces and a second reference image including a portion of the first reference image.

[0143] According to one embodiment, the weight of the MAE may be updated through backpropagation of the difference calculated based on the square of the difference vector value between the first vector value of the first reference image and the vector value of the second reference image.

[0144] The method performed by the electronic device as described above may include an operation of acquiring an image including a face of a user. The method may include an operation of extracting feature values ​​from the acquired image using a face recognition model in which weights of a masked auto encoder (MAE) learned for faces including the face are transferred. The method may include an operation of identifying a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values ​​among stored reference embedding vectors. The method may include an operation of recognizing the user corresponding to the reference embedding vector identified from the image. The face recognition model may be trained through a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating an orientation of the face.

[0145] In one embodiment, the at least one rotation angle value may include a roll relative to the face, a pitch relative to the face, and a yaw relative to the face.

[0146] In one embodiment, the rotation margin value may be calculated based on an activation function for the sum of a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw.

[0147] In one embodiment, the activation function may include a binary step activation function. The rotation margin value may be calculated based on the product between the activation function and an additional margin value for the sum.

[0148] In one embodiment, the reference margin value may be used to distinguish between multiple users. The reference margin value may be applied equally to multiple images containing the face of the user.

[0149] In one embodiment, when the direction is a first direction in which the face faces the front, the rotation margin value may have a first value. When the direction is a second direction in which the face faces a different direction from the front, the rotation margin value may have a second value different from the first value.

[0150] According to one embodiment, when the direction is the first direction, the face recognition model may be further trained using the embedding vector to which the loss function calculated based on the reference margin value among the value and the reference margin value is applied. When the direction is the second direction, the face recognition model may be further trained using the embedding vector to which the loss function calculated based on the sum between the value and the reference margin value is applied.

[0151] In one embodiment, the weights of the face recognition model may be changed as the weights of the masked autoencoder (MAE) are transferred. The MAE may be learned based on the difference between an output image reconstructed from a first reference image including the face of the user among the faces and a second reference image including a portion of the first reference image.

[0152] According to one embodiment, the weight of the MAE may be updated through backpropagation of the difference calculated based on the square of the difference vector value between the first vector value of the first reference image and the vector value of the second reference image.

[0153] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by a processor of an electronic device, cause the processor to obtain an image including a face of a user. The one or more programs may include instructions that, when executed by the processor, cause the processor to extract feature values ​​from the obtained image using a face recognition model in which weights of a masked auto encoder (MAE) learned for faces including the face are transferred. The one or more programs may include instructions that, when executed by the processor, cause the processor to identify a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values ​​among stored reference embedding vectors. The one or more programs may include instructions that, when executed by the processor, cause the processor to recognize the user corresponding to the reference embedding vector identified from the image. The above face recognition model can be trained through a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating the orientation of the face.

[0154] A method for training a model for recognizing a user's face, as described above, may include an operation of obtaining rotation angle values ​​representing an orientation of the face within an image of the user. The rotation angle values ​​may include a roll for the face, a pitch for the face, and a yaw for the face. The method may include an operation of generating an activation function for a sum of a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw. The method may include an operation of calculating a rotation margin value from a product between the generated activation function and an additional margin value. The method may include an operation of calculating a loss function based on a sum of the rotation margin value and a reference margin value. The method may include an operation of applying the loss function to the model.

[0155] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0156] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0157] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0158] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0159] Therefore, other implementations, other embodiments, and equivalents of the claims are also within the scope of the claims described below. According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0160] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, Memory for storing instructions; and comprising a processor operatively coupled with said memory; The above instructions, when executed by the processor, cause the electronic device to: Obtain an image containing the user's face; Extract feature values from the acquired image using a face recognition model in which the weights of a masked auto encoder (MAE) learned for faces including the above face are transferred; Identifying a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values among reference embedding vectors stored in the memory; and To cause the user to be recognized corresponding to the reference embedding vector identified from the image, The above face recognition model is learned through a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating the orientation of the face. Electronic devices.

2. In claim 1, wherein said at least one rotation angle value comprises a roll relative to said face, a pitch relative to said face, and a yaw relative to said face. Electronic devices.

3. In claim 2, The above rotation margin value is calculated based on an activation function for the sum of a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw. Electronic devices.

4. In claim 3, The above activation function includes a binary step activation function, and The above rotation margin value is calculated based on the product between the activation function for the above sum and the additional margin value. Electronic devices.

5. In claim 4, The above reference margin value is used to distinguish between multiple users, and The above reference margin value is applied equally to multiple images including the face of the user. Electronic devices.

6. In claim 1, If the above direction is a first direction in which the face faces the front, the rotation margin value has a first value, and If the direction is a second direction in which the face is different from the front, the rotation margin value has a second value different from the first value. Electronic devices.

7. In claim 6, If the direction is the first direction, the face recognition model is further learned using the embedding vector to which the loss function calculated based on the reference margin value among the rotation margin value and the reference margin value is applied, and If the direction is the second direction, the face recognition model is further learned by using the embedding vector to which the loss function is applied, which is calculated based on the sum between the rotation margin value and the reference margin value. Electronic devices.

8. In claim 1, The weights of the above face recognition model are changed as the weights of the MAE (masked auto encoder) are transferred, and The above MAE is learned based on the difference between the output image reconstructed from a first reference image including the face of the user among the faces and a second reference image including a part of the first reference image. Electronic devices.

9. In claim 8, The weights of the above MAE are updated through backpropagation of the difference calculated based on the square of the difference vector value between the first vector value of the first reference image and the vector value of the second reference image. Electronic devices.

10. A method performed by an electronic device, said method comprising: An action of obtaining an image containing the user's face; An operation of extracting feature values from the acquired image using a face recognition model to which the weights of a masked auto encoder (MAE) learned for faces including the above face are transferred; An operation of identifying a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values among the stored reference embedding vectors; and comprising an action of recognizing the user corresponding to the reference embedding vector identified from the image; The above face recognition model is learned through a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating the orientation of the face. method.

11. In claim 10, wherein said at least one rotation angle value comprises a roll relative to said face, a pitch relative to said face, and a yaw relative to said face. method.

12. In claim 11, The above rotation margin value is calculated based on an activation function for the sum of a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw. method.

13. In claim 12, The above activation function includes a binary step activation function, The above rotation margin value is calculated based on the product between the activation function for the above sum and the additional margin value. method.

14. In claim 13, The above reference margin value is used to distinguish between multiple users, and The above reference margin value is applied equally to multiple images including the face of the user. method.

15. In claim 10, If the above direction is a first direction in which the face faces the front, the rotation margin value has a first value, and If the direction is a second direction in which the face is different from the front, the rotation margin value has a second value different from the first value. method.

16. In claim 15, If the direction is the first direction, the face recognition model is further learned using the embedding vector to which the loss function calculated based on the reference margin value among the value and the reference margin value is applied, and If the direction is the second direction, the face recognition model is further learned by using the embedding vector to which the loss function is applied, which is calculated based on the sum between the value and the reference margin value. method.

17. In claim 10, The weights of the above face recognition model are changed as the weights of the MAE (masked autoencoder) are transferred, and The above MAE is learned based on the difference between the output image reconstructed from a first reference image including the face of the user among the faces and a second reference image including a part of the first reference image. method.

18. In claim 17, The weights of the above MAE are updated through backpropagation of the difference calculated based on the square of the difference vector value between the first vector value of the first reference image and the vector value of the second reference image. method.

19. In a computer-readable storage medium having one or more programs stored thereon, when the one or more programs are executed by a processor of an electronic device: Obtain an image containing the user's face; Extract feature values from the acquired image using a face recognition model in which the weights of a masked auto encoder (MAE) learned for faces including the above face are transferred; Identifying a reference embedding vector corresponding to an embedding vector generated based on the extracted feature values among the stored reference embedding vectors; and Including instructions causing the user to be recognized corresponding to the reference embedding vector identified from the image, The above face recognition model is learned through a loss function calculated based on a rotation margin value and a reference margin value based on at least one rotation angle value indicating the orientation of the face. Computer readable storage medium.

20. A method for learning a model for recognizing a user's face, the method comprising: An operation of obtaining rotation angle values representing an orientation of the face within an image of the user, the rotation angle values including a roll with respect to the face, a pitch with respect to the face, and a yaw with respect to the face; An operation of generating an activation function for the sum of a first absolute value for the roll, a second absolute value for the pitch, and a third absolute value for the yaw; An operation of calculating a rotation margin value from the product between the generated activation function and an additional margin value; An operation of calculating a loss function based on the sum of the above rotation margin value and the reference margin value; and comprising an action of applying the above loss function to the model; method.

Images