Electronic device and operation method of electronic device

Abstract

This electronic device comprises a memory storing instructions and a processor, wherein, when the processor executes the instructions, the electronic device may: acquire a face image comprising a first area covered by an occluder and a second area not covered by the occluder; acquire, from the face image, an image feature vector in which an occlusion feature vector extracted from the first area of the face image is included in a first index and a person feature vector extracted from the second area of the face image is included in a second index; and compare the person feature vector included in the second index in the image feature vector with a plurality of reference feature vectors corresponding to each of a plurality of reference face images to select, from among the plurality of reference face images, at least one reference face image corresponding to the face image.

Description

Electronic devices and methods of operating electronic devices

[0001] The present disclosure relates to an electronic device and a method of operating the electronic device. Specifically, the present disclosure relates to an electronic device for comparing facial images and a method of operating the electronic device.

[0002] Recently, technologies for recognizing and analyzing faces within images have been developing. Specifically, technologies for extracting feature vectors from images and using these extracted feature vectors to recognize faces within images are widely utilized.

[0003] Additionally, the electronic device can determine the similarity between multiple images based on the recognized results and provide similar images as a group to the user.

[0004] Furthermore, recent advancements in technology related to electronic devices that display images and provide them to users have led to advancements. Accordingly, electronic devices can provide users with multiple photos through user interfaces such as galleries, and similar images can be grouped and presented to users.

[0005] One embodiment of the present disclosure provides an electronic device. The electronic device may include a memory storing at least one instruction. The electronic device may include at least one processor including a processing circuit. By having at least one processor individually or collectively execute at least one instruction stored in the memory, the electronic device may obtain a facial image including a first region occluded by an occluder and a second region not occluded by the occluder. By having at least one processor individually or collectively execute at least one instruction, the electronic device may obtain an image feature vector using an artificial intelligence model, wherein an occlusion feature vector extracted from a first region of the facial image is included in a first index, and a person feature vector extracted from a second region of the facial image is included in a second index. By having at least one processor individually or collectively execute at least one command, the electronic device can compare a person feature vector included in a second index among the image feature vectors with a plurality of reference feature vectors corresponding to each of the plurality of reference face images, thereby selecting at least one reference face image corresponding to the face image among the plurality of reference face images.

[0006] In one embodiment of the present disclosure, a method of operating an electronic device may be provided. The method of operating the electronic device may include a step of obtaining a face image including a first region occluded by an occluder and a second region not occluded by the occluder. The method of operating the electronic device may include a step of obtaining an image feature vector, using an artificial intelligence model, in which an occlusion feature vector extracted from a first region of the face image is included in a first index and a person feature vector extracted from a second region of the face image is included in a second index. The method of operating the electronic device may include a step of comparing a person feature vector included in the second index among the image feature vectors with a plurality of reference feature vectors corresponding to each of a plurality of reference face images, and selecting at least one reference face image corresponding to the face image from among the plurality of reference face images.

[0007] As one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing at least one of the operating methods of the disclosed electronic device on a computer can be provided.

[0008] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0009] The present disclosure may be understood in conjunction with the following detailed description and accompanying drawings, wherein reference numerals refer to structural elements.

[0010] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0013] FIG. 4 is a diagram for explaining a learning method of an artificial intelligence model according to one embodiment of the present disclosure.

[0014] FIG. 5 is a diagram for explaining an image feature vector obtained from a facial image through an artificial intelligence model according to one embodiment of the present disclosure.

[0015] FIG. 6 is a flowchart illustrating an operation of comparing a person feature vector with a plurality of reference person feature vectors selected from a plurality of reference feature vectors according to one embodiment of the present disclosure.

[0016] FIG. 7 is a flowchart illustrating an operation of obtaining a person feature vector from an image feature vector based on an occlusion segmentation map according to one embodiment of the present disclosure.

[0017] FIG. 8 is a diagram for explaining an operation of obtaining a person feature vector based on a two-dimensional image feature vector and an occlusion segmentation map according to one embodiment of the present disclosure.

[0018] FIG. 9 is a flowchart illustrating an operation of obtaining a person feature vector among image feature vectors by comparing preset facial feature values ​​and feature values ​​according to one embodiment of the present disclosure.

[0019] FIG. 10 is a flowchart illustrating an operation of comparing similarity by unit area included in a second area by comparing a person feature vector and a reference person feature vector according to one embodiment of the present disclosure.

[0020] FIG. 11 is a diagram for explaining an operation of comparing the similarity of facial features of a face included in a second area by comparing a person feature vector and a reference person feature vector according to one embodiment of the present disclosure.

[0021] FIG. 12 is a flowchart illustrating an operation of clustering selected reference face images and face images into one group according to one embodiment of the present disclosure.

[0022] FIG. 13 is a diagram illustrating an interface for proposing clustering based on similarity according to one embodiment of the present disclosure.

[0023] The terms used in this disclosure will be briefly explained, and one embodiment of the present disclosure will be specifically described.

[0024] Throughout this disclosure, unless specifically stated otherwise, "or" is inclusive and not exclusive. Thus, unless explicitly stated otherwise or context dictates otherwise, "A or B" may refer to "A, B, or both."

[0025] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0026] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0027] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0028] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like described herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0029] The expression “configured to” as used herein can be used interchangeably with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean something that is “specifically designed to” in terms of hardware. Instead, in some contexts, the expression “a system configured to” can mean that the system is “capable of” doing something together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0030] Additionally, when a component is referred to as being “connected” or “connected” to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0031] It should be understood that the blocks and combinations of flowcharts in each flowchart can be executed by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory, or may be stored in separate portions across multiple different memories.

[0032] Any function or operation described in this document may be performed by a single processor or a combination of multiple processors.

[0033] The artificial intelligence-related functions according to the present disclosure are operated via a processor and memory. One or more processors control the processing of input data according to predefined operating rules or artificial intelligence models stored in the memory. Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0034] The predefined operation rules or artificial intelligence model are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed in the electronic device itself using the artificial intelligence model according to the present disclosure, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0035] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0036] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0037] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0038] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0039] Referring to FIG. 1, in one embodiment of the present disclosure, FIG. 1 illustrates an electronic device (100) and a facial image (200) provided to the electronic device (100).

[0040] In one embodiment of the present disclosure, the electronic device (100) may be implemented as an electronic device of various shapes, such as a mobile device, a smart phone, a laptop computer, a desktop, a tablet PC, and a wearable device.

[0041] In one embodiment of the present disclosure, an electronic device (100) may acquire a face image (200). The face image (200) may be an image including a human face. In one embodiment of the present disclosure, the face image (200) may be an image captured by a camera included in the electronic device (100). However, the present disclosure is not limited thereto, and the face image (200) may also be an image stored in an external electronic device or an external server.

[0042] In one embodiment of the present disclosure, the facial image (200) may be an image acquired by recognizing a facial portion from an entire image including a person. The electronic device (100) may acquire the entire image and recognize a facial portion from the entire image to acquire the facial image (200).

[0043] In one embodiment of the present disclosure, the electronic device (100) can extract an image feature vector (230) including features of the face image (200) from the acquired face image (200). The electronic device (100) can obtain the image feature vector (230) from the face image (200) by using a pre-learned artificial intelligence model to extract the feature vector from the image.

[0044] In one embodiment of the present disclosure, the face image (200) may include an occluder. The occluder may refer to an object that obscures the face. In one embodiment of the present disclosure, the occluder may include an object worn by a person to obscure the face, such as a mask, sunglasses, or a hat. In addition, the occluder may include an object that overlaps the face and obscures the face, such as a hand positioned on the face, another person's shoulder, a pole, or a leaf, but is not limited thereto.

[0045] In one embodiment of the present disclosure, a face image (200) may include a first region (210) covered by a shield and a second region (220) not covered by the shield. In this case, the shield is depicted as a mask in FIG. 1 . In addition, the second region (220) may refer to the remaining facial region not covered by the mask.

[0046] In one embodiment of the present disclosure, an artificial intelligence model used to extract an image feature vector (230) from a face image (200) may be an artificial intelligence model that is pre-trained to output an image feature vector (230) to which a plurality of unit indices (233) corresponding to a plurality of unit regions included in a face image (200) provided as an input are assigned. Each of the plurality of unit indices (233) assigned to the image feature vector (230) obtained through the artificial intelligence model may correspond to a plurality of unit regions included in the face image (200). In one embodiment of the present disclosure, N unit indices from 1 to N may be assigned to the image feature vector (230). In this case, N may be a natural number greater than or equal to 2.

[0047] In one embodiment of the present disclosure, each of the plurality of unit regions may be a basic unit for distinguishing a region of a face image (200). Each of the plurality of unit indices (233) may mean the number of digits or position information of each of the plurality of elements included in the image feature vector (230).

[0048] In one embodiment of the present disclosure, the first region (210) and the second region (220) may each include at least one unit region. At least one unit index corresponding to the first region (210) may be referred to as a first index. At least one unit index corresponding to the second region (220) may be referred to as a second index. In one embodiment of the present disclosure, the first index may include unit indices from 1 to K. The second index may include unit indices from K+1 to N. In this case, K may be a natural number greater than 1 and less than N.

[0049] In one embodiment of the present disclosure, a feature vector included in a first index of an image feature vector (230) may be referred to as a masking feature vector (231). A feature vector included in a second index of an image feature vector (230) may be referred to as a person feature vector (232). In this case, the masking feature vector (231) may be a vector including features extracted from a first area (210) of a face image (200). The person feature vector (232) may be a vector including features extracted from a second area (220) of a face image (200).

[0050] In one embodiment of the present disclosure, an electronic device (100) may include a memory (110). A plurality of reference face images (300) may be stored in the memory (110). A plurality of reference face images (300) and a plurality of reference feature vectors (310) corresponding to each of the plurality of reference face images (300) may be stored in the memory (110).

[0051] At this time, the plurality of reference feature vectors (310) may be feature vectors obtained using an artificial intelligence model that has been trained in advance to infer, as output, a plurality of reference feature vectors (310) including a plurality of unit indices each corresponding to a plurality of unit areas included in each of the plurality of reference face images (300) provided as input. The artificial intelligence model used to obtain the plurality of reference feature vectors (310) may be the same model as the artificial intelligence model used to obtain the image feature vector (230).

[0052] In one embodiment of the present disclosure, a plurality of reference face images (300) may be preprocessed so that they have similar poses (e.g., a pose looking straight ahead) and similar sizes of faces included in each image. The plurality of reference face images (300) stored in the memory (110) may have undergone preprocessing such as resizing, rotation, or cropping.

[0053] In one embodiment of the present disclosure, the electronic device (100) may perform preprocessing on the acquired face image (200) so that the acquired face image has a pose and face size similar to those of multiple reference face images (300) stored in the memory (110). The electronic device (100) may also acquire an image feature vector (230) based on the preprocessed face image (200).

[0054] In one embodiment of the present disclosure, the electronic device (100) can compare an image feature vector (230) with a plurality of reference feature vectors (310) to select at least one reference face image (301) corresponding to a face image (200) among a plurality of reference face images (300).

[0055] In one embodiment of the present disclosure, the electronic device (100) may calculate a similarity between each of a plurality of reference feature vectors (310) and an image feature vector (230) using cosine similarity or the like. The electronic device (100) may select at least one reference face image (301) corresponding to at least one reference feature vector having a similarity higher than a preset reference similarity as an image corresponding to the face image (200).

[0056] In one embodiment of the present disclosure, a plurality of reference feature vectors (310) may be assigned a first index and a second index. A feature vector included in the first index of each of the plurality of reference feature vectors (310) may be referred to as a first reference feature vector (311). A feature vector included in the second index of each of the plurality of reference feature vectors (310) may be referred to as a second reference feature vector (312).

[0057] In one embodiment of the present disclosure, the electronic device (100) may select a plurality of second reference feature vectors (312) included in a second index from each of a plurality of reference feature vectors (310). In this case, the plurality of second reference feature vectors (312) may be referred to as a plurality of reference person feature vectors (312).

[0058] In one embodiment of the present disclosure, the electronic device (100) may compare a person feature vector (232) with each of a plurality of reference person feature vectors (312) to select at least one reference face image (301) corresponding to the face image among a plurality of reference face images. At this time, by comparing the person feature vector (232) included in the second index with the plurality of reference person feature vectors (312), the similarity between the face images may be determined by comparing the features of the second area (220) that is not covered by the occlusion among the person image (200) with the features of the area corresponding to the second area (220) among the plurality of reference person images (300).

[0059] For example, if a mask included in a face image (200) serves as a mask and covers the lower part of the face, the remaining area not covered by the mask may be the upper part of the face.

[0060] The electronic device (100) may select a feature vector included in an index corresponding to the upper part of the face among the image feature vectors (230) extracted from the face image (200) as a person feature vector. The electronic device (100) may select a feature vector included in an index corresponding to the upper part of the face among a plurality of reference feature vectors (310) as a plurality of reference person feature vectors.

[0061] The electronic device (100) can compare the selected person feature vector with a plurality of reference person feature vectors to select at least one reference face image (301) corresponding to the face image among the plurality of reference face images. The selected at least one reference face image (301) may be selected based on a similarity calculated using only the upper part of the face that is not covered by a shield.

[0062] Accordingly, the electronic device (100) can select at least one corresponding reference face image (301) from among a plurality of reference face images (300) stored in the memory (110) with high accuracy even if the acquired face image (200) includes a occlusion. In addition, since the similarity is determined using only the feature vectors included in the index corresponding to the area not occluded by the occlusion among the entire feature vectors, the time and amount of calculation required can be reduced.

[0063] In one embodiment of the present disclosure, the electronic device (100) can cluster the face image (200) and at least one reference face image (301) selected as corresponding to the face image (200) among a plurality of reference face images (300) into one group (400).

[0064] In one embodiment of the present disclosure, a plurality of reference face images (300) may be provided to a user of an electronic device (100) in the form of a gallery user interface. At this time, the electronic device (100) may group and provide two or more reference face images having a high degree of similarity among the plurality of reference face images (300), for example, including face images of the same person, as a single group. The electronic device (100) may compare an acquired face image (200) with the plurality of reference face images (300) and include the face image (200) in an already grouped group or group it into a new group and provide the same to the user of the electronic device (100).

[0065] However, the present disclosure is not limited thereto, and the electronic device (100) may be implemented as an electronic device of various shapes, such as a set-top box or a server device. In this case, the electronic device (100) may obtain a facial image (200) from an external electronic device or an external server through an input / output interface or a communication interface. The electronic device (100) may also provide a similarity result or a clustering result determined by comparing a plurality of reference facial images (300) with the facial image (200) to an external electronic device or an external server through an input / output interface or a communication interface.

[0066] In addition, a plurality of reference face images (300) are stored in a database (DB) included in an external server, and the electronic device (100) can also obtain a plurality of reference face images (300) from an external server or an external electronic device through a communication interface.

[0067] Hereinafter, the electronic device (100) of the present disclosure and the operating method of the electronic device (100) will be described in detail with reference to FIGS. 2 to 13.

[0068] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0069] Referring to FIGS. 1 and 2, in one embodiment of the present disclosure, an electronic device (100) may include a display (120), a camera (130), a memory (110), at least one processor (140), an input / output interface (150), and a communication interface (160).

[0070] However, not all of the components illustrated in FIG. 2 are essential components. The electronic device (100) may be implemented with more components than those illustrated in FIG. 2, or may be implemented with fewer components.

[0071] The display (120), camera (130), memory (110), at least one processor (140), input / output interface (150), and communication interface (160) included in the electronic device (100) may each be electrically connected to each other.

[0072] In one embodiment of the present disclosure, at least one processor (140) can control the display (120) to display at least one of the clustered group (400), the acquired facial image (200), or the plurality of reference facial images (300). The display (120) can include any one of a liquid crystal display, a plasma display, an organic light emitting diode display, and an inorganic light emitting diode display. However, the present disclosure is not limited thereto, and the display (120) can include another type of display capable of displaying at least one of the clustered group (400), the acquired facial image (200), or the plurality of reference facial images (300).

[0073] In one embodiment of the present disclosure, the camera (130) can capture a physical environment space around the electronic device (100) or a user using the electronic device (100) to obtain a face image (200). In one embodiment of the present disclosure, the camera (130) may include an RGB camera capable of obtaining an image including RGB information. However, the present disclosure is not limited thereto, and the camera (130) may include a stereo camera including two RGB cameras, an RGB-Depth camera that obtains an image including RGB information and depth information, or a black-and-white camera that obtains a black-and-white image, and the like, but is not limited to any one of them. At least one processor (140) can obtain an image including a human face through the camera (130), and recognize the face from the obtained image to obtain a face image (200).

[0074] In one embodiment of the present disclosure, the memory (110) may store instructions, data structures, and program codes that can be read by at least one processor (140). In one embodiment of the present disclosure, there may be more than one memory (110). Operations performed by the electronic device (100) may be implemented by at least one processor (140) executing instructions or codes of a program stored in the memory (110).

[0075] In one embodiment of the present disclosure, the memory (110) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a Mask ROM, a Flash ROM, etc.), a hard disk drive (HDD), or a solid state drive (SSD).

[0076] In one embodiment of the present disclosure, the memory (110) may not exist separately and may be configured to be included in at least one processor (140).

[0077] In one embodiment of the present disclosure, the memory (110) may store instructions or program codes for performing functions or operations of the electronic device (100). The instructions, algorithms, data structures, program codes, and application programs stored in the memory (110) may be implemented in a programming or scripting language such as, for example, C, C++, Java, Python, or an assembler.

[0078] In one embodiment of the present disclosure, various types of modules that can be used to perform operations of the electronic device (100) may be stored in the memory (110).

[0079] In one embodiment of the present disclosure, a face recognition module (111), a feature extraction module (112), a feature selection module (114), an image segmentation module (115), and a similarity determination module (116) may be stored in the memory (110). However, not all of the modules illustrated in FIG. 2 are required. The memory (110) may store more or fewer modules than the modules illustrated in FIG. 2.

[0080] In one embodiment of the present disclosure, a 'module' included in the memory (110) may mean a unit that processes a function or operation performed by at least one processor (140). The 'module' included in the memory (110) may be implemented as software such as instructions, an algorithm, a data structure, or a program code.

[0081] In one embodiment of the present disclosure, the face recognition module (111) may include commands or program codes for performing an operation of recognizing a human face and obtaining a face image (200) from an image obtained through a camera (130).

[0082] The face recognition module (111) may include commands or program codes for performing an operation of detecting a human face from an image acquired through a camera (130) and segmenting the detected face to acquire a face image (200).

[0083] At least one processor (140) can obtain a facial image (200) from an image obtained through a camera (130) by executing commands or program codes of a facial recognition module (111).

[0084] However, the present disclosure is not limited thereto, and if the image acquired through the camera (130) is an image acquired by photographing with the face as the center, at least one processor (140) may not execute the face recognition module (111).

[0085] In one embodiment of the present disclosure, the feature extraction module (112) may include instructions or program codes for performing an operation of extracting a feature vector including features of the face image (200) from the face image (200). In one embodiment of the present disclosure, the feature vector may be a vector in a one-dimensional form to which a plurality of unit indices are assigned. The feature vector may include a feature value for each of the plurality of unit indices. The feature of the face image (200) means information for describing the face image (200), and the feature value may indicate the amount of information or the content of the information.

[0086] However, the present disclosure is not limited thereto, and the feature vector may be a two-dimensional vector.

[0087] In one embodiment of the present disclosure, the feature extraction module (112) may include an artificial intelligence model (113).

[0088] The artificial intelligence model (113) included in the feature extraction module (112) may include a pre-trained artificial intelligence model that receives a facial image (200) as input and outputs an image feature vector (230) in which a plurality of unit indices corresponding to each of a plurality of unit regions included in the facial image (200) are assigned. The artificial intelligence model included in the feature extraction module (112) is a DNN (Deep Neural Network), and may include a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), a Transformer model, etc., and may utilize an encoder. However, the artificial intelligence model in the present disclosure is not limited to the above-described example.

[0089] At least one processor (130) can obtain an image feature vector (230) from a facial image (200) by executing commands or program codes of a feature extraction module (112). At least one processor (130) can obtain an image feature vector (230) from a facial image (200) by using an artificial intelligence model (113).

[0090] Hereinafter, the description of the artificial intelligence model (113) will be described later with reference to FIGS. 4 and 5.

[0091] In one embodiment of the present disclosure, the feature selection module (114) may include instructions or program code for performing an operation of selecting a feature vector included in a specific unit index among a plurality of unit indices assigned to an image feature vector (230).

[0092] In one embodiment of the present disclosure, an image feature vector (230) obtained through a feature extraction module (112) may be assigned a plurality of unit indices corresponding to a plurality of unit regions included in a face image (200), respectively. A feature selection module (114) may include commands or program codes for performing an operation of selecting a feature vector included in at least one unit index corresponding to at least one unit region included in a specific region of a face image (200) among the image feature vectors (230).

[0093] At least one processor (130) can select a feature vector included in a specific unit index among image feature vectors (230) by executing instructions or program codes of the feature extraction module (112).

[0094] In one embodiment of the present disclosure, the image segmentation module (115) may be configured with commands or program codes related to an operation or function of segmenting an occlusion included in a face image (200) to obtain an occlusion segmentation map. The image segmentation module (115) may be configured with commands or program codes related to an operation or function of segmenting facial features included in a face image (200) to obtain a segmentation map in which facial features are distinguished. The image segmentation module (115) may include an artificial intelligence model that has been pre-learned to segment an object included in an image. The artificial intelligence model included in the image segmentation module (115) may be a DNN (Deep Neural Network), and may include a CNN (Convolutional Neural Network), a U-net, an RNN (Recurrent Neural Network), a Transformer model, etc., and the artificial intelligence model in the present disclosure is not limited to the above-described examples.

[0095] In one embodiment of the present disclosure, at least one processor (140) may obtain an occlusion segmentation map or a segmentation map in which facial features are segmented from a face image (200) by executing instructions or program codes of an image segmentation module (115). The present disclosure is not limited thereto, and it goes without saying that a segmentation map in which occlusions and facial features included in a face image (200) are segmented together may be obtained through the image segmentation module (115).

[0096] In one embodiment of the present disclosure, the similarity determination module (116) may be configured with instructions or program codes related to operations or functions for determining similarity between different feature vectors. The similarity determination module (116) may include a cosine similarity algorithm or a Euclidean distance algorithm, etc. However, the present disclosure is not limited thereto, and the similarity determination module (116) may of course include various algorithms that can be used to determine similarity between different feature vectors.

[0097] At least one processor (130) can determine the similarity between different feature vectors by executing instructions or program codes of the similarity determination module (116).

[0098] In one embodiment of the present disclosure, at least one processor (140) may be configured as one or more processors, which control a series of processes to allow the electronic device (100) to operate according to the embodiments described below.

[0099] In one embodiment of the present disclosure, at least one processor (140) may be configured as at least one of a Central Processing Unit, a microprocessor, a Graphic Processing Unit, an Application Processor (AP), an Application Specific Integrated Circuits (ASICs), a Digital Signal Processor (DSPs), a Digital Signal Processing Device (DSPDs), a Programmable Logic Device (PLDs), a Field Programmable Gate Array (FPGAs), a Communication Processor (CP), a Neural Processing Unit, or an artificial intelligence (AI) processor designed with a hardware structure specialized for learning and processing an artificial intelligence (AI) model, but is not limited thereto.

[0100] In one embodiment of the present disclosure, if one or more processors included in at least one processor (140) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0101] In one embodiment of the present disclosure, at least one processor (140) may be configured as a circuit, such as a System on Chip (SoC) or an Integrated Circuit (IC). At least one processor (130) may include a processing circuit.

[0102] In one embodiment of the present disclosure, at least one processor (140) can execute various types of modules stored in the memory (110). At least one processor (140) can execute at least one instruction constituting the various types of modules stored in the memory (110). By executing the program or at least one instruction stored in the memory (110), at least one processor (140) can process data according to predefined operating rules or artificial intelligence models.

[0103] In one embodiment of the present disclosure, at least one processor (140) may include multiple processors. In one embodiment of the present disclosure, at least one module among the multiple modules in the memory (110) may be executed by any one of the multiple processors. The remaining modules among the multiple modules stored in the memory (110) may be executed by other processors among the multiple processors.

[0104] In one embodiment of the present disclosure, at least one processor (140) can obtain a facial image (200) from an external electronic device or the like through an input / output interface (150).

[0105] In addition, at least one processor (140) can provide at least one reference face image corresponding to the face image (200) among the plurality of reference face images (300) or a result of determining similarity between the plurality of reference face images and the face image to an external electronic device through an input / output interface (150).

[0106] In one embodiment of the present disclosure, the input / output interface (150) can perform input / output operations with an external electronic device using at least one of input / output methods including an HDMI port (High-Definition Multimedia Interface port), a DVI (Digital Visual Interface), a component jack, a PC port, or a USB port (Universal Serial Bus port). However, the present disclosure is not limited to the above-described input / output methods.

[0107] In one embodiment of the present disclosure, the communication interface (160) can perform data communication with an external server or external electronic device under the control of at least one processor (140).

[0108] The communication interface (160) can perform data communication with an external server or an external electronic device using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, WFD (Wi-Fi Direct), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication.

[0109] In addition, at least one processor (140) can obtain a face image (200) from an external electronic device or an external server through a communication interface (160). At least one processor (140) can provide at least one reference face image corresponding to the face image (200) among a plurality of reference face images (300) or a result of determining similarity between the plurality of reference face images and the face image to an external electronic device or an external server through a communication interface (160).

[0110] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0111] Referring to FIGS. 1, 2 and 3, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S100) of acquiring a face image (200) including a first area (210) covered by a shield and a second area (220) not covered by the shield.

[0112] In the step (S100) of obtaining a face image (200), the electronic device (100) can obtain the face image (200) as at least one processor (140) controls the camera (130). However, the present disclosure is not limited thereto, and the electronic device (100) can also obtain the face image (200) from an external electronic device or an external server through an input / output interface (150) or a communication interface (160).

[0113] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S200) of obtaining an image feature vector (230) in which a occlusion feature vector (231) extracted from a first area (210) of the face image (200) is included in a first index and a person feature vector (232) extracted from a second area (220) of the face image (200) is included in a second index using an artificial intelligence model (113).

[0114] In the step (S200) of obtaining an image feature vector (232), at least one processor (140) executes commands or program codes of the feature extraction module (112), so that the electronic device (100) can obtain an image feature vector (230) from a face image (200) in which a occlusion feature vector (231) is included in a first index and a person feature vector (232) is included in a second index. The electronic device (100) can obtain an image feature vector (230) from a face image (200) in which a occlusion feature vector (231) is included in a first index and a person feature vector (232) is included in a second index through an artificial intelligence model (113) included in the feature extraction module (112).

[0115] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S300) of selecting a person feature vector (232) included in a second index among image feature vectors (230).

[0116] In the step (S300) of selecting a person feature vector (232) from among the image feature vectors (230), at least one processor (140) executes commands or program codes of the feature selection module (114), so that the electronic device (100) can select a person feature vector (232) included in the second index from among the image feature vectors (230).

[0117] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S400) of comparing a plurality of reference feature vectors (310) and a person feature vector (230) corresponding to each of a plurality of reference face images (300) to select at least one reference face image (301) corresponding to a face image (200) among the plurality of reference face images (300).

[0118] In the step (S400) of selecting at least one reference face image (301), at least one processor (140) executes commands or program codes of a similarity judgment module (116), so that the electronic device (100) can select at least one reference face image (301) corresponding to the face image (200) among a plurality of reference face images (300).

[0119] However, the present disclosure is not limited thereto. The operation in the step (S300) of selecting a character feature vector (232) and the operation in the step (S420) of selecting at least one reference face image (301) corresponding to the face image (200) among a plurality of reference face images (300) may not be divided into two steps, but may be performed in one step.

[0120] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step of comparing a person feature vector (232) included in a second index among image feature vectors (230) with a plurality of reference feature vectors (310) corresponding to each of a plurality of reference face images (300), thereby selecting at least one reference face image (301) corresponding to the face image (200) among the plurality of reference face images (300).

[0121] By having at least one processor (140) execute instructions or program codes of a feature selection module (114) and a similarity judgment module (116), the electronic device (100) can compare a person feature vector (232) included in a second index among image feature vectors (230) with a plurality of reference feature vectors (310) to select at least one reference face image (301) corresponding to a face image (200) among a plurality of reference face images (300).

[0122] FIG. 4 is a diagram for explaining a learning method of an artificial intelligence model according to one embodiment of the present disclosure.

[0123] Referring to FIGS. 1 and 4, in one embodiment of the present disclosure, an artificial intelligence model (113) is illustrated in FIG. 4. The artificial intelligence model (113) illustrated in FIG. 4 may be included in a feature extraction module (112). In one embodiment of the present disclosure, the artificial intelligence model (113) may include a plurality of neural network layers. Each of the plurality of neural network layers may include a plurality of weight values. The artificial intelligence model may perform an operation of a current neural network layer through an operation result of a previous neural network layer and an operation of a plurality of weight values.

[0124] In one embodiment of the present disclosure, the artificial intelligence model (113) may be a pre-trained artificial intelligence model that receives a facial image (200) as input using a training data set (500) and outputs an image feature vector (230) in which a plurality of unit indices corresponding to each of a plurality of unit regions included in the facial image (200) are assigned.

[0125] In one embodiment of the present disclosure, a training data set (500) may include a training image (501) including a plurality of unit regions (502), and a training masking image (502) including a masking pattern (503) in at least one unit region included in the training image (501).

[0126] In one embodiment of the present disclosure, a single unit area may be a basic unit for dividing an area of ​​a training image (501). In one embodiment of the present disclosure, when the entire area of ​​the training image (501) is divided into 4*4 areas, each of the 4*4 divided areas may be referred to as a unit area. However, the present disclosure is not limited thereto, and it goes without saying that the unit area may be changed depending on the degree to which the entire area of ​​the training image (501) is divided.

[0127] In one embodiment of the present disclosure, the masking pattern (503) may be a pattern that masks at least one unit area of ​​the training image (501). The masking pattern (503) may be included in one unit area, or in two or more adjacent unit areas. However, the present disclosure is not limited thereto, and the masking pattern (503) may also be included in two or more unit areas spaced apart from each other.

[0128] In one embodiment of the present disclosure, a training masking image (502) included in a training data set (500) may be an image including a masking pattern (503) in at least one unit area randomly selected from among a plurality of unit areas of a training image (501). The training masking image (502) may include a plurality of masking images.

[0129] In one embodiment of the present disclosure, the artificial intelligence model (113) may receive a training data set (500) as input and extract a training feature vector (520) from an image included in the training data set (500). At this time, the artificial intelligence model (113) may include an encoder (510). At this time, either a training image (501) or a training occlusion image (502) included in the training data set (500) may be provided as an input to the artificial intelligence model (113).

[0130] In one embodiment of the present disclosure, the training feature vector (520) may include a first feature vector (521) extracted from a training image (501). The training feature vector (520) may include a second feature vector (522) extracted from a training masking image (502). The artificial intelligence model (113) may extract a feature vector corresponding to an image provided as an input.

[0131] In one embodiment of the present disclosure, a specific index of the second feature vector (522) may include a feature value different from that of the first feature vector (521) due to a masking pattern (503) included in the training masking image (502). The specific index where the second feature vector (522) includes a feature value having a different value from that of the first feature vector (521) may be an index corresponding to the masking pattern (503). A specific index among the second feature vectors (522) may include a masking feature vector (523) corresponding to an area where the masking pattern (503) is included.

[0132] In one embodiment of the present disclosure, a training feature vector (520) obtained from an artificial intelligence model (113) may be provided to an image generation model (530) that generates an image based on the feature vector provided as an input. The image generation model (530) may include a pre-trained artificial intelligence model capable of generating an image from the feature vector. The artificial intelligence model included in the image generation model (530) is a DNN (Deep Neural Network), and may include a CNN, a Transformer, a VAE, etc., and may utilize a decoder.

[0133] In one embodiment of the present disclosure, an image generated by an image generation model (530) may be referred to as a generation data set (540). The generation data set (540) may include a first generation image (541) generated by providing a first feature vector (521) to the image generation model (530) and a second generation image (542) generated by providing a second feature vector (522) to the image generation model (530).

[0134] The first generated image (541) is an image generated based on the first feature vector (521) extracted from the training image (501), and may not include a masking pattern. The second generated image (542) is an image generated based on the second feature vector (522) extracted from the training shielding image (502), and may include a masking pattern (543).

[0135] In one embodiment of the present disclosure, the artificial intelligence model (113) can be learned through a process of updating a plurality of weights using a loss function based on a training data set (500) and a generation data set (540) generated using a training feature vector (520) extracted from the training data set (500).

[0136] In one embodiment of the present disclosure, the artificial intelligence model (113) may be trained through a process of updating a plurality of weights so that the training data set (500) and the generation data set (540) become similar by using the above-described loss function. Accordingly, the artificial intelligence model (113) may be trained so that the accuracy of the operation of extracting features of the remaining area excluding the masking pattern (503) is improved even if the training masking image (502) including the masking pattern (503) is provided. In addition, the artificial intelligence model included in the image generation model (530) may also be trained to generate a second generation image (542) similar to the training masking image (502) based on a second feature vector (522) extracted from the training masking image (502) including the masking pattern (503).

[0137] In one embodiment of the present disclosure, as the number and position of at least one unit region where the masking pattern (503) included in the training masking image (502) is located changes, the number and position of indices including the masking feature vector (523) among the second feature vectors (522) may also change. In this case, the masking feature vector (523) may mean a vector including a different feature value compared to the first feature vector (521). Accordingly, the artificial intelligence model (113) may be pre-trained to extract an image feature vector including a plurality of unit indices corresponding to each of a plurality of unit regions constituting the face image, from a face image provided as an input.

[0138] In one embodiment of the present disclosure, the image feature vector acquired through the artificial intelligence model (113) may include information regarding which unit index among a plurality of unit indices corresponds to which unit area among a plurality of unit areas constituting a facial image provided as an input. The electronic device (100) may select a feature vector included in a specific index corresponding to a specific area (e.g., an area not covered by a shielding object) of the facial image among the image feature vectors.

[0139] In one embodiment of the present disclosure, the training data set (500) may include a training segmentation map (560) obtained by segmenting a training face image (501). The training segmentation map (560) may include information segmenting facial features of the face included in the training face image (501).

[0140] In one embodiment of the present disclosure, during the learning process of the artificial intelligence model (113), a training segmentation map (560) may be provided to the artificial intelligence model (113) along with either a training image (501) or a training masking image (502) provided to the artificial intelligence model (113). Accordingly, information regarding where in the facial features of the face included in the training image (501) the masking area (503) included in the training masking image (502) is located may also be provided to the artificial intelligence model (113).

[0141] In one embodiment of the present disclosure, the artificial intelligence model (113) may be pre-trained using the learning method illustrated in FIG. 4. In one embodiment of the present disclosure, at least one processor (140) may train the artificial intelligence model (113) using the method illustrated in FIG. 4. The at least one processor (140) may also perform transfer learning and fine-tuning using a pre-trained model to train the artificial intelligence model.

[0142] However, the present disclosure is not limited thereto, and at least one processor (140) may receive a pre-learned artificial intelligence model from an external server or peripheral electronic devices through a communication interface (160) in the manner of FIG. 4.

[0143] FIG. 5 is a diagram for explaining an image feature vector obtained from a facial image through an artificial intelligence model according to one embodiment of the present disclosure.

[0144] Referring to FIGS. 1, 2, and 5, in one embodiment of the present disclosure, an artificial intelligence model (113) may receive as input a facial image (200) divided into a plurality of unit regions (500), and infer an image feature vector (230). The facial image (200) includes a first region (210) covered by a shield and a second region (220) not covered by the shield, and each of the first region (210) and the second region (220) may include at least one unit region.

[0145] In one embodiment of the present disclosure, the electronic device (100) may select a person feature vector (232) including a feature value extracted from a second region (220) of a face image (200) among the image feature vectors (230). At this time, the electronic device (100) may detect a occlusion among the face image (200), and may select the remaining feature vectors, excluding the occlusion feature vector (231) included in the first index corresponding to the first region (210) among the person feature vectors (232), as the person feature vector (232), by setting the region where the detected occlusion is located as the first region (210).

[0146] FIG. 6 is a flowchart illustrating an operation of comparing a person feature vector with a plurality of reference person feature vectors selected from a plurality of reference feature vectors according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 3 are assigned the same reference numerals, and redundant descriptions are omitted.

[0147] Referring to FIGS. 1, 2, 3, and 6, in one embodiment of the present disclosure, the method of operating the electronic device (100) may include a step (S410) of selecting a plurality of reference person feature vectors (312) included in a second index from each of a plurality of reference feature vectors (310).

[0148] In one embodiment of the present disclosure, the step (S410) of selecting a plurality of reference character feature vectors (312) may be an operation executed after step S300.

[0149] In one embodiment of the present disclosure, in the step (S410) of selecting a plurality of reference character feature vectors (312), at least one processor (140) executes instructions or program codes of a feature selection module (114), so that the electronic device (100) can select a plurality of reference character feature vectors (312) included in the second index from each of the plurality of reference feature vectors.

[0150] In one embodiment of the present disclosure, the plurality of reference feature vectors (310) may be feature vectors obtained using the same artificial intelligence model (113) as the artificial intelligence model (113) used to obtain the image feature vector (230) from the plurality of reference face images (300). Accordingly, each of the plurality of reference person feature vectors (312) selected from the second index of each of the plurality of reference feature vectors (310) may be a feature vector including a feature value extracted from the second region of each of the plurality of reference face images (300).

[0151] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S420) of comparing a plurality of reference person feature vectors (312) and a person feature vector (232) to select at least one reference face image (301) corresponding to a face image among a plurality of reference face images (300).

[0152] In one embodiment of the present disclosure, in step S420, at least one processor (140) executes instructions or program codes of a similarity judgment module (116), so that the electronic device (100) can compare a plurality of reference person feature vectors (312) and a person feature vector (232) to select at least one reference face image (301) corresponding to a face image among a plurality of reference face images (300).

[0153] In one embodiment of the present disclosure, each of a plurality of reference face images (300) may be preprocessed to have a similar pose and size. The plurality of reference feature vectors (300) may be obtained from the plurality of preprocessed reference face images.

[0154] In one embodiment of the present disclosure, the face image (200) may be preprocessed to have a pose and size similar to those of a plurality of preprocessed reference face images (300). The image feature vector (230) may be obtained from the preprocessed face image. Accordingly, the second region of the face image (200) and the second region of each of the plurality of reference face images (300) may be regions including the same facial features.

[0155] In one embodiment of the present disclosure, the first region (210) of the face image (200) is a region corresponding to the lower part of the face, which is an area covered by a shield, and the second region (220) of the face image (200) is a region corresponding to the upper part of the face, which may be an area including eyes, eyebrows, nose, hair, forehead, ears, etc.

[0156] In one embodiment of the present disclosure, the first region of each of the plurality of reference face images (300) may be a region corresponding to the lower part of the face, including the mouth, and the second region of each of the plurality of reference face images (300) may be a region corresponding to the upper part of the face, including the eyes, eyebrows, nose, hair, forehead, ears, etc.

[0157] Accordingly, by comparing the plurality of reference person feature vectors (312) and the person feature vector (232), the upper part of the face included in the face image (200) and the upper part of the face included in each of the plurality of reference face images (300) can be compared to select at least one reference face image (301) corresponding to the face image among the plurality of reference face images (300).

[0158] However, the present disclosure is not limited thereto. The operation in the step (S410) of selecting a plurality of reference person feature vectors (312) and the operation in the step (S420) of selecting at least one reference face image (301) corresponding to the face image (200) among the plurality of reference face images (300) may not be divided into two steps, but may be performed in one step.

[0159] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step of comparing a plurality of reference person feature vectors (312) included in a second index of each of the plurality of reference feature vectors (310) with a person feature vector (232) included in a second index among the image feature vectors (230), thereby selecting at least one reference face image (301) corresponding to the face image (200) among the plurality of reference face images (300).

[0160] By having at least one processor (140) execute instructions or program codes of a feature selection module (114) and a similarity judgment module (116), the electronic device (100) can compare a plurality of reference person feature vectors (312) included in a second index of each of a plurality of reference feature vectors (310) with a person feature vector (232) included in a second index among image feature vectors (230) to select at least one reference face image (301) corresponding to a face image (200) among a plurality of reference face images (300).

[0161] FIG. 7 is a flowchart illustrating an operation of obtaining a person feature vector from an image feature vector based on an occlusion segmentation map according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an operation of obtaining a person feature vector based on a two-dimensional image feature vector and an occlusion segmentation map according to an embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 3 are given the same reference numerals, and redundant descriptions are omitted.

[0162] Referring to FIGS. 1, 2, 3, 7, and 8, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S110) of obtaining a occlusion segmentation map (610) in which a occlusion is segmented from a face image (200). In one embodiment of the present disclosure, the step (S110) of obtaining the occlusion segmentation map (610) may be performed after step S100.

[0163] In the step (S110) of obtaining the occlusion segmentation map (610), at least one processor (140) executes commands or program codes of the image segmentation module (115), so that the electronic device (100) can obtain the occlusion segmentation map (610) in which the occlusion is segmented from the face image (200). In one embodiment of the present disclosure, the occlusion segmentation map (610) may be a map having a value of “0” in an area corresponding to the occlusion among the face image (200) and a value of “1” in an area not corresponding to the occlusion. The occlusion segmentation map (610) may be a map for masking an area corresponding to the occlusion.

[0164] However, the present disclosure is not limited thereto, and by having at least one processor (140) execute instructions or program codes of an image segmentation module (115), the electronic device (100) may obtain a segmentation map that segments facial features included in a masking object and an area not covered by the masking object from a face image (200). In this case, facial features included in an area not covered by the masking object may have different values.

[0165] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S210) of obtaining an image feature vector (600) including a plurality of unit indices each corresponding to a plurality of unit regions from a facial image (200) using an artificial intelligence model (113).

[0166] In step S210, by having at least one processor (140) execute commands or program codes of a feature extraction module (112), the electronic device (100) can obtain an image feature vector (600) including a plurality of unit indices each corresponding to a plurality of unit regions from a facial image (200) using an artificial intelligence model (113).

[0167] In one embodiment of the present disclosure, the occlusion segmentation map (610) obtained in step S110 and the image feature vector (600) obtained in step S210 may each be a two-dimensional vector. The two-dimensional image feature vector (600) may include a plurality of sub-image feature vectors (601). The plurality of sub-image feature vectors (601) may be feature vectors extracted to have different information from the face image (200) according to the layer structure of the artificial intelligence model included in the feature extraction module (112). Each of the plurality of sub-image feature vectors (601) may be a two-dimensional vector.

[0168] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S310) of obtaining a person feature vector (620) included in a second index corresponding to a second area not covered by an occlusion among image feature vectors (600) based on an occlusion segmentation map (610).

[0169] In step S310, by having at least one processor (140) execute instructions or program codes of a feature selection module (114), the electronic device (100) can obtain a person feature vector (620) included in a second index corresponding to a second area not covered by an occlusion among image feature vectors (600) based on an occlusion segmentation map (610).

[0170] Step S310 may include a step of obtaining a person feature vector (620) through an element-wise product of an image feature vector (600) and an occlusion segmentation map (610). The electronic device (100) may obtain a person feature vector (620) through an element-wise product of an image feature vector (600) and an occlusion segmentation map (610).

[0171] In one embodiment of the present disclosure, the image feature vector (600) and the occlusion segmentation map (610) may be resized to have the same resolution. The electronic device (100) may obtain a person feature vector (620) through an element-wise product of the two-dimensional image feature vector (600) and the occlusion segmentation map (610) having the same resolution.

[0172] In one embodiment of the present disclosure, a person feature vector (620) can be obtained through an element-wise product of each of a plurality of sub-image feature vectors (601) and an occlusion segmentation map (610).

[0173] In one embodiment of the present disclosure, the person feature vector (620) may be a two-dimensional vector. The person feature vector (620) may include a plurality of sub-person feature vectors (621). Each of the plurality of sub-person feature vectors (621) may be obtained by performing an element-wise product of each of the plurality of sub-image feature vectors (601) and the occlusion segmentation map (610).

[0174] FIG. 9 is a flowchart illustrating an operation of obtaining a human feature vector from among image feature vectors by comparing preset facial feature values ​​with feature values ​​according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 3 are assigned the same reference numerals, and redundant descriptions are omitted.

[0175] Referring to FIGS. 1, 2, 3, and 9, in one embodiment of the present disclosure, the step (S230) of obtaining an image feature vector (230) may include a step (S220) of assigning a plurality of unit indices (233) from a face image (200) using an artificial intelligence model (113), and obtaining an image feature vector (230) including a feature value for each of the plurality of unit indices (233).

[0176] In step S220, by having at least one processor (140) execute instructions or program codes of a feature extraction module (112), the electronic device (100) can obtain an image feature vector (230) in which a plurality of unit indices (233) are assigned from a face image (200) through an artificial intelligence model (113) and each of the plurality of unit indices (233) includes a feature value.

[0177] In one embodiment of the present disclosure, the feature value included in each of the plurality of unit indices (233) may be a value representing information of an image included in each of the plurality of unit regions of the face image (200). In one embodiment of the present disclosure, the feature value included in a unit index corresponding to a unit region where a human eye is located among the image feature vectors (230) may be a value including information about the shape, size, or color of the human eye.

[0178] In one embodiment of the present disclosure, the step (S300) of selecting a character feature vector (230) may include a step (S320) of comparing a preset face feature value and a feature value representing a face feature to obtain a character feature vector (232) included in a second index corresponding to a second region (220) among the image feature vectors (230).

[0179] In step S320, at least one processor (140) executes commands or program codes of a feature selection module (114), so that the electronic device (100) can compare preset facial feature values ​​representing facial features with feature values ​​of an image feature vector (230), thereby obtaining a person feature vector (232) included in a second index corresponding to a second region (220) among the image feature vectors (230).

[0180] In one embodiment of the present disclosure, the preset facial feature values ​​may mean a range of feature values ​​included in a feature vector when extracting the feature vector from an image including facial features of the face.

[0181] In one embodiment of the present disclosure, when extracting a feature vector from an image including a human eye, the feature value extracted may vary depending on the shape, size, color, etc. of the human eye, but may have a value included within a specific range (for example, a value between 0.1 and 1). That is, the feature value of the feature vector extracted from various human eye images may have a value between 0.1 and 1. In addition, the feature value extracted depending on the shape, size, color, etc. of the human mouth may vary, but may have a value included within a specific range (for example, a value between 2.0 and 3.0).

[0182] In one embodiment of the present disclosure, during the learning process of the artificial intelligence model (113), a training data set (500, see FIG. 4) may include a plurality of training images having different facial features. Accordingly, a facial feature value representing a facial feature may be obtained based on a feature value included in a plurality of training feature vectors each extracted from a plurality of training images including various facial features. The obtained facial feature value may be a specific range value indicating that the value is extracted from a human facial feature. A feature value that is different from the obtained facial feature value, for example, a value smaller than or larger than the facial feature value, may be determined to be extracted from another image rather than from a human facial feature.

[0183] In one embodiment of the present disclosure, the electronic device (100) may acquire facial feature values ​​acquired during the learning process of the artificial intelligence model (113) as preset facial feature values. However, the present disclosure is not limited thereto, and the electronic device (100) may also acquire preset facial feature values ​​from an external electronic device or an external server via a communication interface (160).

[0184] In one embodiment of the present disclosure, by having at least one processor (140) execute instructions or program codes of a feature selection module (114), the electronic device (100) can compare preset facial feature values ​​with feature values ​​of an image feature vector (230), thereby obtaining a person feature vector (232) including feature values ​​that are different from the facial feature values ​​among the image feature vectors (230).

[0185] At this time, the person feature vector (232) may be a feature vector included in the second index. The second index may be an index corresponding to the second area (220) covered by the occlusion among the face image (200). Since the person feature vector (232) includes feature values ​​extracted from the occlusion rather than the person's facial features, it may include feature values ​​that are smaller than or larger than the face feature values.

[0186] FIG. 10 is a flowchart illustrating an operation of comparing similarities of unit regions included in a second region by comparing a person feature vector with a reference person feature vector according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating an operation of comparing similarities of facial features included in a second region by comparing a person feature vector with a reference person feature vector according to an embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 3 are given the same reference numerals, and redundant descriptions are omitted.

[0187] Referring to FIGS. 1, 2, 3, and 10, in one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S500) of obtaining similarity for at least one unit area included in the second area (220) by comparing a person feature vector (232) included in a second index among image feature vectors (230) with a plurality of reference person feature vectors (312) included in a second index among a plurality of reference feature vectors (310). In one embodiment of the present disclosure, step S500 may be performed after step S410.

[0188] In step S500, at least one processor (140) executes commands or program codes of a similarity judgment module (116), so that the electronic device (100) can obtain similarity for at least one unit area included in the second area (220) by comparing a person feature vector (232) included in a second index among image feature vectors (230) with a plurality of reference person feature vectors (312) included in a second index among a plurality of reference feature vectors (310).

[0189] In one embodiment of the present disclosure, any one of the unit indices assigned to the person feature vector (232) and the unit index that is identical to any one of the unit indices of each of the plurality of reference person feature vectors (312) may include feature values ​​extracted from the same unit area included in each of the face image (200) and the plurality of reference face images (300).

[0190] Accordingly, by comparing the person feature vector (232) with the plurality of reference person feature vectors (312), it is possible to obtain not only the overall similarity between the second region of the plurality of reference face images (300) and the second region of the face image (200), but also the similarity for at least one unit region included in the second region of the plurality of reference face images (300) and the second region of the face image (200).

[0191] Referring to FIG. 11, in one embodiment of the present disclosure, an image feature vector (230) may be assigned N unit indices. At this time, a first sub-feature vector (701) included in a first sub-index among the image feature vectors (230) may include a feature value extracted from an image including an eye (700) among the face images (200). At this time, the first sub-index may include unit indices from 1 to 3.

[0192] The second sub-feature vector (711) included in the second sub-index among the image feature vectors (230) may include feature values ​​extracted from an image including a nose (710) among the face images (200). At this time, the second sub-index may include unit indices from 4 to 5. The second sub-feature vector (721) included in the third sub-index among the image feature vectors (230) may include feature values ​​extracted from an image including a mouth (720) among the face images (200). At this time, the third sub-index may include unit indices from 6 to 7.

[0193] In one embodiment of the present disclosure, a plurality of reference feature vectors (310) included in a memory (110) may each be assigned N unit indices. At this time, a plurality of first reference sub-feature vectors (742) included in a first sub-index of each of the plurality of reference feature vectors (310) may include feature values ​​extracted from an image including an eye (741) among a plurality of reference face images (300).

[0194] In one embodiment of the present disclosure, a plurality of reference face images (300) and a face image (200) that have the same resolution and are preprocessed to have the same pose of a person may have the same facial features located in the same area, respectively. In one embodiment of the present disclosure, an area where an eye (741) is located in the plurality of preprocessed reference face images (300) and an area where an eye (700) is located in the face image (200) may be the same. The first sub-index may be an index that includes a feature vector extracted from an image of an area where an eye (741) is located in the plurality of reference face images (300) and an image of an area where an eye (700) is located in the face image (200).

[0195] In one embodiment of the present disclosure, at least one processor (140) executes instructions or program codes of a similarity judgment module (116), so that the electronic device (100) may obtain similarity by comparing the first sub-feature vector (701) with a plurality of first reference sub-feature vectors (742), and comparing the included eye (700) of the face image (200) with the eye (741) of each of the plurality of reference face images (300).

[0196] The present disclosure is not limited thereto, and the electronic device (100) may obtain a similarity between a specific facial feature included in a second area (220) of a face image (200) and a specific facial feature included in a second area of ​​a plurality of reference face images (300) by comparing a feature vector included in a specific index among the character feature vectors (232) with a feature vector included in a specific index among a plurality of reference character feature vectors (312).

[0197] Accordingly, in addition to obtaining at least one reference face image (301) determined to be the same or similar to the face image (200) among the plurality of reference face images (300), the electronic device (100) may also obtain at least one reference face image (301) having similar or identical specific facial features to specific facial features of the face image (200) among the plurality of reference face images (300).

[0198] FIG. 12 is a flowchart illustrating an operation of grouping selected reference face images and face images into a single group according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 3 are assigned the same reference numerals, and any redundant descriptions are omitted.

[0199] Referring to FIGS. 1, 2, 3, and 12, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S500) of grouping at least one selected reference face image (301) and a face image (200) into one group (400). In this case, step S500 may be performed after step S400.

[0200] In step S500, at least one processor (140) executes commands or program codes of a similarity judgment module (116), thereby grouping at least one reference face image (301) selected as corresponding to the face image (200) among a plurality of reference face images (300) and the face image (200) into one group (400).

[0201] In one embodiment of the present disclosure, a group (400) may refer to a group that is displayed as the same person when a plurality of reference face images (300) are provided to a user of an electronic device (100) in the form of a user interface such as a gallery. However, the present disclosure is not limited thereto, and a group (400) may also refer to a higher-level classification method that recognizes a plurality of reference face images (300) stored in a memory (110) as the same person and stores them in a single folder.

[0202] In one embodiment of the present disclosure, when the similarity between at least one reference face image (301) and the face image (200) is greater than a preset clustering threshold, the electronic device (100) may cluster the at least one reference face image (301) and the face image (200) into one group. At this time, the similarity between the at least one reference face image (301) and the face image (200) may be a value that increases as the degree of similarity between a face included in the at least one reference face image (301) and a face included in the face image (200) increases. The preset clustering threshold may be a value set based on a similarity obtained by comparing person images obtained by photographing the same person from various angles or positions.

[0203] Additionally, commands or program codes for performing an operation of clustering at least one reference face image (301) and a face image (200) into one group (400) may be stored in a separate module distinct from the similarity judgment module (116).

[0204] FIG. 13 is a diagram illustrating an interface for proposing clustering based on similarity according to one embodiment of the present disclosure.

[0205] Referring to FIGS. 1, 2 and 13, in one embodiment of the present disclosure, FIG. 13 illustrates a user interface that displays a process of clustering using similarities obtained through comparison between a facial image (200) by an electronic device (100) and a plurality of reference facial images (300).

[0206] Hereinafter, for convenience of explanation, both the multiple reference face images (300) and the face image (200) will be referred to as multiple photographic images. By using the similarity obtained by comparing the multiple photographic images with each other, the electronic device (100) can cluster at least one photographic image determined to include the same person so that it is included in the same group.

[0207] In one embodiment of the present disclosure, the user interface illustrated in FIG. 13 shows a plurality of groups in which a plurality of photographic images are clustered based on similarity determination results. In one embodiment of the present disclosure, the plurality of groups may include a first group (800), a second group (810), a third group (820), and a fourth group (850).

[0208] In one embodiment of the present disclosure, a first group (800) may be a group comprising at least one photo image determined to include a first user. A second group (810) may be a group comprising at least one photo image determined to include a second user. A third group (820) may be a group comprising at least one photo image determined to include a third user.

[0209] In one embodiment of the present disclosure, the second user may include a face having an area obscured by a covering (e.g., a mask). The electronic device (100) may obtain a first image feature vector from a photographic image including the first user. The electronic device (100) may obtain a second image feature vector from a photographic image including the second user.

[0210] In one embodiment of the present disclosure, the electronic device (100) may select a person feature vector included in a specific index corresponding to an area not covered by a occlusion among the second image feature vectors. The electronic device (100) may select a person feature vector included in the feature index among the first image feature vectors.

[0211] In one embodiment of the present disclosure, the electronic device (100) may display a first notification (830) such as "They appear to be the same person, but I'm not sure because their faces are covered. Should I show you that they are the same person?" on the user interface when the electronic device (100) compares a person feature vector selected from a first image feature vector with a person feature vector selected from a second image feature vector and obtains a similarity that is greater than a preset first clustering threshold and less than a preset second clustering threshold. When the electronic device (100) obtains an input to cluster the first user and the second user as the same person through the first notification (830), the electronic device (100) may cluster a photo image including the first user and a photo image including the second user into a fourth group (850) and display the clustered photo image on the user interface.

[0212] At this time, the second clustering threshold may be set to a value greater than the first clustering threshold. The second clustering threshold may be a value such that the similarity between the people included in the two images is sufficiently high that the probability that the people included in the two images are the same person is at least 90%. The first clustering threshold may be a value such that the probability that the people included in the two images are the same person is at least 80%. However, the above probabilities are merely examples, and the present disclosure is not limited thereto.

[0213] At this time, the electronic device (100) may provide the first notification (830) together with the thumbnail (801) of the first user and the thumbnail (811) of the second user to help the user determine whether or not there is a cluster.

[0214] However, the present disclosure is not limited thereto. In one embodiment of the present disclosure, the electronic device (100) may cluster the photo images including the first user and the photo images including the second user into one group (850) based on the similarity obtained by comparing the person feature vector selected from the first image feature vector and the person feature vector selected from the second image feature vector, which is equal to or greater than a preset second clustering threshold. In this case, the electronic device (100) may cluster the photo images including the first user and the photo images including the second user into one group (850) and provide the group to the user without providing the first notification (830).

[0215] In one embodiment of the present disclosure, the electronic device (100) may obtain a third image feature vector from a photograph image including a third user. The electronic device (100) may compare the first image feature vector with the third image feature vector. The electronic device (100) may compare a feature vector included in a specific index of the first image feature vector with a feature vector included in a specific index of the third image feature vector. In this case, the specific index may refer to an index corresponding to a facial feature. The feature vector included in the specific index may refer to a feature vector obtained from each facial feature.

[0216] In one embodiment of the present disclosure, the electronic device (100) can determine that the first user and the third user are not the same person when the similarity obtained by comparing the first image feature vector and the third image feature vector is less than the first clustering threshold.

[0217] However, the electronic device (100) may determine that the facial features of the first user and the facial features of the third user are similar when the obtained similarity is greater than a preset region clustering threshold by comparing the feature vector included in a specific index of the first image feature vector with the feature vector included in a specific index of the third image feature vector. When the electronic device (100) determines that the facial features of the first user and the facial features of the third user are similar, the electronic device (100) may display a second notification (840) on the user interface, such as "The shapes of the eyes, nose, and mouth are all similar! Shall I register you as brothers?", to confirm whether the first user and the third user should be clustered as relatives such as brothers and sisters.

[0218] The electronic device (100) may display a connection interface (860) for grouping the fourth group (850) and the third group (820) into one on the user interface as it obtains an input to group the first user and the third user into relatives through the second notification (840).

[0219] At this time, the region clustering threshold may be a preset value such that the facial features of the two people included in the two images have a sufficiently high similarity so that the two people can be determined to be related. In one embodiment of the present disclosure, the region clustering threshold may be set to a value when the facial features of the two people have a similarity of 80%. However, the above probability is merely an example, and the present disclosure is not limited thereto.

[0220] The electronic device (100) of the present disclosure, through the above-described description, can provide images of the same person to a user by grouping them together, even if the face is obscured by a masking material. Furthermore, even if the individuals are different, if they have similar facial features and are in a special relationship, such as a kinship, a separate connection relationship can be displayed and provided, thereby enhancing user convenience.

[0221] In order to solve the above-described technical problem, in one embodiment of the present disclosure, an electronic device is provided. The electronic device may include a memory storing at least one instruction. The electronic device may include at least one processor including a processing circuit. By having at least one processor individually or collectively execute at least one instruction stored in the memory, the electronic device may obtain a face image including a first area covered by an occluder and a second area not covered by the occluder. By having at least one processor individually or collectively execute at least one instruction, the electronic device may use an artificial intelligence model to obtain an image feature vector, in which an occlusion feature vector extracted from a first area of ​​the face image is included in a first index, and a person feature vector extracted from a second area of ​​the face image is included in a second index. By having at least one processor individually or collectively execute at least one command, the electronic device can compare a person feature vector included in a second index among the image feature vectors with a plurality of reference feature vectors corresponding to each of the plurality of reference face images, thereby selecting at least one reference face image corresponding to the face image among the plurality of reference face images.

[0222] In one embodiment of the present disclosure, the plurality of reference feature vectors may be feature vectors extracted from a plurality of reference face images using an artificial intelligence model. At least one processor may individually or collectively execute at least one command, so that the electronic device may compare a plurality of reference person feature vectors included in a second index from each of the plurality of reference feature vectors with a person feature vector included in a second index among the image feature vectors, thereby selecting at least one reference face image corresponding to the face image from among the plurality of reference face images.

[0223] In one embodiment of the present disclosure, a face image may include a plurality of unit regions including a first region and a second region. At least one processor may individually or collectively execute at least one command, thereby allowing an electronic device to obtain an occlusion segmentation map in which occlusions are segmented from the face image. At least one processor may individually or collectively execute at least one command, thereby allowing the electronic device to obtain an image feature vector in which a plurality of unit indices corresponding to the plurality of unit regions are assigned, respectively, from the face image using an artificial intelligence model. At least one processor may individually or collectively execute at least one command, thereby allowing the electronic device to obtain a person feature vector included in a second index corresponding to a second region that is not occluded by an occlusion among the image feature vectors, based on the occlusion segmentation map.

[0224] In one embodiment of the present disclosure, each of the image feature vector and the occlusion segmentation map may be a two-dimensional vector. By having at least one processor individually or collectively execute at least one instruction, the electronic device may obtain a person feature vector through an element-wise product of the image feature vector and the occlusion segmentation map.

[0225] In one embodiment of the present disclosure, a face image may include a plurality of unit regions including a first region and a second region. At least one processor may individually or collectively execute at least one command, so that the electronic device may obtain an image feature vector in which a plurality of unit indices corresponding to the plurality of unit regions are assigned from the face image through an artificial intelligence model, and each of the plurality of unit indices includes a feature value. At least one processor may individually or collectively execute at least one command, so that the electronic device may compare the feature values ​​with preset facial feature values ​​representing features of the face, and obtain a person feature vector included in a second index corresponding to a second region that is not covered by an occlusion among the image feature vectors.

[0226] In one embodiment of the present disclosure, a face image may include a plurality of unit regions, and each of a plurality of reference face images may include a plurality of unit regions. An image feature vector may be assigned a plurality of unit indices corresponding to the plurality of unit regions, respectively. Each of the plurality of reference feature vectors may be assigned a plurality of unit indices corresponding to the plurality of unit regions, respectively. By having at least one processor individually or collectively execute at least one command, an electronic device may compare a person feature vector included in a second index among the image feature vectors with a plurality of reference person feature vectors included in the second index among the plurality of reference feature vectors, thereby obtaining a similarity for at least one unit region included in the second region.

[0227] In one embodiment of the present disclosure, the artificial intelligence model may be a pre-trained artificial intelligence model that receives a facial image as input using a training data set and outputs an image feature vector in which a plurality of unit indices corresponding to each of a plurality of unit regions included in the facial image are assigned. The training data set may include a training facial image including a plurality of unit regions and a training masking image including a masking pattern in at least one unit region included in the training image.

[0228] In one embodiment of the present disclosure, the training data set may include a training segmentation map obtained by segmenting a training face image.

[0229] In one embodiment of the present disclosure, the artificial intelligence model may be a pre-trained artificial intelligence model using a loss function based on a training data set and a generation data set generated using training feature vectors extracted from the training data set.

[0230] In one embodiment of the present disclosure, the electronic device can cluster at least one selected reference face image and the face images into one group by having at least one processor individually or collectively execute at least one instruction.

[0231] In order to solve the above-described technical problem, one embodiment of the present disclosure provides an operating method of an electronic device. The operating method of the electronic device may include a step of obtaining a face image including a first area occluded by an occluder and a second area not occluded by the occluder. The operating method of the electronic device may include a step of obtaining an image feature vector, using an artificial intelligence model, in which an occlusion feature vector extracted from a first area of ​​the face image is included in a first index and a person feature vector extracted from a second area of ​​the face image is included in a second index. The operating method of the electronic device may include a step of comparing a person feature vector included in the second index among the image feature vectors with a plurality of reference feature vectors corresponding to each of a plurality of reference face images, thereby selecting at least one reference face image corresponding to the face image from among the plurality of reference face images.

[0232] In one embodiment of the present disclosure, the plurality of reference feature vectors may be feature vectors extracted from a plurality of reference face images using an artificial intelligence model, respectively. The step of selecting at least one reference face image may include the step of comparing a plurality of reference person feature vectors included in a second index from each of the plurality of reference feature vectors with a person feature vector included in a second index among the image feature vectors, thereby selecting at least one reference face image corresponding to the face image from among the plurality of reference face images.

[0233] In one embodiment of the present disclosure, a face image may include a plurality of unit regions including a first region and a second region. A method of operating an electronic device may include a step of obtaining an occlusion segmentation map in which an occlusion is segmented from the face image. The step of obtaining an image feature vector may include a step of obtaining an image feature vector to which a plurality of unit indices corresponding to each of the plurality of unit regions are assigned using an artificial intelligence model. The step of selecting a person feature vector may include a step of obtaining a person feature vector included in a second index corresponding to a second region that is not occluded by an occlusion among the image feature vectors based on the occlusion segmentation map.

[0234] In one embodiment of the present disclosure, each of the image feature vector and the occlusion segmentation map may be a two-dimensional vector. The step of selecting the person feature vector may include a step of obtaining the person feature vector through an element-wise product of the image feature vector and the occlusion segmentation map.

[0235] In one embodiment of the present disclosure, a face image may include a plurality of unit regions including a first region and a second region. The step of obtaining an image feature vector may include a step of assigning a plurality of unit indices corresponding to the plurality of unit regions from the face image through an artificial intelligence model, and obtaining an image feature vector including a feature value for each of the plurality of unit indices. The step of selecting a person feature vector may include a step of comparing a preset facial feature value representing a feature of a face with the feature value, and obtaining a person feature vector included in a second index corresponding to a second region that is not covered by an occlusion among the image feature vectors.

[0236] In one embodiment of the present disclosure, a face image may include a plurality of unit regions. Each of a plurality of reference face images may include a plurality of unit regions. An image feature vector may be assigned a plurality of unit indices corresponding to the plurality of unit regions, respectively. Each of the plurality of reference feature vectors may be assigned a plurality of unit indices corresponding to the plurality of unit regions, respectively. An operating method of an electronic device may include a step of comparing a person feature vector included in a second index among the image feature vectors with a plurality of reference person feature vectors included in the second index among the plurality of reference feature vectors, thereby obtaining a similarity for at least one unit region included in the second region.

[0237] In one embodiment of the present disclosure, the artificial intelligence model may be a pre-trained artificial intelligence model that receives a facial image as an input using a training data set and outputs an image feature vector in which a plurality of unit indices corresponding to each of a plurality of unit regions included in the facial image are assigned. The training data set may include a training facial image including a plurality of unit regions, a training masking image including a masking pattern in at least one unit region included in the training image, and a training segmentation map obtained by segmenting the training facial image.

[0238] In one embodiment of the present disclosure, the artificial intelligence model may be a pre-trained artificial intelligence model using a loss function based on a training data set and a generation data set generated using training feature vectors extracted from the training data set.

[0239] In one embodiment of the present disclosure, a method of operating an electronic device may include a step of clustering at least one selected reference face image and the face images into one group.

[0240] In order to solve the above-described technical problem, a computer-readable recording medium having recorded thereon a program for performing at least one method of an embodiment of an operating method of an electronic device disclosed in the present disclosure on a computer can be provided.

[0241] The program executed by the electronic device described in this disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0242] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0243] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The storage media may be readable by a computer, stored in a memory, and executed by a processor.

[0244] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage media" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage media and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0245] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0246] A computer program product may include a software program and a computer-readable storage medium storing the software program. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by an electronic device manufacturer or through an electronic marketplace (e.g., the Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the electronic device manufacturer, a server of the electronic marketplace, or a storage medium of an intermediary server that temporarily stores the software program.

[0247] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. A memory (110) storing at least one instruction; and At least one processor (140) comprising a processing circuit, The electronic device (100) executes the at least one processor (140) individually or collectively the at least one instruction stored in the memory (110), Acquire a face image (200) including a first area covered by an occluder and a second area not covered by the occluder, Using an artificial intelligence model, an image feature vector (feature vector, 230) is obtained in which a occlusion feature vector extracted from the first region of the face image (200) is included in a first index, and a person feature vector extracted from the second region of the face image is included in a second index. An electronic device (100) that compares the person feature vector included in the second index among the image feature vectors (230) with a plurality of reference feature vectors corresponding to each of a plurality of reference face images, and selects at least one reference face image (301) corresponding to the face image among the plurality of reference face images.

2. In paragraph 1, The above plurality of reference feature vectors are feature vectors extracted from the plurality of reference face images using the artificial intelligence model, The above electronic device (100) is, An electronic device (100) that compares a plurality of reference person feature vectors included in the second index from each of the plurality of reference feature vectors and the person feature vector included in the second index among the image feature vectors (230) to select at least one reference face image (301) corresponding to the face image (200) from among the plurality of reference face images.

3. In either of paragraphs 1 or 2, The above facial image (200) includes a plurality of unit areas including the first area and the second area, The above electronic device, Obtaining a segmentation map in which the occlusion is segmented from the above face image (200), Using the artificial intelligence model, the image feature vector (230) is obtained from the facial image (200) to which a plurality of unit indices corresponding to each of the plurality of unit regions are assigned, An electronic device (100) that obtains the person feature vector included in the second index corresponding to the second area that is not covered by the occlusion among the image feature vectors (230) based on the occlusion segmentation map.

4. In paragraph 3, Each of the above image feature vector (230) and the above occlusion segmentation map is a two-dimensional vector, By having the at least one processor (140) execute the at least one instruction, the electronic device (100) An electronic device (100) that obtains the person feature vector through an element-wise product of the image feature vector (230) and the occlusion segmentation map.

5. In either of paragraphs 1 or 2, The above facial image (200) includes a plurality of unit areas including the first area and the second area, The above electronic device, Through the artificial intelligence model, a plurality of unit indices corresponding to the plurality of unit areas are assigned from the facial image (200), and the image feature vector (230) including a feature value is obtained for each of the plurality of unit indices. An electronic device (100) that obtains the person feature vector included in the second index corresponding to the second area that is not covered by the occlusion among the image feature vectors by comparing the preset facial feature values ​​representing the features of the face with the feature values.

6. In any one of paragraphs 1 to 5, The above facial image (200) includes a plurality of unit areas, and each of the plurality of reference facial images includes the plurality of unit areas, The above image feature vector (230) is assigned a plurality of unit indices corresponding to each of the plurality of unit areas, Each of the plurality of reference feature vectors is assigned a plurality of unit indices corresponding to each of the plurality of unit areas, The above electronic device (100) is, An electronic device (100) that obtains similarity for at least one unit area included in the second area by comparing the person feature vector included in the second index among the image feature vectors (230) with the plurality of reference person feature vectors included in the second index among the plurality of reference feature vectors.

7. In any one of paragraphs 1 to 6, The above artificial intelligence model, An artificial intelligence model that is trained to receive the facial image (200) as input using a training data set and output the image feature vector (230) in which a plurality of unit indices corresponding to each of a plurality of unit regions included in the facial image (200) are assigned, An electronic device (100) wherein the training data set comprises a training face image including a plurality of unit regions and a training shield image including a masking pattern in at least one unit region included in the training image.

8. In paragraph 7, The above training data set is an electronic device (100) including a training segmentation map obtained by segmenting the training face image.

9. In either of paragraphs 7 or 8, The above artificial intelligence model, An electronic device (100) which is an artificial intelligence model that is pre-learned using a loss function based on the above training data set and a generation data set generated using the training feature vector extracted from the above training data set.

10. In any one of paragraphs 1 to 9, The above electronic device (100) is, An electronic device (100) that clusters at least one selected reference face image and the face image (200) into one group.

11. In the operating method of an electronic device (100), A step (S100) of acquiring a facial image (200) including a first area covered by an occluder and a second area not covered by the occluder; A step (S200) of obtaining an image feature vector (feature vector, 230) using an artificial intelligence model, wherein a occlusion feature vector extracted from the first area of ​​the face image (200) is included in a first index, and a person feature vector extracted from the second area of ​​the face image (200) is included in a second index; An operating method of an electronic device, comprising a step (S400) of comparing the person feature vector included in the second index among the image feature vectors (230) with a plurality of reference feature vectors corresponding to each of a plurality of reference face images, and selecting at least one reference face image corresponding to the face image (200) among the plurality of reference face images.

12. In paragraph 11, The above plurality of reference feature vectors are feature vectors extracted from the plurality of reference face images using the artificial intelligence model, The step (S400) of selecting at least one reference face image (301) is as follows: An operating method of an electronic device (100), comprising a step of comparing a plurality of reference person feature vectors included in the second index of each of the plurality of reference feature vectors with the person feature vector included in the second index among the image feature vectors (230), thereby selecting at least one reference face image (301) corresponding to the face image (200) among the plurality of reference face images.

13. In either of paragraphs 11 or 12, The above facial image (200) includes a plurality of unit areas including the first area and the second area, The operating method of the above electronic device (100) is: It further includes a step of obtaining a segmentation map in which the occlusion is segmented from the above face image (200), In the step (S200) of obtaining the above image feature vector (230), Using the artificial intelligence model, a step of obtaining the image feature vector (230) to which a plurality of unit indices corresponding to the plurality of unit areas are assigned from the facial image (200), In the step (S300) of selecting the above character feature vector, An operating method of an electronic device (100), comprising a step of obtaining the person feature vector included in the second index corresponding to the second area not covered by the occlusion among the image feature vectors (230), based on the occlusion segmentation map.

14. In paragraph 13, Each of the above image feature vector (230) and the above occlusion segmentation map is a two-dimensional vector, In the step (S300) of selecting the above character feature vector, An operating method of an electronic device (100), comprising a step of obtaining the person feature vector through an element-wise product of the image feature vector (230) and the occlusion segmentation map.

15. A computer-readable recording medium having recorded thereon a program for performing the method of operation described in any one of Articles 11 to 14 on a computer.

Images