Image processing method and apparatus

Abstract

To provide an image processing method and apparatus.SOLUTION: A method disclosed herein comprises detecting facial keypoints from an input face image, determining a face area of the input face image and a facial feature area of the input face image based on the facial keypoints, and determining the input face image to be an invalid face image when the facial feature area satisfies a first preset condition.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The following embodiments relate to an image processing method and apparatus. [Background technology]

[0002] The technical automation of the recognition process is realized, for example, through an artificial intelligence (e.g., neural network model) model embodied in a processor as a specialized computational structure, which after considerable training can provide a computationally intuitive mapping between input and output patterns. The trained ability to generate such a mapping is said to be the learning ability of the artificial intelligence. Moreover, with specialized training, such a specialized trained artificial intelligence may have a generalization ability to generate relatively accurate outputs for input patterns that it has not been trained on, for example. Such artificial intelligence techniques are used for identity recognition, for example, face recognition. Summary of the Invention [Means for solving the problem]

[0003] According to one embodiment, the image processing method includes the operations of detecting facial key points from an input facial image, determining a face area of ​​the input facial image and a facial feature area of ​​the input facial image based on the facial key points, and determining the input facial image as an invalid facial image if the facial feature area satisfies a first preset condition, the first preset condition including a shape condition regarding a shape of the facial feature area, a position condition regarding a relationship between the position of the facial feature area and the position of the facial area, or a combination thereof.

[0004] The electronic device includes one or more processors and a memory storing instructions, and when the instructions are executed by the one or more processors, the electronic device performs operations of detecting facial key points from an input facial image, determining a face area of ​​the input facial image and a facial feature area of ​​the input facial image based on the facial key points, and determining the input facial image as an invalid facial image if the facial feature area satisfies a first preset condition, the first preset condition including a shape condition regarding a shape of the facial feature area, a position condition regarding a relationship between the position of the facial feature area and the position of the facial area, or a combination thereof.

[0005] According to one embodiment, the image processing method includes detecting key points from an input image, determining a region of the input image and a feature region of the input image based on the key points, and determining the input image as an invalid image based on an aspect ratio of the feature region, a ratio between the distance between a central axis of the height direction of the region of the input image and a central axis of the height direction of the feature region and a width of the input image, or a combination thereof. Effect of the Invention

[0006] According to the present invention, an image processing method and apparatus can be provided. [Brief description of the drawings]

[0007] [Figure 1] 1 is a flowchart illustrating an image processing method according to an embodiment. [Diagram 2] 3 is a diagram illustrating an example of facial feature regions and face regions of an input face image according to an embodiment; [Diagram 3] FIG. 2 is a diagram illustrating an example of a face recognition operation using a face classification model according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a face recognition operation using a face classification model according to an embodiment. [Diagram 5] 4 is a diagram illustrating an example of a face region box and a face region of an input face image according to an embodiment. [Figure 6]11 is a diagram illustrating a face recognition operation using a preset condition according to an embodiment; [Figure 7] FIG. 1 is a diagram illustrating an example configuration of an electronic device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The specific structural or functional description of the embodiments is disclosed for the purpose of illustration only, and may be modified in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.

[0009] Although terms such as first or second may be used to describe multiple components, such terms should be construed only for the purpose of distinguishing one component from the other components. For example, a first component may be named a second component, and similarly, a second component may be named a first component.

[0010] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.

[0011] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the terms "comprise" or "have" and the like indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0012] As used herein, phrases such as "at least one of A or B" and "at least one of A, B, or C" may include any one or all possible combinations of the items listed with that phrase.

[0013] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as ideal or overly formal unless expressly defined in this specification.

[0014] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the drawings, the same components are given the same reference numerals regardless of the reference numerals, and the duplicated description thereof will be omitted.

[0015] FIG. 1 is a flowchart illustrating an example of an image processing method according to an embodiment, and FIG. 2 is a diagram illustrating an example of facial feature regions and face regions of an input face image according to an embodiment.

[0016] According to an embodiment, the face recognition may include face detection, face keypoint detection, face alignment, face feature extraction, face feature comparison, or a combination thereof. An artificial intelligence model (e.g., a neural network model) may be used for face recognition. For example, the artificial intelligence model may include a face detection model, a face keypoint detection model, a face classification model, or a combination thereof. The performance of face feature extraction is affected by the quality of the input face image. For example, when the quality of the input face image is low, a face may be misrecognized. For example, there may be a case where the face area is very small in the input face image, a case where the face deflection angle is very large in the input face image, a case where the front face detection from the input face image fails, or a case where the image quality of the input face image is low enough to make it difficult to identify the face, but is not limited thereto.

[0017] According to an embodiment, the accuracy of face recognition may be improved by filtering out inappropriate input face images during face detection. For example, the accuracy of face recognition may be improved by filtering out low quality face images and / or non-face images.

[0018] Referring to FIG. 1, in operation 110, the electronic device detects facial keypoints from the input facial image. The electronic device can detect facial keypoints from the input facial image using a facial keypoint detection model. For example, the facial keypoint detection model may be an artificial intelligence model (e.g., a neural network model) trained to detect facial keypoints from the input facial image. In this case, the facial keypoint detection model is pre-trained to detect facial keypoints from the input facial image via machine learning. The facial keypoints include points used to identify the location of facial features. For example, facial features include, but are not limited to, facial contours, eyes, pupils, eyebrows, nose, mouth, ears, etc.

[0019] In operation 120, the electronic device determines a facial region of the input facial image based on the facial key points and a facial feature region of the input facial image. The facial feature region may be a salient area. The electronic device can determine a region corresponding to a facial feature based on the region in which the facial key points are located.

[0020] The electronic device may determine an area including a facial key point as a facial area. For example, the electronic device may determine an area including an entire face as a facial area. The area including an entire face is an area including all of the facial key points. For example, the electronic device may determine an area 210 including all of the facial key points including the facial key point 201 in the input facial image 200 shown in FIG. 2 as a facial area.

[0021] The electronic device may determine, as a facial feature region, a region 220 including facial key points other than the contour key points corresponding to the facial contour among the facial key points. The facial feature region includes more facial key points for identity recognition than the facial region. In other words, the facial feature region may include more identification information for identity recognition than the facial region. For example, among the facial key points, the key points of the outer contour correspond to the contour key points. For example, the key points of the outer contour may be the outermost point among the facial key points and / or a key point that is at a certain distance from the outermost point and / or has a certain relationship with the outermost key point. For example, the facial feature region may include regions corresponding to predetermined facial feature parts (e.g., the center of the left eye, the center of the right eye, the tip of the nose, the left mouth, and the right mouth). Regions other than the regions corresponding to the predetermined facial feature parts are facial contour regions. The key points of the facial contour region correspond to the contour key points. In the input face image 200 shown in FIG. 2, a region 220 may include feature key points corresponding to preset facial feature parts.

[0022] In operation 130, the electronic device determines the input facial image as an invalid facial image if the facial feature region satisfies a first preset condition. The first preset condition may include a shape condition regarding the shape of the facial feature region, a position condition regarding the relationship between the position of the facial feature region and the position of the facial region, or a combination thereof. The shape condition is satisfied when the shape of the facial feature region is a preset shape. The position condition is satisfied when the position of the facial region does not correspond to the position of the facial feature region. If the facial feature region satisfies the first preset condition, the electronic device filters the input facial image determined as an invalid facial image. In this case, the electronic device can perform an authentication procedure using an image other than the input facial image determined as an invalid facial image.

[0023] For example, when the face of the input face image has an extreme head pose (e.g., as a non-limiting example, a face in profile at a large angle), facial feature information is difficult to identify compared to facial contour information. For example, in this case, the face region is easily detected, but the location of the facial feature region is occluded by the head itself, etc., and a reliable facial feature region cannot be obtained.

[0024] According to one embodiment, the electronic device may determine whether a face in an input facial image has an extreme head pose by checking whether the shape of the facial feature region is a pre-defined shape (e.g., by way of non-limiting example, an elongated shape, etc.). The electronic device may determine that the shape condition of the first pre-defined condition is satisfied when the shape of the feature envelope box is the pre-defined shape. The feature envelope box is a box of the smallest size that includes key points belonging to the facial feature region among the key points of the face. For example, the region 220 shown in FIG. 2 corresponding to the facial feature region may be determined as the feature envelope box. For example, the feature envelope box may be determined as EF=(E x , E y , F x , F y ) where EF is the feature envelope box, E x is the x-coordinate of position E, E y is the y coordinate of position E, F x is the x-coordinate of position F, F y indicates the y coordinate of position F. The feature envelope box is the coordinate of the upper left vertex (E x , E y ) and the coordinates of the bottom right vertex (F x , F y ), but is not limited to this.

[0025] According to an embodiment, the electronic device may determine that the shape of the feature envelope box is a preset shape if the aspect ratio of the feature envelope box is greater than a first threshold. For example, the electronic device may determine whether the shape of the feature envelope box is a preset shape using the following Equation (1):

[0026]

number

[0027] In formula (1), HWR is the aspect ratio of the feature envelope box, E x is the x-coordinate of the top-left vertex of the feature envelope box, E y is the y-coordinate of the top-left vertex of the feature envelope box, F x is the x-coordinate of the bottom right vertex of the feature envelope box, F y is the y-coordinate of the bottom right vertex of the feature envelope box, T 1 indicates a first threshold value. The first threshold value may be preset. The first threshold value may be determined experimentally. For example, the first threshold value may be 2.8, but is not limited thereto.

[0028] If the aspect ratio of the feature envelope box is less than a first threshold, the electronic device may determine that the shape of the feature envelope box is a non-predefined shape. In this case, it may be determined that the face of the input face image does not have an extreme head pose, for example, the face has a normal pose such as a frontal face. Such an input face image may be used for identity recognition without additional processing related to facial feature regions.

[0029] The position condition is satisfied when the position of the face region does not correspond to the position of the facial feature region. According to one embodiment, the position condition is satisfied when the widthwise position of the face envelope box of the face region does not correspond to the widthwise position of the feature envelope box of the facial feature region. The face envelope box is the smallest box that contains the face key points, and the feature envelope box is the smallest box that contains the face key points that belong to the facial feature region. For example, the region 210 shown in FIG. 2 that corresponds to the face region may be determined as the face envelope box. For example, the face envelope box may be determined as follows: CD=(C x , C y , Dx , D y ) where CD is the face envelope box, C x is the x-coordinate of position C, C y is the y coordinate of position C, and D is the x is the x-coordinate of position D, D y indicates the y coordinate of position D. The feature envelope box is the coordinate of the upper left vertex (C x , C y ) and the coordinates of the bottom right vertex (D x , D y ), but is not limited to this.

[0030] According to an embodiment, the electronic device may determine that the position of the face envelope box in the width direction is not corresponding to the position of the feature envelope box in the width direction when the ratio between the distance between the center axis of the face envelope box in the height direction and the center axis of the feature envelope box in the height direction and the width of the face envelope box is smaller than a second threshold. For example, the electronic device may determine whether the position of the face region is not corresponding to the position of the face feature region using the following Equation (2):

[0031]

number

[0032] In formula (2), MRR is the positional relationship between the face region and the facial feature region, C x is the x-coordinate of the upper left corner of the face envelope box, D x is the x-coordinate of the bottom right vertex of the face envelope box, E x is the x-coordinate of the top-left vertex of the feature envelope box, F x is the x-coordinate of the lower right corner of the feature envelope box. (F x +E x ) / 2 indicates the center axis of the face envelope box in the height direction, and (D x +C x ) / 2 indicates the center axis of the feature envelope box. x -C xdenotes the width of the face envelope box. According to an embodiment, the width direction corresponds to the x-axis, and the height direction corresponds to the y-axis. The position in the width direction is specified via the x-coordinate, and the position in the height direction is specified via the y-coordinate. If the positional relationship MRR between the face region and the facial feature region is smaller than a second threshold, the electronic device may determine that the position in the width direction of the face envelope box and the position in the width direction of the feature envelope box of the facial feature region do not correspond to each other. The second threshold is T 2 As another example, if the positional relationship MRR between the face region and the facial feature region is equal to or greater than a different threshold, the electronic device may determine that the widthwise position of the face envelope box and the widthwise position of the feature envelope box of the face feature region do not correspond to each other. For example, the second threshold may be 0.15, but is not limited thereto.

[0033] MRR is T 2 If the threshold is less than 1, the positions of the facial feature region and the facial region may not correspond to each other. In this case, the structural consistency between the facial feature region and the facial region in the input facial image is insufficient. Such an input facial image reduces the accuracy of face recognition. According to an embodiment, such a facial image corresponds to an interference image. The second threshold may be determined experimentally.

[0034] According to one embodiment, a noisy test dataset is defined to test the accuracy of face recognition technology. The test dataset includes a positive sample pair and a negative sample pair. The positive sample pair may include two images corresponding to the same person. The negative sample pair may include two images corresponding to different people. The test results are evaluated using true accept rate (TAR) and false accept rate (FAR). TAR is calculated using the following formula (3), and FAR is calculated using the following formula (4).

[0035]

number

[0036]

number

[0037] In Equation (3) and Equation (4), T denotes a threshold value. For security reasons, FAR may be set low. A threshold value T that satisfies FAR is determined using Equation (3). A TAR is determined as a final evaluation index using Equation (4). According to the embodiment, interference samples can be identified. In this test, regardless of whether the image samples are filtered by the image processing method of the embodiment, all sample pairs are regarded as sample pairs that have failed to be registered, and the ratio of sample pairs that have failed to be registered among all sample pairs is calculated as a registration failure rate. The sample pairs that have failed to be registered may be used in the TAR and FAR calculations, and the score of the sample pairs that have failed to be registered is regarded as 0.

[0038] Sample pairs are compared using cosine similarity. For example, the sample comparison score is given by Equation (5):

[0039]

number

[0040] In Equation (5), A and B are the feature vectors of the two images of the sample pair, and n is the dimension of the feature vector. The higher the sample comparison score of a sample pair, the higher the probability that the sample pair is a positive sample pair.

[0041] According to the test results, the image processing method of the embodiment can greatly improve the accuracy of interference sample filtering. For example, when the head rotation angle of the image is greater than or close to 90 degrees, the image does not appear valid to the naked eye, and it is difficult to obtain reliable identity information from working on such an image. It is effective to filter such an image based on the structural consistency of the face region and the face feature region.

[0042] The image processing method according to the embodiment can filter highly-interfering face images in a targeted manner, and can efficiently improve the accuracy of recognition without changing the general face recognition process or increasing the amount of calculation. The filtering of the image processing method can take into account a situation where an error occurs in the face detection model and / or the face keypoint detection model, or an incomplete face is included in the face image.

[0043] According to an embodiment, the electronic device may determine an invalid face image based on reclassifying the face image using a face region box and / or spatial consistency between the face region box and the face region. According to an embodiment, the electronic device may detect a face region box from the input face image and determine the input face image as an invalid face image if the face region box satisfies a second preset condition. The face region box is detected by a face detection model. The second preset condition may include a face condition regarding whether the face region box is determined as a non-face, a spatial condition regarding whether the face region box and the face region do not spatially correspond to each other, or a combination thereof.

[0044] 3 and 4 are diagrams illustrating an exemplary face recognition operation using a face classification model according to an embodiment. Referring to FIG. 3, in step 310, the electronic device detects a face area box from an input face image. According to an embodiment, the face area box may be detected by a face detection model. For example, the face detection model may be an artificial intelligence model (e.g., a neural network model) trained to detect a face area box from an input face image. In this case, the face detection model may be pre-trained to detect a face area box from an input face image through machine learning. A face area may be determined based on face key points detected by a key point detection model, and the face area box may be detected by the face detection model.

[0045] In step 320, the electronic device updates the face region box by enlarging the face region box. The initial face region box may include only a portion of the face, and may not include the entire face. According to an embodiment, the face region box may be updated by enlarging the face region box so that the face region box includes the entire face. For example, the electronic device may extract a partial image corresponding to the enlarged face region box from the input face image to determine the updated face region box.

[0046] According to an embodiment, the electronic device may enlarge the face area box by a preset multiple (e.g., 1.1 times) based on the center point of the face area box. Alternatively, the distance between each point of the face area box and the center point may be enlarged by a fixed multiple, or the area of ​​the face area box may be enlarged by a fixed multiple, but is not limited thereto.

[0047] In step 330, the electronic device determines a face confidence of the updated face region box. The face confidence indicates whether the input face image corresponds to a face. The electronic device may determine whether it is necessary to use a face classification model based on the face confidence. The face confidence is calculated using a confidence estimation model. The confidence estimation model is an artificial intelligence model (e.g., a neural network model), which is small and lightweight compared to the face classification model. Using two thresholds for the face confidence can reduce waste of computing resources.

[0048] In step 340, the electronic device determines whether the face region box is a non-face based on the face confidence level. If the face confidence level is less than a third threshold, the input face image is unlikely to be a face image. The third threshold is T 3 In this case, classification using a face classification model may not be necessary. For example, in FIG. 4, the input face image B1 is not input to the face classification model and is immediately determined to be a non-face image. A non-face image corresponds to an invalid image.

[0049] If the face confidence is greater than the fourth threshold, the input face image is likely to be a face image. The fourth threshold is T 4 In this case, classification using a face classification model is not required. For example, in FIG. 4, the input face image B4 is not input to the face classification model and is immediately determined as a face image. The face image corresponds to a valid image.

[0050] If the confidence level of the face is greater than the third threshold and less than the fourth threshold, it may be difficult to predict the possibility that the input face image is a face image. In this case, a classification operation using a face classification model is necessary. In this case, the electronic device may determine the updated face region box as a face or a non-face using the face classification model. For example, in FIG. 4, the input face images B2 and B3 may be input to the face classification model.

[0051] According to an embodiment, the face classification model may determine whether the face region box includes a facial feature depending on the input of the face region box. For example, the face classification model may classify the face region box as a face or a non-face. Here, the face region box may be an updated face region box. The update of the face region box may include enlarging the face region box. According to an embodiment, the face region box may be enlarged to fit a fixed input size of the face classification model. The face classification model is an artificial intelligence model (e.g., a neural network model) trained to classify whether the face region box is a face image or a non-face image. In this case, the face classification model may be pre-trained to classify whether the face region box is a face image or a non-face image via machine learning.

[0052] According to an embodiment, a face classification model may be trained using a dataset that ensures diversity and balance, where balance means a balance between positive samples and negative samples.

[0053] The dataset may include positive samples corresponding to face image samples and negative samples corresponding to non-face image samples. The positive samples may be obtained by extracting face regions from a given image. For example, a complete face may be extracted from at least one of an enlarged face region box, a face label box, and a window whose IOU (intersection over union) with the face label box satisfies a specific condition. The negative samples may be obtained by extracting non-face regions from a given image.

[0054] The image processing method according to the embodiment may filter various interference samples. For example, but not limited to, images containing environmental noise or heavy occlusions that make it difficult to obtain identity information, low-quality face images, etc. may be filtered. The image processing according to the embodiment may improve the accuracy of face recognition and improve the reliability of the recognition results. The face confidence-based image processing method according to the embodiment may provide excellent results with minimal resources.

[0055] 5 is a diagram illustrating an example of a face region box and a face region of an input face image according to an embodiment. Referring to FIG. 5, the electronic device may determine whether a spatial condition regarding whether a face region box 520 and a face region spatially correspond to each other is satisfied. The electronic device may determine whether a second preset condition is satisfied based on whether the spatial condition is satisfied. If the spatial condition and / or the second preset condition is satisfied, the electronic device determines the input face image 500 as an invalid face image.

[0056] According to one embodiment, the electronic device can determine whether the face region box 520 and the face region spatially do not correspond to each other based on the degree of spatial overlap between the face region box 520 and the face envelope box 510 of the face region. The face envelope box 510 may be a box of the smallest size that contains the face key points. The face envelope box 510 is determined by the following formula: CD=(C x , C y , D x , D y ) The face region box 520 is shown as AB=(A x , A y , B x , B y 5. The face key points are detected by a face key point detection model, and a face envelope box 510 is determined based on the face key points. A face region box 520 may be detected by the face detection model.

[0057] According to one embodiment, the electronic device determines an IOU between the face region box 520 and the face envelope box 510, and determines that the face region box 520 and the face region do not spatially correspond to each other if the IOU is less than a fifth threshold. 5 The IOU is used to measure the degree of overlap between the face region box 520 and the face envelope box 510. For example, the IOU is calculated using Equation 6 below.

[0058]

number

[0059] If IOU is smaller than the fifth threshold, the electronic device determines that the face region box 520 and the face region do not spatially correspond to each other. In this case, the spatial consistency between the face region box 520 and the face region is insufficient, and it is difficult to perform face recognition using the input face image. The fifth threshold may be determined experimentally. For example, the fifth threshold may be 0.47, but is not limited thereto.

[0060] FIG. 6 is a diagram illustrating an exemplary face recognition operation using a preset condition according to an embodiment. Referring to FIG. 6, in step 610, the electronic device determines a face region, a facial feature region, a face region box, or a combination thereof based on an input face image. The electronic device may detect facial key points of the input face image using a key point detection model. The electronic device may determine the face region and the facial feature region based on the facial key points. For example, the electronic device may determine a face envelope box corresponding to the face region and a feature envelope box corresponding to the facial feature region based on the facial key points. The electronic device may determine the face region box using a face detection model.

[0061] In step 620, the electronic device checks whether a first preset condition is satisfied. The first preset condition may include a shape condition regarding the shape of the facial feature region, a position condition regarding the relationship between the position of the facial feature region and the position of the face region, or a combination thereof. The shape condition is satisfied when the shape of the facial feature region is a preset shape, and the position condition is satisfied when the position of the face region and the position of the face region do not correspond to each other. If the first preset condition is satisfied, in step 650, the electronic device discards the input face image.

[0062] In step 630, the electronic device checks whether a second preset condition is satisfied. The second preset condition may include a face condition regarding whether the face region box is determined as a non-face, a spatial condition regarding whether the face region box and the face region do not spatially correspond to each other, or a combination thereof. The face condition is satisfied when the face region box is determined as a non-face, and the spatial condition is satisfied when the face region box and the face region do not spatially correspond to each other. If the second preset condition is satisfied, in step 650, the electronic device discards the input face image.

[0063] Step 620 and step 630 may be performed in parallel. If either the first preset condition in step 620 or the second preset condition in step 630 is satisfied, step 650 is performed. If neither the first preset condition nor the second preset condition is satisfied, step 640 is performed.

[0064] In step 640, the electronic device performs further processing of facial recognition on the input facial image. For example, the further processing may include face alignment, facial feature extraction, facial feature comparison, or a combination thereof. The further processing may use the result of the face detection in step 610 (e.g., face region box) and / or the result of the facial key point detection in step 610 (e.g., facial key points). The electronic device may include an additional artificial intelligence model (e.g., neural network model) for further processing, or may perform further processing using the additional artificial intelligence model. In step 640, the input facial image on which further processing is performed corresponds to a valid facial image, and the input facial image discarded in step 650 corresponds to an invalid facial image.

[0065] After steps 640 and 650, step 610 is performed for the next frame. For example, face recognition is performed on an input video, and the input video may include successive image frames. The input face image of step 610 may be one of such successive image frames. Steps 610 to 640 may be repeatedly performed for the successive image frames.

[0066] 7 is a diagram illustrating an exemplary configuration of an electronic device according to an embodiment. Referring to FIG. 7, the electronic device 700 includes one or more processors 710, memory 720, camera 730, storage device 740, input device 750, output device 760, and network interface 770, which can communicate via a communication bus 780. For example, the electronic device 700 may be implemented as at least a part of a mobile device such as a mobile phone, a smartphone, a PDA, a netbook, a tablet computer, a laptop computer, etc., a wearable device such as a smart watch, a smart band, a smart glasses, etc., a computing device such as a desktop, a server, etc., a home appliance such as a television, a smart television, a refrigerator, etc., a security device such as a door rack, etc., a vehicle such as an autonomous vehicle, a smart vehicle, etc.

[0067] The one or more processors 710 execute functions and instructions to be executed within the electronic device 700. For example, the one or more processors 710 process instructions stored in the memory 720 or the storage device 740. The instructions, when executed by the one or more processors 710, cause the electronic device 700 to perform the operations described with reference to FIGS. 1-6.

[0068] Memory 720 may include a computer-readable storage medium or a computer-readable storage device. Memory 720 may store instructions for execution by one or more processors 710 and may store relevant information during execution of software and / or applications by electronic device 700.

[0069] Camera 730 takes pictures and / or videos. Storage 740 includes a computer readable storage medium or computer readable storage device. Storage 740 can store a larger amount of information than memory 720 and can store information for long periods of time. For example, storage 740 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other forms of non-volatile memory known in the art.

[0070] The input device 750 can receive input from a user through traditional input methods through a keyboard and mouse, new input methods such as touch input, voice input, and image input. For example, the input device 750 may include a keyboard, a mouse, a touch screen, a microphone, or any other device capable of detecting input from a user and communicating the detected input to the electronic device 700. The output device 760 can provide output of the electronic device 700 to a user through a visual, auditory, or tactile channel. The output device 760 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device capable of providing output to a user. The network interface 770 can communicate with external devices through a wired or wireless network.

[0071] The embodiments described above may be implemented with hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or other devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, the processing device may be described as being one in which only one processing device is used, but one skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0072] Software may include computer programs, codes, instructions, or any combination of one or more thereof, capable of configuring or instructing a processing device to operate as desired, either independently or in combination. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, either permanently or temporarily, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed across computer systems coupled to a network, and may be stored and executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

[0073] The method according to the present invention may be embodied in the form of program instructions to be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and the program instructions may be specially designed and constructed for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, and the like. Examples of program instructions include not only machine language code, such as that generated by a compiler, but also high-level language code executed by a computer using an interpreter, etc.

[0074] The hardware devices described above may be configured to operate as one or more software modules to perform the operations illustrated in the present invention, and vice versa.

[0075] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than described, and may be replaced or substituted with other components or equivalents to achieve appropriate results.

[0076] Accordingly, other implementations, embodiments, and equivalents of the claims are intended to be within the scope of the following claims. [Explanation of symbols]

[0077] 200 input face images 201 Face Key Points 210, 220 area

Claims

1. 1. A method for image processing, comprising: Detecting facial key points from an input facial image; determining a face region of the input facial image and facial feature regions of the input facial image based on the facial key points; determining the input facial image as an invalid facial image if the facial feature region satisfies a first preset condition; Including, The image processing method, wherein the first preset condition includes a shape condition regarding a shape of the facial feature region, a position condition regarding a relationship between a position of the facial feature region and a position of the face region, or a combination thereof.

2. 2. The image processing method of claim 1, wherein the shape condition is satisfied when the shape of the facial feature region is a pre-defined shape.

3. 2. The image processing method of claim 1, wherein the location condition is satisfied when the locations of the face regions and the locations of the facial feature regions do not correspond to each other.

4. The operation of determining the face region and the facial feature region includes: An operation of determining an area including the face key points as the face area; determining, as the face feature region, a region including feature key points other than contour key points corresponding to a facial contour among the face key points; 2. The image processing method of claim 1, comprising:

5. The position condition is: is satisfied when the widthwise position of the face envelope box of the face region does not correspond to the widthwise position of the feature envelope box of the facial feature region; The shape condition is: is satisfied when the shape of the feature envelope box is a predefined shape; the face envelope box is a box of a minimum size that contains the face key points; The image processing method of claim 1 , wherein the feature envelope box is a box of a minimum size that includes key points belonging to the facial feature region among the face key points.

6. If the aspect ratio of the feature envelope box is greater than a first threshold, the shape of the feature envelope box is determined to be the preset shape; 6. The image processing method of claim 5, wherein if a ratio between a distance between a central axis of the height direction of the face envelope box and a central axis of the height direction of the feature envelope box and a width of the face envelope box is smaller than a second threshold, it is determined that a position of the width direction of the face envelope box does not correspond to a position of the width direction of the feature envelope box of the facial feature region.

7. The image processing method includes: detecting a face region box from the input face image; determining the input face image as an invalid face image if the face area box satisfies a second preset condition; Further comprising:

2. The image processing method of claim 1, wherein the second preset condition includes a face condition regarding whether the face area box is determined as a non-face, a spatial condition regarding whether the face area box and the face area do not spatially correspond to each other, or a combination thereof.

8. The operation of determining the input facial image as the invalid facial image includes: updating the face area box by expanding the face area box of the input facial image; determining a face confidence of the updated face region box indicating a probability that the updated face region box contains a face; determining the updated face region box as the non-face using a face classification model if the confidence level of the face is greater than a third threshold and less than a fourth threshold; 8. The image processing method of claim 7, comprising:

9. The operation of determining the input face image as an invalid face image includes: determining whether the face area box and the face area spatially do not correspond to each other based on a degree of spatial overlap between the face area box and a face envelope box of the face area; The image processing method of claim 7, wherein the face envelope box is a box of smallest size that contains the face key points.

10. The operation of determining whether the face region box and the face region do not spatially correspond to each other includes: determining an intersection over union (IOU) between the face region box and the face envelope box; determining that the face region box and the face region do not spatially correspond to each other if the IOU is less than a fifth threshold; 10. The image processing method of claim 9, comprising:

11. A computer program stored on a computer-readable recording medium for executing the method according to any one of claims 1 to 10 in combination with hardware.

12. 1. An electronic device comprising: one or more processors; A memory for storing instruction words; Including, The instructions, when executed by the one or more processors, cause the electronic device to: Detecting facial key points from an input facial image; determining a face region of the input facial image and facial feature regions of the input facial image based on the facial key points; determining the input facial image as an invalid facial image if the facial feature region satisfies a first preset condition; is executed, The electronic device, wherein the first preset condition includes a shape condition regarding a shape of the facial feature region, a position condition regarding a relationship between a position of the facial feature region and a position of the face region, or a combination thereof.

13. the shape condition is satisfied when the shape of the facial feature region is a predetermined shape; The electronic device of claim 12 , wherein the location condition is satisfied when the locations of the face regions and the locations of the facial feature regions do not correspond to each other.

14. The operation of determining the face region and the facial feature region includes: An operation of determining an area including the face key points as the face area; determining, as the face feature region, a region including feature key points other than contour key points corresponding to a facial contour among the face key points; The electronic device of claim 12 .

15. the position condition is satisfied when a widthwise position of a face envelope box of the face region and a widthwise position of a feature envelope box of the facial feature region do not correspond to each other; The shape condition is: is satisfied when the shape of the feature envelope box is a predefined shape; the face envelope box is a box of the smallest size that contains the face key points; The electronic device according to claim 12 , wherein the feature envelope box is a box of a minimum size that includes key points of the face that belong to the facial feature region.

16. If the aspect ratio of the feature envelope box is greater than a first threshold, the shape of the feature envelope box is determined to be the preset shape; 16. The electronic device of claim 15, wherein if a ratio between a distance between a central axis in the height direction of the face envelope box and a central axis in the height direction of the feature envelope box and a width of the face envelope box is smaller than a second threshold, it is determined that the width direction position of the face envelope box does not correspond to the width direction position of the feature envelope box of the facial feature region.

17. The instructions, when executed by the one or more processors, cause the electronic device to: detecting a face region box from the input face image; determining the input face image as an invalid face image if the face area box satisfies a second preset condition; is executed, The electronic device of claim 12 , wherein the second preset condition includes a face condition regarding whether the face area box is determined as a non-face, a spatial condition regarding whether the face area box and the face area do not spatially correspond to each other, or a combination thereof.

18. The operation of determining the input facial image as the invalid facial image includes: updating the face area box by expanding the face area box of the input facial image; determining a face confidence of the updated face region box indicating a probability that the updated face region box contains a face; determining the updated face region box as the non-face using a face classification model if the confidence level of the face is greater than a third threshold and less than a fourth threshold; 20. The electronic device of claim 17, comprising:

19. The operation of determining the input face image as an invalid face image includes: determining whether the face area box and the face area spatially do not correspond to each other based on a degree of spatial overlap between the face area box and a face envelope box of the face area; The electronic device of claim 17 , wherein the face envelope box is a box of minimum size that contains key points of the face.

20. 1. A method for image processing, comprising: Detecting keypoints from an input image; determining regions of the input image and feature regions of the input image based on the keypoints; determining the input image as an invalid image based on an aspect ratio of the feature region, a ratio between a center axis of the region of the input image in a height direction and a center axis of the feature region in a height direction and a width of the input image, or a combination thereof; 13. An image processing method comprising:

Images