A feature point reconstruction method for complex face poses

Abstract

The application discloses a feature point reconstruction method for complex human face posture, comprising the following steps: obtaining an image set of an original occluded human face, marking feature points, and judging the human face posture; extracting a gradient histogram feature of each picture on a training sample set to form a feature set; learning a sparse matrix by using a multi-initialization mode to extract a large number of image sample features to obtain multi-posture feature mapping parameters; inputting a human face image to perform target detection, extracting a gradient histogram feature to perform parameter calculation, and reconstructing human face features. The application is suitable for the problem of face occlusion within 50%, and the occlusion of some key parts such as eyes and mouths appears at a high frequency under a monitoring camera. The human face occlusion is a common problem, and the method has great application value in a camera environment such as security monitoring and intelligent home.

Description

Technical Field

[0001] This invention relates to the technical field of face recognition, and more particularly to a method for reconstructing feature points of complex facial poses. Background Technology

[0002] Facial recognition, as a prime example of the rapid development of artificial intelligence technology, is currently widely used in fields such as public security, transportation, and e-commerce, including facial search, facial recognition for fast passage, and facial payment. Under good lighting and shooting angles, facial recognition algorithms can achieve excellent results. However, in real-world scenarios, insufficient lighting, facial occlusion, and the shooting angle can all interfere with the recognition results. For payment systems, recognition failures will lead to unsuccessful e-commerce transactions, reducing user experience; for gate recognition systems, recognition errors may reduce passage efficiency; and for criminal identification systems, low recognition rates could allow criminals to escape justice.

[0003] In the application of facial recognition algorithms, especially under surveillance cameras, facial occlusion is a common problem, with key areas such as the eyes and mouth being frequently obscured. While there are existing methods for feature point reconstruction for small-area occlusions such as those caused by masks or sunglasses, methods for feature point reconstruction in complex poses such as side profiles, upward or downward views are extremely rare. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by this invention is to address the problem of facial feature point reconstruction when complex facial poses are partially occluded.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: It includes acquiring image sets of original occluded faces, marking feature points, and performing face pose determination; extracting gradient histogram features from each image in the training sample set to form a feature set; using multiple initialization methods to extract features from a large number of image samples to learn the sparse matrix and obtain multi-pose feature mapping parameters; inputting a face image for target detection, extracting gradient histogram features for parameter calculation, and reconstructing face features.

[0008] As a preferred embodiment of the feature point reconstruction method for complex human face poses described in this invention, the image set includes side view, upward view, and downward view image sets.

[0009] As a preferred embodiment of the feature point reconstruction method for complex facial poses described in this invention, the feature point marking includes: annotating the input facial image, with the annotated area divided into a normal area and a supplementary area; if there is no occlusion or deviation, the supplementary area is empty, and feature points are directly extracted using the annotated points; if there is occlusion or deviation, the Criminisi algorithm is used to find the optimal matching block from the known area of ​​the image for filling, and feature points are extracted from the filled face using the annotated points.

[0010] As a preferred embodiment of the feature point reconstruction method for complex facial poses described in this invention, the facial pose determination includes: when |l1-θ1|>0, it is a deflection pose; when |l2-θ2|<0, it is an upward-looking pose; when |l3-θ3|>0, it is a downward-looking pose; where d1 is the distance between the vertical line from the center of the face and the vertical line from the center of the left eye, d2 is the distance between the vertical line from the center of the face and the vertical line from the center of the right eye, the intersection of the line connecting the two eyes and the vertical line from the center of the face is denoted as C, d3 is the distance between the center of the face boundary and point C, and d4 is the distance between the center of the face boundary and the center point of the mouth. Set the deflection attitude threshold θ1, the upward attitude threshold θ2, and the downward attitude threshold θ3 respectively.

[0011] As a preferred embodiment of the feature point reconstruction method for complex facial poses described in this invention, the method includes: extracting gradient histogram features ρ corresponding to each image around the accurately marked feature point locations. ij And form a feature set S, where S = {S1, S2, S3}, where S1, S2, and S3 are the gradient histogram feature sets of the side view, upward view, and downward view of the face, respectively.

[0012] As a preferred embodiment of the feature point reconstruction method for complex facial poses described in this invention, wherein: learning the sparse matrix includes enabling the learned sparse matrix to correct incomplete image samples, and obtaining matrix D by minimizing the feature sparse reconstruction error:

[0013]

[0014] st‖D i ||0≤T

[0015] Where M is the set of facial features extracted locally from the database, N is the number of initializations, δ represents the sparse coefficients of texture features on the boundary of facial contour feature points, the sparse matrix is ​​arranged column-wise and the number of columns is greater than the feature dimension, and T is the number of non-zero values ​​in the coefficient vector, i.e., the sparsity, which is solved using the OMP method, by extracting S from matrix D. i The corresponding three different feature point indices are used to extract subset rows of D to construct matrices D1, D2, and D3 respectively.

[0016] As a preferred embodiment of the feature point reconstruction method for complex facial poses described in this invention, it further includes: learning the feature mapping parameter γ for the i-th pose based on the contour matrices D1, D2, and D3 learned offline for facial features. i Its learning expression is:

[0017]

[0018] st‖γ i ||0≤β i

[0019] Among them, namely D i The contour matrix representing the i-th pose, β i Sparsity is the number of non-zero values ​​in the coefficient vector, which is also solved using the OMP method.

[0020] As a preferred embodiment of the feature point reconstruction method for complex facial poses described in this invention, the reconstructed facial features include obtaining... The image is determined to be the i-th face pose, thus obtaining the contour subset matrix D of the corresponding pose. i Obtain the feature mapping parameter γ i ;

[0021] Substitute the contour sparsity coefficient γ of the i-th pose i The complete facial features can be reconstructed using the following formula:

[0022] ρ′=ρD i γ i .

[0023] Beneficial effects of the present invention: The present invention is applicable to the problem of facial occlusion within 50%. Under surveillance cameras, occlusion of key parts such as the eyes and mouth occurs frequently, and facial occlusion is a common problem. This method has great application value in security monitoring, smart home and other camera environments. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0025] Figure 1 This is a schematic diagram of the parameter learning stage of the feature point reconstruction method for complex face poses according to an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of the feature reconstruction and pose determination stages of the feature point reconstruction method for complex face poses according to an embodiment of the present invention. Detailed Implementation

[0027] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0028] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0029] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0030] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0031] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0032] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0033] Example 1

[0034] Reference Figure 1 and Figure 2 This is the first embodiment of the present invention, providing a method for reconstructing feature points of complex facial poses, characterized by: including,

[0035] S1: Obtain the original image set of the occluded face, label feature points, and determine the face pose. It should be noted that the image set includes:

[0036] A collection of side profile, upward view, and downward view images;

[0037] Furthermore, feature point labeling includes:

[0038] The input face image is annotated, and the annotated area is divided into normal area and supplementary area;

[0039] If there is no occlusion or deflection, the supplementary area is empty, and the feature points are extracted directly using the labeled points.

[0040] If there is occlusion or deflection, the Criminisi algorithm is used to find the optimal matching block from the known area of ​​the image for filling, and the feature points of the filled face are extracted using the annotation points.

[0041] Specifically, facial pose determination includes:

[0042] When |l1-θ1|>0, it is the deflection attitude;

[0043] When |l2-θ2|<0, it is an upward-looking posture;

[0044] When |l3-θ3|>0, it is a top-down view.

[0045] Where d1 is the distance between the perpendicular line from the center of the face and the perpendicular line from the center of the left eye, d2 is the distance between the perpendicular line from the center of the face and the perpendicular line from the center of the right eye, the intersection of the line connecting the two eyes and the perpendicular line from the center of the face is denoted as C, d3 is the distance between the center of the face boundary and point C, and d4 is the distance between the center of the face boundary and the center point of the mouth. Set the deflection attitude threshold θ1, the upward attitude threshold θ2, and the downward attitude threshold θ3 respectively.

[0046] S2: Extract the gradient histogram features of each image in the training sample set to form a feature set. It should be noted that:

[0047] Gradient histogram features ρ are extracted from the areas surrounding the precisely labeled feature points for each image. ij And form a feature set S, where S = {S1, S2, S3}, where S1, S2, and S3 are the gradient histogram feature sets of the side view, upward view, and downward view of the face, respectively.

[0048] S3: A large number of image sample features are extracted using multiple initialization methods to learn the sparse matrix, resulting in multi-pose feature mapping parameters. It should also be noted that learning the sparse matrix includes:

[0049] The learned sparse matrix can correct incomplete image samples. Matrix D is obtained by minimizing the feature sparse reconstruction error.

[0050]

[0051] st‖D i ||0≤T

[0052] Where M is the set of facial features extracted locally from the database, N is the number of initializations, δ represents the sparse coefficients of texture features on the boundary of facial contour feature points, the sparse matrix is ​​arranged column-wise and the number of columns is greater than the feature dimension, and T is the number of non-zero values ​​in the coefficient vector, i.e., the sparsity, which is solved using the OMP method, by extracting S from matrix D. i The corresponding three different feature point indices are used to extract subset rows of D to construct matrices D1, D2, and D3 respectively.

[0053] For the contour matrices D1, D2, and D3 learned offline for facial features, the feature mapping parameter γ for the i-th pose is learned. i Its learning expression is:

[0054]

[0055] st‖γ i ||0≤β i

[0056] Among them, namely D i The contour matrix representing the i-th pose, β i Sparsity is the number of non-zero values ​​in the coefficient vector, which is also solved using the OMP method.

[0057] S4: Input a face image for object detection, extract gradient histogram features for parameter calculation, and reconstruct face features. This step also requires further explanation:

[0058] Obtain the original image of the occluded face;

[0059] Perform object detection on the image to eliminate interference from non-face image regions;

[0060] Extract the gradient histogram features ρ and form a feature set;

[0061] Seeking The image is determined to be the i-th face pose, thus obtaining the contour subset matrix D of the corresponding pose. i Obtain the feature mapping parameter γ i ;

[0062] Substitute the contour sparsity coefficient γ of the i-th pose i The complete facial features can be reconstructed using the following formula:

[0063] ρ′=ρD i γ i

[0064] Example 2

[0065] To better verify and illustrate the technical effects of the method used in this invention, this embodiment selects a traditional face recognition method and the method of this invention for comparative testing. The test results are compared using scientific demonstration methods to verify the true effectiveness of the method of this invention.

[0066] Traditional face recognition methods cannot be applied to face feature recognition in environments with certain occlusions, which has significant limitations and large recognition errors. To verify that the method of the present invention has higher recognition accuracy and wider applicability than traditional methods, this embodiment will use traditional methods and the method of the present invention to perform real-time measurement and comparison on the sample image set of the simulation platform.

[0067] Test environment: The sample image set was imported into the simulation platform for simulation and a large-area (over 50%) occlusion scenario was simulated. Face recognition tests were conducted using traditional methods and the test results were obtained. When using the method of this invention, the automated testing equipment was turned on and the simulation test of the method of this invention was implemented using MATLAB software. Simulation data was obtained based on the experimental results. 10,000 sets of data were tested for each method, and the mean error of each set of data was calculated and compared with the actual predicted value input in the simulation to calculate the error.

[0068] Early stage / % Interim / % Later stage / % Average error / % Traditional methods 34.276 36.531 38.921 36.576 Method of the present invention 22.531 21.092 22.197 21.940

[0069] Referring to Table 1, it can be clearly seen that the method of the present invention has a smaller error and a higher accuracy compared to the traditional method.

[0070] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for reconstructing feature points of complex facial poses, characterized in that: include, Obtain the original occluded face image set, including side view, upward view, and downward view, and mark the feature points to determine the face pose. Extract the gradient histogram features of each image in the training sample set to form a feature set; The input face image is annotated, and the annotated area is divided into normal area and supplementary area; If there is no occlusion or deflection, the supplementary area is empty, and the feature points are extracted directly using the marked points; If there is occlusion or deflection, the Criminist algorithm is used to find the optimal matching block from the known region of the image to fill it in, and the feature points of the filled face are extracted using the annotation points. When |l1-θ1|>0, it is the deflection attitude; When |l²-θ²|<0, it is a looking-up posture; When |l3-θ3|>0, it is a top-down view. Where d1 is the distance between the perpendicular line from the center of the face and the perpendicular line from the center of the left eye, d2 is the distance between the perpendicular line from the center of the face and the perpendicular line from the center of the right eye, the intersection of the line connecting the two eyes and the perpendicular line from the center of the face is denoted as C, d3 is the distance between the center of the face boundary and point C, and d4 is the distance between the center of the face boundary and the center point of the mouth. Set the deflection attitude threshold θ1, the upward attitude threshold θ2, and the downward attitude threshold θ3 respectively; Gradient histogram features ρ are extracted from the areas surrounding the precisely labeled feature points for each image. ij And form a feature set S, where S = {S1, S2, S3}, where S1, S2, and S3 are the gradient histogram feature sets of the side view, upward view, and downward view of the face, respectively; By using multiple initialization methods to extract features from a large number of image samples, the sparse matrix is ​​learned to obtain multi-pose feature mapping parameters; The learning of the sparse matrix includes, The learned sparse matrix can be used to correct incomplete image samples. Matrix D is obtained by minimizing the feature sparse reconstruction error. s.t.||D i ||0≤T Where M is the set of facial features extracted locally from the database, N is the number of initializations, δ represents the sparse coefficients of texture features on the boundary of facial contour feature points, the sparse matrix is ​​arranged column-wise and the number of columns is greater than the feature dimension, and T is the number of non-zero values ​​in the coefficient vector, i.e., the sparsity, which is solved using the OMP method, by extracting S from matrix D. i The corresponding three different feature point indices are used to extract subset rows of D to construct matrices D1, D2, and D3 respectively; For the contour matrices D1, D2, and D3 learned offline for facial features, the feature mapping parameter γ for the i-th pose is learned. i Its learning expression is, st||c i ||0≤β i Among them, namely D i The contour matrix representing the i-th pose, β i The sparsity is the number of non-zero values ​​in the coefficient vector, which is also solved using the OMP method. Input a face image for object detection, extract gradient histogram features for parameter calculation, and reconstruct face features; The reconstructed facial features include, Seek The image is determined to be the i-th face pose, thus obtaining the contour subset matrix D of the corresponding pose. i Obtain the feature mapping parameter γ i ; Substitute the contour sparsity coefficient γ of the i-th pose i The complete facial features can be reconstructed using the following formula. ρ′=ρD i c i 。

Images