A method for generating a recognizable face image from a side face image of a person

Abstract

The application provides a method for generating identifiable face images from face side images. The method comprises: extracting face key points from face left and right side images, establishing a corresponding three-dimensional rotation model according to the change process of face rotation paths and camera imaging rules; establishing a corresponding relationship between the actually extracted left and right face key points and the face key points in the imaging plane derived from the three-dimensional rotation model, and converting it into an optimization problem, and solving the optimization problem to obtain unknown parameters in the three-dimensional rotation model; based on the three-dimensional rotation model and the solved parameters, the face key point positions at any angle in the rotation path can be obtained, and the identifiable face images can be generated according to the intermediate state face key point coordinates. The application directly represents the rotation path of the face key points in the two-dimensional image through a mathematical model and an optimization problem, and more quickly calculates the face key point positions in the images obtained by shooting at any viewing angle.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition, and in particular to a method for generating identifiable facial key points based on a side face image. Background Art

[0002] With the widespread application of facial recognition technology and the proliferation of surveillance cameras, facial recognition and identification are increasingly being performed through surveillance footage. Faces captured in actual surveillance video are subject to numerous factors, hindering the accuracy of facial recognition. For example, in many factories or large rooms, surveillance cameras are typically mounted on the walls. Coupled with the flexible and free movement of faces, faces captured by surveillance video are typically viewed from the left or right side, rather than the front. Existing face detection and recognition algorithms are unable to effectively identify non-frontal faces with large angles of deflection in images. Therefore, it is necessary to recover facial images from multiple different angles to obtain a more accurate face image. Entering these recovered facial images into a face comparison system can provide clearer and more accurate recognition results.

[0003] Currently, there is no method in the prior art for efficiently generating a recognizable frontal face image from a large-angle side face image. Summary of the Invention

[0004] An embodiment of the present invention provides a method for generating a recognizable facial image based on a side face image, so as to effectively generate a recognizable facial image from a side face image at a large angle.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0006] A method for generating identifiable frontal facial key points based on a side face image, comprising:

[0007] Extract the key points of the left and right faces from the captured left and right side images, abstract the left and right side face shooting process into an arc-shaped face rotation path, and establish a corresponding three-dimensional rotation model based on the change process of the face rotation path and the camera imaging law;

[0008] Establishing a correspondence between the actually extracted left and right facial key points and the facial key points in the imaging plane derived from the three-dimensional rotation model, converting the correspondence into an optimization problem, and solving the optimization problem using a quasi-Newton method to obtain unknown parameters in the three-dimensional rotation model;

[0009] Based on the established three-dimensional rotation model and the obtained unknown parameters, the positions of facial key points at any angle on the rotation path are calculated by setting different rotation angles, and a recognizable facial image is generated according to the intermediate states of the facial key points on the rotation path.

[0010] Preferably, the ERT algorithm is used to extract left and right facial key points from the left and right side images obtained by shooting, and the shooting process of the left and right side faces is abstracted into an arc-shaped face rotation path. According to the change process of the face rotation path and the camera imaging law, a corresponding three-dimensional rotation model is established, including:

[0011] The shooting process is abstracted as an arc-shaped rotation path. The left and right side face images are regarded as the images obtained by rotating the face around the camera optical axis. According to the rotation process of the shooting model, a three-dimensional rotation model between the key points of the left and right side faces is established. Assuming that the Z axis is the straight line where the camera optical axis is located, after the camera takes an image, the face is rotated around the fixed Z axis by a certain angle, and the second image is taken again. The coordinates of the key points of the face before rotation are set to P(X i ,Y i ,Z i ), rotate point P clockwise around the Z axis by θ degrees to obtain point P'. The physical meanings of points P and P' correspond to the coordinates of the left and right facial key points in three-dimensional space, respectively. When point P is rotated around the Z axis, point P' and point P' have the same Z coordinate. Project points P and P' onto the XOY plane and solve for the coordinates on the X and Y axes. The rotation process of the shooting model is expressed in a matrix as follows:

[0012]

[0013] Where θ represents the rotation angle, (X i ,Y i ) and (X i ',Y i ') represent the projections of point P and point P' on the XOY plane respectively. The coordinates of point P' are:

[0014] (X i cosθ-Y i sinθ,Y i ·cosθ+X i sinθ,Z i ) (2)

[0015] By simulating the imaging principle of a camera, the three-dimensional coordinates are projected into a two-dimensional plane. A projection plane is set in the three-dimensional rotation model, which is the imaging plane. The three-dimensional space points P and P' are projected onto the imaging plane, and the three-dimensional coordinates of the facial key points in the three-dimensional space in the projection plane are calculated. The three-dimensional coordinates of the facial key points in the projection plane are transformed from the three-dimensional coordinate system to the pixel coordinate system.

[0016] Preferably, projecting the three-dimensional space point P and point P' onto the imaging plane and calculating the three-dimensional coordinates of the facial key points in the three-dimensional space on the projection plane includes:

[0017] Assume that the projection points of point P and point P' on the imaging plane are P_t and P_t', the projection lines in the parallel projection are parallel lines, the Z coordinates of the projection points P_t and P_t' are equal to the Z coordinates of point P and point P', project P_t and P_t' onto the XOY plane, and calculate the coordinates on the x and y axes, where the straight line T: x = -y, P T (X i ,Y i ) is the projection point of point P on the XOY plane, P I is the projection point of P_t on the XOY plane, and the projection method is parallel orthographic projection, P T P I Perpendicular to the TOZ plane;

[0018] P T P I ⊥T, with the foot of the perpendicular being P I

[0019] Line P T P I The expression is:

[0020] x=y+X i -Y i (3)

[0021] Therefore, the straight line P T P I Intersection point P with line T I The coordinates are:

[0022]

[0023] Through the coordinates of point P, the coordinates of the projection point P_t are calculated as:

[0024]

[0025] Through the coordinates of point P', the coordinates of point P_t' are calculated as:

[0026]

[0027] Preferably, the step of converting the three-dimensional coordinates of the facial key points in the projection plane into a pixel coordinate system includes:

[0028] Establish a rectangular coordinate system TOZ in the projection plane TZ, with the origin of the coordinate system as (0,0,0), calculate the coordinates of the projection point P_t and point P_t' in the coordinate system, and record them as P A and P B .Depend on Figure 3 It can be seen that P A and P B The vertical coordinate is equal to the Z coordinate of point P_t and point P_t'. I is the projection point of P_t on the XOY plane, then P A The horizontal axis is OP I The length of the projection point P_t in the three-dimensional coordinate system O-XYZ is used to calculate the coordinate P of the projection point in the TOZ coordinate system. A :

[0029]

[0030] According to the coordinates of point P_t', calculate the coordinates P of the projection point in the TOZ coordinate system B :

[0031]

[0032] The camera shooting process is simulated based on the three-dimensional rotation model, and the coordinate change of the facial key points from the left face to the right face is derived in theory.

[0033] Preferably, establishing a correspondence between the actually extracted left and right facial key points and the facial key points in the imaging plane derived from the three-dimensional rotation model, converting the correspondence into an optimization problem, and solving the optimization problem by a quasi-Newton method to obtain unknown parameters in the three-dimensional rotation model includes:

[0034] The actual extracted left and right facial key points are matched one by one with the facial key point coordinates in the image calculated by formula 7 and formula 8, and a corresponding relationship is established. The left facial key point corresponds to point P A The coordinates of , expressed as a matrix, are as follows:

[0035]

[0036] in Represents the 68 key points actually extracted from the left side of the face, i = 1, 2, ..., 68, and the corresponding points P of the key points on the right side of the face B The coordinates are as follows:

[0037]

[0038] in represents the 68 key points actually extracted from the right side of the face, i = 1, 2, ..., 68;

[0039] The scaling factor S, the rotation matrix E, and the translation matrix b are introduced to adjust the key points actually extracted to correspond to the theoretical key point positions calculated by the model. The corresponding relationship is as follows:

[0040]

[0041] Among them, the scaling factor S is a constant greater than 0, and the rotation matrix E and translation matrix b are as follows:

[0042]

[0043] Where α represents the rotation angle between the key points actually extracted and the theoretical key points in the model, b1 and b2 represent the offsets of the key points actually extracted and the theoretical key points in the model on the horizontal and vertical axes, respectively. According to the corresponding relationship between formulas (9) and (11), a set of equations are obtained as follows:

[0044]

[0045] in, and Respectively represent the key point coordinates actually extracted from the left and right faces, which are known variables; (X i ,Y i ,Z i ), i = 1, 2, ..., 68 are the three-dimensional coordinates of the facial key points in the left image of the model, which are unknown parameters; θ represents the rotation angle of the face from the position in the left image to the position in the right image, which is an unknown parameter; the scaling factor S, the rotation matrix E and the translation matrix b are also unknown variables. Formula (13) is simplified as follows

[0046]

[0047] Among them, S, α, b1, b2, θ and (X i ) i＝1,2,3,...,68 There are 73 variables in total.

[0048] Find a set of parameters (S,α,b1,b2,θ,X i ) i＝1,2,3,...,68 Minimize the sum of squared residuals on both sides of equation (13) and solve the following optimization problem:

[0049]

[0050]

[0051] The quasi-Newton method is used to solve the least squares optimization problem, and the solutions of the optimization problems are used to approximately replace the solutions of the unknowns in the equation.

[0052] Preferably, the method of calculating the positions of facial key points at any angle on the rotation path by setting different rotation angles based on the established three-dimensional rotation model and the obtained unknown parameters, and generating a recognizable facial image according to the intermediate states of the facial key points on the rotation path, includes:

[0053] On the rotation path in the three-dimensional rotation model, given an arbitrary rotation angle, the coordinates of the key points of the face in the image at that rotation angle are calculated. The specific calculation formula is as follows:

[0054]

[0055] in, is the coordinate of the facial key points extracted from the left image, β represents any rotation angle on the path, and its value range is between [0,θ]. Substitute the parameters obtained by equations (15) and (16) into equation (17), calculate the coordinates of the facial key points in the rotation path, and plot the coordinates of the facial key points in a two-dimensional scatter plot. The change of β angle represents the change of rotation angle. Divide the calculated θ into 10 equal parts, and plot the scatter plots of the facial key points in sequence. When When , it is the frontal face key point at the midpoint of the entire path, and a face image that is easy to identify can be generated based on the frontal face key point.

[0056] As can be seen from the technical solutions provided by the embodiments of the present invention described above, the present invention directly calculates the correspondence between facial key points in a two-dimensional image by using a mathematical model and solving an optimization problem. This allows for more rapid calculation of the positions of facial key points in images captured at any viewing angle. Even at wider viewing angles, the present invention can still reconstruct the rotational paths of facial key points.

[0057] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 A processing flow chart of a method for generating identifiable facial key points based on a side face image provided by an embodiment of the present invention;

[0060] Figure 2 A simulation diagram of the left and right side face rotation shooting process provided by an embodiment of the present invention;

[0061] Figure 3 A three-dimensional rotation model diagram provided by an embodiment of the present invention;

[0062] Figure 4 A planar XOY schematic diagram provided by an embodiment of the present invention;

[0063] Figure 5 A schematic diagram of parallel projection provided by an embodiment of the present invention;

[0064] Figure 6 A diagram of a coordinate calculation process of a projection point provided by an embodiment of the present invention;

[0065] Figure 7 A 3DMM facial key point scatter plot provided by an embodiment of the present invention;

[0066] Figure 8 A schematic diagram of a 3DMM facial key point projection scatter plot provided by an embodiment of the present invention;

[0067] Figure 9 A schematic diagram of the locations of key points on a face when simulating left and right image capture provided by an embodiment of the present invention;

[0068] Figure 10 A schematic diagram showing the left and right face projection points calculated in a pixel coordinate system according to a three-dimensional rotation model provided by an embodiment of the present invention;

[0069] Figure 11 A schematic diagram of facial key points extracted from left and right images provided by an embodiment of the present invention;

[0070] Figure 12 A schematic diagram of the rotation path of key points of a face provided by an embodiment of the present invention;

[0071] Figure 13 A schematic diagram of a rotation path of facial key points with a rotation angle of 1.76 provided in an embodiment of the present invention;

[0072] Figure 14 A schematic diagram of a rotation path of facial key points with a rotation angle of 1.8 provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0074] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0075] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0076] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0077] The present invention uses the key points of the left and right sides of the face to recover the key points of the face at any angle in this rotation path. First, the left and right facial key points are extracted from the captured left and right side images using the ERT (Ensemble of Regression Trees) algorithm. Secondly, the shooting process of the left and right sides of the face is abstracted into an arc-shaped rotation path. According to the change process of the face rotation path and the imaging law of the camera, a corresponding three-dimensional rotation model is established. Then, the left and right facial key points actually extracted are matched with the facial key points in the imaging plane derived from the three-dimensional rotation model. Then, this correspondence is converted into an optimization problem, and the unknown parameters in the model are solved by the quasi-Newton method. Finally, based on the established three-dimensional rotation model and the obtained unknown parameters, the coordinates of the facial key points of any rotation path on the rotation path are calculated by setting different rotation angles.

[0078] The processing flow chart of a method for generating a recognizable face image based on a side face image provided by an embodiment of the present invention is as follows: Figure 1 As shown in Figure 1, it mainly includes two parts: three-dimensional rotation modeling and optimization problem solving.

[0079] 3D rotation modeling: First, the shooting process is abstracted into an arc-shaped path, such as Figure 2 As shown in , the left and right side face images can be regarded as images obtained by rotating the face around the camera optical axis. Secondly, based on the rotation process of this shooting model, a three-dimensional rotation model between the key points of the left and right side faces is established, as shown in Figure 3 As shown. Assume that the Z axis is the straight line where the camera optical axis is located. After the camera takes an image, it rotates the face around the fixed Z axis by a certain angle and takes the second image again. The coordinates of the key points of the face before rotation are set to P(X i ,Y i ,Z i ), rotate point P clockwise around the Z axis by θ degrees to obtain point P'. The physical meanings of points P and P' correspond to the coordinates of the left and right facial key points in three-dimensional space, respectively. Since point P rotates around the Z axis, the Z coordinates of points P' and P are equal. Project points P and P' onto the XOY plane and solve for the coordinates on the X and Y axes, as follows: Figure 4 As shown. The above rotation process can be expressed by a matrix as follows:

[0080]

[0081] Where θ represents the rotation angle, (X i ,Y i ) and (X i ',Y i ') represent the projections of point P and point P' on the XOY plane. Therefore, the coordinates of point P' are:

[0082] (X i cosθ-Y i sinθ,Y i ·cosθ+X i sinθ,Z i ) (2)

[0083] Then, by simulating the imaging principle of the camera, the three-dimensional coordinates are projected into the two-dimensional plane. Figure 3 It can be seen that the 3D rotation model only takes into account the rotation angle of the face. Therefore, in order to simplify the camera imaging model, the face in the image is directly regarded as a parallel projection of the face in 3D space, so as to reduce the error caused by the coordinate system conversion and facilitate subsequent calculations. Parallel projection refers to the image formed by a cluster of parallel lines from a certain direction projected onto a certain plane through a point in 3D space. It is usually used to represent a correspondence between a point and a plane, such as Figure 5 As shown. Accordingly, a projection plane is set in the three-dimensional rotation model, which is the imaging plane, as shown Figure 3 As shown, it is a plane formed by the line T:x+y=0 and the Z axis. Based on this, by projecting points P and P' in 3D space onto this imaging plane, we can calculate the coordinates of the facial key points in 3D space on the projection plane. Next, we calculate the coordinates of the projected points P_t and P_t' of points P and P' on the imaging plane.

[0084] Specifically, since the projection lines in parallel projection are parallel lines, the Z coordinates of the projected points P_t and P_t' are equal to the Z coordinates of points P and P'. Then project P_t and P_t' onto the XOY plane and calculate the coordinates on the x and y axes. Figure 6 As shown, where the straight line T:x=-y, P T (X i ,Y i ) is the projection point of point P on the XOY plane, P I is the projection point of P_t on the XOY plane. Since the projection method is parallel orthographic projection, P T P I Perpendicular to the TOZ plane, and because line T is in the TOZ plane, therefore:

[0085] P T P I ⊥T, with the foot of the perpendicular being P I

[0086] From this, we can get the straight line P T P I The expression is:

[0087] x=y+X i -Yi (3)

[0088] Therefore, the straight line P T P I Intersection point P with line T I The coordinates are:

[0089]

[0090] Therefore, through the coordinates of point P, the coordinates of the projection point P_t are calculated as:

[0091]

[0092] Similarly, through the coordinates of point P', the coordinates of point P_t' are calculated as:

[0093]

[0094] Through the above three-dimensional rotation model, we calculated the three-dimensional coordinates of the key points of the left and right sides of the face in the projection plane. However, the key points of the face extracted from the image are two-dimensional coordinates in the pixel coordinate system, and the two cannot be directly matched. Therefore, it is also necessary to convert the three-dimensional coordinates of the projection points from the three-dimensional coordinate system to the pixel coordinate system. Specifically, first establish a plane rectangular coordinate system TOZ in the projection plane TZ, and the origin of the coordinate system is (0,0,0). Secondly, calculate the coordinates of the projection point P_t and point P_t' in this coordinate system, and record them as P A and P B .Depend on Figure 3 It can be seen that P A and P B The vertical coordinate of is equal to the Z coordinate of point P_t and point P_t'. Figure 6 As shown, P I is the projection point of P_t on the XOY plane, then P A The horizontal axis is OP I Therefore, according to the coordinates of the projection point P_t in the three-dimensional coordinate system O-XYZ, the coordinates of the projection point in the TOZ coordinate system P can be calculated. A :

[0095]

[0096] Similarly, based on the coordinates of point P_t', the coordinates P of the projection point in the TOZ coordinate system can be calculated. B :

[0097]

[0098] According to the three-dimensional rotation model, the camera shooting process is simulated, and the theoretical coordinate change of the facial key points from the left face to the right face is finally derived. In order to verify the correctness of the model, we will perform forward deduction through a three-dimensional face model. The common 3DMM three-dimensional face shape appearance model is selected here. The model contains 68 three-dimensional points, corresponding to the 68 key points of the face. Since we only need to verify whether the projection coordinates of the 68 key points of the three-dimensional face on the imaging surface are consistent with the 68 extracted facial key points, we do not need to fit the face. We only need to know the 3D coordinate positions corresponding to the 68 facial key points in the general model, for example:

[0099] Nose tip: (1.23, 36.94, 8.41)

[0100] Lower limit: (0.10, 0.88, -77.06)

[0101] Left eye corner: (45.17, -4.89, 38.08)

[0102] Right eye corner: (-46.04, -7.03, 37.47)

[0103] Left corner of mouth: (28.79, 3.21, -28.07)

[0104] Right corner of mouth: (-28.92, 2.24, -28.61)

[0105] First, draw the 68 key points of the 3D face in three-dimensional space and adjust their positions to meet the following requirements: Figure 3 The face of the model shown is shot in the position where the face faces the Z axis. Figure 7 , the straight line is the Z axis, and the asterisk is the origin of the three-dimensional coordinate system. Then, according to formula 7 and formula 8, the projection coordinates of the three-dimensional face on the imaging plane are calculated, and a two-dimensional scatter plot is drawn, as shown in Figure 8 , we can see that the projection result is consistent with the key point position of the front face. Then, according to formula 2, the 3D face model is rotated left and right around the fixed axis, and the rotation angle is assumed to be ±15°, respectively, to simulate the shooting positions of the left and right sides of the face, as shown in Figure 9 Finally, the projection points of the left and right groups of facial key points are calculated and drawn using Formula 7 and Formula 8, as shown in Figure 10 As shown. Compare the actual extracted facial key points with Figure 11 It can be seen that the facial contours of the two are the same, thus proving the correctness of the model.

[0106] Solving optimization problems:

[0107] According to the above-mentioned constructed face key point three-dimensional rotation model, the rotation transformation relationship of the left face to the right face in the image is finally derived. Then, the actually extracted left and right face key points are one-to-one corresponding to the face key point coordinates calculated by formula 7 and formula 8. The coordinates of the left face key point corresponding point P A are expressed in the form of a matrix as follows:

[0108]

[0109] wherein represents the actually extracted 68 key points of the left face, i = 1, 2,..., 68. Similarly, the coordinates of the right face key point corresponding point P B are as follows:

[0110]

[0111] wherein represents the actually extracted 68 key points of the right face, i = 1, 2,..., 68.

[0112] Although the position of the camera is fixed during shooting, the face is rotated and positionally transformed during shooting, and it is difficult to avoid some offsets, so in the actual situation, it cannot be guaranteed that the face key points are completely arranged according to the theoretical key point position, therefore, the scaling factor S, the rotation matrix E and the translation matrix b are introduced to adjust the corresponding relationship between the actually extracted key points and the theoretical key point position calculated by the model, and the corresponding relationship is as follows:

[0113]

[0114] wherein the scaling factor S is a constant greater than 0, the rotation matrix E and the translation matrix b are as follows

[0115]

[0116] wherein, α represents the rotation angle between the actually extracted key points and the theoretical key points in the model, b1 and b2 respectively represent the offsets of the actually extracted key points and the theoretical key points in the model in the horizontal and vertical axes.

[0117] According to the above, a set of equations can be obtained according to the corresponding relationship of formula 9 and 11, as follows

[0118]

[0119] wherein, and respectively represent the coordinates of the actually extracted key points of the left face and the right face, and (X i , Y i , Zi ), i = 1, 2, ..., 68 are the three-dimensional coordinates of the facial key points in the left image of the model, which are unknown parameters; θ represents the rotation angle of the face from the position in the left image to the position in the right image, which is also an unknown parameter; in addition, the scaling factor S, the rotation matrix E, and the translation matrix b are also unknown variables. Formula 13 is simplified as follows

[0120]

[0121] Among them, S, α, b1, b2, θ and (X i ) i＝1,2,3,...,68 There are 73 variables to be solved. The following will solve this equation.

[0122] Since the equations shown in Formula 14 contain 73 unknown parameters, which is much larger than the 68 known data samples, it is impossible to calculate these parameters by solving the equations. Therefore, we convert it into two sets of optimization problems and use the least squares method to calculate the solution of the parameters. Specifically, we find a set of parameters (S, α, b1, b2, θ, X i ) i＝1,2,3,...,68 Minimize the sum of squared residuals on both sides of the equation in Formula 13, that is, calculate the following optimization problem:

[0123]

[0124]

[0125] Then, the solution of this set of optimization problems is used to approximate the solution of the unknowns in the equation. Here, the quasi-Newton method is chosen to solve the least squares optimization problem. Considering that there are fewer unknowns in formula 16, the optimization problem is solved first. Figure 11 For the example shown, the calculation results are as follows:

[0126] α=0.006, b2=0, S=1.006, min=51867

[0127] Substituting variables α, b2, and S into Equation 15 and using the quasi-Newton method again, we obtain:

[0128] θ=1.636, b1=-899.937, min=0.006, (X i ) i＝1,2,3,...,68 for:

[0129]

[0130] according to Figure 3The three-dimensional rotation model shown can be seen that in the rotation path, the rotation angle of any angle can be calculated out the key point coordinates of the face in the image at the rotation angle, the specific calculation formula is as follows

[0131]

[0132] Wherein, The face key point coordinates extracted in the left image, beta represents the rotation angle of the path, and the value range is between [0, theta]. The parameters obtained in the above example are substituted into formula (17), the face key point coordinates in the rotation path are calculated, and the two-dimensional scatter plot is drawn, and the change of beta angle represents the change of the rotation angle. We divide the calculated theta into 10 equal parts, and draw the scatter plot of the face key point in sequence, such as Figure 12 When , it is exactly the face key point at the midpoint of the entire path, that is, the front face key point, such as Figure 12 -e. According to the front face key point, a face image convenient for recognition can be generated.

[0133] In order to effectively illustrate the performance of the present application, the following uses several groups of data to show and analyze the experimental results, so as to prove that the present application has excellent performance.

[0134] As Figure 13 shown is the face key point rotation path when the rotation angle is 1.7, Figure 14 shown is the face key point rotation path when the rotation angle is 1.8.

[0135] In summary, the front face key point reconstruction method based on three-dimensional rotation modeling proposed in the embodiment of the present application can realize the reconstruction of the entire rotation path of the face key point under the condition that the face image has large disparity, and has great advantage in speed.

[0136] The present application directly calculates the rotation path of the face key point in the two-dimensional image through the solution of mathematical model and optimization problem, and more efficiently calculates the position of the face key point in the image obtained by shooting at any angle. Under a larger shooting angle, the present application can still reconstruct the rotation path of the face key point.

[0137] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily necessary for implementing the present application.

[0138] Those skilled in the art can clearly understand the present application by the description of the above embodiments. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to cause a computer device (which can be a personal computer, a server, or a network device, and the like) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments mainly describes the difference from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the related parts can be referred to the part of the method embodiments. The above-described device and system embodiments are merely illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to the actual needs. Those skilled in the art can understand and implement it without creative labor.

[0140] The above describes only the preferred embodiments of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating a recognizable face image based on a side face image, characterized in that: include: Extract the key points of the left and right faces from the captured left and right side images, abstract the left and right side face shooting process into an arc-shaped face rotation path, and establish a corresponding three-dimensional rotation model based on the change process of the face rotation path and the camera imaging law; Establishing a correspondence between the actually extracted left and right facial key points and the facial key points in the imaging plane derived from the three-dimensional rotation model, converting the correspondence into an optimization problem, and solving the optimization problem using a quasi-Newton method to obtain unknown parameters in the three-dimensional rotation model; Based on the established three-dimensional rotation model and the unknown parameters obtained, the positions of facial key points at any angle on the rotation path are calculated by setting different rotation angles, and a recognizable face image is generated based on the intermediate states of the facial key points on the rotation path; The ERT algorithm is used to extract the key points of the left and right faces from the captured left and right side images. The left and right side face shooting process is abstracted into an arc-shaped face rotation path. According to the changing process of the face rotation path and the camera imaging law, a corresponding 3D rotation model is established, including: The shooting process is abstracted as an arc-shaped rotation path. The left and right side face images are regarded as the images obtained by rotating the face around the camera optical axis. According to the rotation process of the shooting model, a three-dimensional rotation model between the key points of the left and right side faces is established. Assuming that the Z axis is the straight line where the camera optical axis is located, after the camera takes an image, the face is rotated around the fixed Z axis by a certain angle, and the second image is taken again. The coordinates of the key points of the face before rotation are set to P(X i ,Y i ,Z i ), rotate point P clockwise around the Z axis by θ degrees to obtain point P'. The physical meanings of points P and P' correspond to the coordinates of the left and right facial key points in three-dimensional space, respectively. When point P is rotated around the Z axis, point P' and point P' have the same Z coordinate. Project points P and P' onto the XOY plane and solve for the coordinates on the X and Y axes. The rotation process of the shooting model is expressed in a matrix as follows: Where θ represents the rotation angle, (X i ,Y i ) and (X i ',Y i ') represent the projections of point P and point P' on the XOY plane respectively. The coordinates of point P' are: (X i ·cosθ-Y i ·sinθ,Y i ·cosθ+X i ·sinθ,Z i ) (2) By simulating the imaging principle of a camera, the three-dimensional coordinates are projected into a two-dimensional plane. A projection plane is set in the three-dimensional rotation model, which is the imaging plane. The three-dimensional space points P and P' are projected onto the imaging plane, and the three-dimensional coordinates of the facial key points in the three-dimensional space in the projection plane are calculated. The three-dimensional coordinates of the facial key points in the projection plane are transformed from the three-dimensional coordinate system to the pixel coordinate system.

2. The method according to claim 1, characterized in that The projecting of the three-dimensional space point P and point P' onto the imaging plane and calculating the three-dimensional coordinates of the facial key points in the three-dimensional space on the projection plane includes: Assume that the projection points of point P and point P' on the imaging plane are P_t and P_t', the projection lines in the parallel projection are parallel lines, the Z coordinates of the projection points P_t and P_t' are equal to the Z coordinates of point P and point P', project P_t and P_t' onto the XOY plane, and calculate the coordinates on the x and y axes, where the straight line T: x = -y, P T (X i ,Y i ) is the projection point of point P on the XOY plane, P I is the projection point of P_t on the XOY plane, and the projection method is parallel orthographic projection, P T P I Perpendicular to the TOZ plane; P T P I ⊥T, with the foot of the perpendicular being P I Line P T P I The expression is: x=y+X i -AND i (3) Therefore, the straight line P T P I Intersection point P with line T I The coordinates are: Through the coordinates of point P, the coordinates of the projection point P_t are calculated as: Through the coordinates of point P', the coordinates of point P_t' are calculated as:

3. The method according to claim 2, characterized in that The coordinate system conversion of the three-dimensional coordinates of the facial key points in the projection plane from the three-dimensional coordinate system to the pixel coordinate system includes: Establish a rectangular coordinate system TOZ in the projection plane TZ, with the origin of the coordinate system as (0,0,0), calculate the coordinates of the projection point P_t and point P_t' in the coordinate system, and record them as P A and P B , P A and P B The vertical coordinate is equal to the Z coordinate of point P_t and point P_t'. I is the projection point of P_t on the XOY plane, then P A The horizontal axis is OP I The length of the projection point P_t in the three-dimensional coordinate system O-XYZ is used to calculate the coordinate P of the projection point in the TOZ coordinate system. A : According to the coordinates of point P_t', calculate the coordinates P of the projection point in the TOZ coordinate system B : The camera shooting process is simulated based on the three-dimensional rotation model, and the coordinate change of the facial key points from the left face to the right face is theoretically derived.

4. The method according to claim 3, characterized in that The method of establishing a correspondence between the actually extracted left and right facial key points and the facial key points in the imaging plane derived from the three-dimensional rotation model, converting the correspondence into an optimization problem, and solving the optimization problem using a quasi-Newton method to obtain unknown parameters in the three-dimensional rotation model includes: The actual extracted left and right facial key points are matched one by one with the facial key point coordinates in the image calculated by formula 7 and formula 8, and a corresponding relationship is established. The left facial key point corresponds to point P A The coordinates of , expressed as a matrix, are as follows: in Represents the 68 key points actually extracted from the left side of the face, i = 1, 2, ..., 68, and the corresponding points P of the key points on the right side of the face B The coordinates are as follows: in represents the 68 key points actually extracted from the right side of the face, i = 1, 2, ..., 68; The scaling factor S, the rotation matrix E, and the translation matrix b are introduced to adjust the key points actually extracted to correspond to the theoretical key point positions calculated by the model. The corresponding relationship is as follows: Among them, the scaling factor S is a constant greater than 0, and the rotation matrix E and translation matrix b are as follows: Where α represents the rotation angle between the key points actually extracted and the theoretical key points in the model, b1 and b2 represent the offsets of the key points actually extracted and the theoretical key points in the model on the horizontal and vertical axes, respectively. According to the corresponding relationship between formulas (9) and (11), a set of equations are obtained as follows: in, and i=1,2,...,68 represent the coordinates of key points actually extracted from the left and right faces, which are known variables; (X i ,Y i ,Z i ), i = 1, 2, ..., 68 are the three-dimensional coordinates of the facial key points in the left image of the model, which are unknown parameters; θ represents the rotation angle of the face from the position in the left image to the position in the right image, which is an unknown parameter; the scaling factor S, the rotation matrix E and the translation matrix b are also unknown variables. Formula (13) is simplified as follows Among them, S, α, b1, b2, θ and (X i ) i＝1,2,3,...,68 There are 73 variables to be solved. Find a set of parameters (S,α,b1,b2,θ,X i ) i＝1,2,3,...,68 Minimize the sum of squared residuals on both sides of equation (13) and solve the following optimization problem: The quasi-Newton method is used to solve the least squares optimization problem, and the solutions of the optimization problems are used to approximately replace the solutions of the unknowns in the equation.

5. The method according to claim 4, characterized in that The method of calculating the positions of facial key points at any angle on the rotation path by setting different rotation angles based on the established three-dimensional rotation model and the obtained unknown parameters, and generating a recognizable facial image based on the intermediate states of the facial key points on the rotation path, includes: On the rotation path in the three-dimensional rotation model, given an arbitrary rotation angle, the coordinates of the key points of the face in the image at that rotation angle are calculated. The specific calculation formula is as follows: in, is the coordinate of the facial key points extracted from the left image, β represents any rotation angle on the path, and its value range is between [0,θ]. Substitute the parameters obtained by equations (15) and (16) into equation (17), calculate the coordinates of the facial key points in the rotation path, and plot the coordinates of the facial key points in a two-dimensional scatter plot. The change of β angle represents the change of rotation angle. Divide the calculated θ into 10 equal parts, and plot the scatter plots of the facial key points in sequence. When When , it is the frontal face key point at the midpoint of the entire path, and a face image that is easy to identify can be generated based on the frontal face key point.

Images