Correction of foreshortening distortion on faces

Abstract

This document describes techniques and systems that enable correction of perspective foreshortening distortion on faces. The techniques and systems include an electronic device (102) that generates a depth map (304) of objects in an input image (302) captured by a camera (118). Based on the depth map, the electronic device (102) generates a warp grid (306) on the input image (302) for correcting perspective foreshortening distortion. The electronic device (102) then warps the input image (302) according to the warp grid (306) to provide an output image (308) having reduced perspective foreshortening distortion compared to the input image (302).

Description

Background Art

[0001] Photos taken of oneself using a handheld electronic device are called "selfies." Because selfies are captured at arm's length using a handheld camera, the captured image often looks different from the person himself. Notably, a person's nose may appear larger and the person's face may exhibit an unpleasant fish-eye effect. This problem is called foreshortening distortion and is caused by the short distance between the camera and the user's face. Professional photographers avoid foreshortening distortion by using a telephoto camera (e.g., 150mm or more) at a greater distance from the subject (e.g., three meters or more). However, none of these solutions are applicable to handheld devices such as smartphones when they are used for close-up imaging. Summary of the invention

[0002] This document describes techniques and systems that enable foreshortening distortion correction on faces. These techniques include image-based schemes that utilize a deep proxy (e.g., a depth map) over the face region in an image and compute an optimal warp to forward project the face at a larger focal length. Given a selfie and a desired focal length as input, these techniques output a new selfie image as if the face was captured by a camera at a larger focal length.

[0003] According to one aspect, a method for correcting perspective foreshortening distortion on a face in an image is provided, the method comprising: generating a depth map of an object in an input image captured by a camera; based on the depth map, generating a warp grid on the input image for correcting the perspective foreshortening distortion; and warping the input image according to the warp grid to provide an output image having reduced perspective foreshortening distortion compared to the input image.

[0004] The method may also include the following optional features. Generating the depth map may include: fitting a three-dimensional facial model with the face of the object from the input image; and rendering the depth map at the viewpoint of the camera using the fitted three-dimensional facial model. Generating the depth map may include: detecting a set of landmarks on the object in the input image; adjusting a general three-dimensional facial model to adopt the detected set of landmarks to provide an adjusted facial model, thereby providing an adjusted 3D facial model with the face of the object; outputting the adjusted 3D facial model. The method may also include: refining the adjusted 3D facial model by removing one or more features outside the facial area of ​​the adjusted 3D facial model to provide a refined 3D facial model; smoothing the refined 3D facial model to remove sharp contours caused by the refinement; and rasterizing the depth map according to the smoothed 3D model. Removing the one or more features may include removing one or more of the following from the adjusted 3D facial model: ears, the back of the face, and the neck. Generating the warped mesh may include: placing a two-dimensional (2D) grid on the input image, the 2D grid having a plurality of vertices; for each vertex of the plurality of vertices corresponding to the facial region of the subject in the input image, forward projecting the corresponding point on the input image based on the depth map and camera geometry; and selecting the center of the facial region as the reprojection center to maintain the shape of one or more objects worn by the subject in the facial region. The forward projection may include: determining the gaze direction of the subject in the input image; placing a virtual camera at a lower position than the camera based on the gaze direction being below the camera; and tilting the virtual camera upward so that the result of the warping changes the gaze direction toward the camera. The method may also include employing an energy function: uniformly distributing the plurality of vertices locally; conforming each vertex in the facial region to the corresponding point; reducing the warping of vertices adjacent to the contour of the facial region; and preventing the warping of vertices outside the facial region. The energy function may include a face term, and the method may further include determining the face term by applying a lower weight to vertices adjacent to the outline of the face region than other weights applied to vertices within the face region that are not adjacent to the outline; and allowing the vertices within the face region to be translated and scaled uniformly as a group. The energy function may include a regularization term, and the method may further include determining the regularization term by adding padded vertices outside the warped mesh; and clipping the padded vertices to reduce artifacts near the border of the input image.The regularization term may also be determined by: penalizing the deformation direction of the warped mesh to reduce the variation in the orientation of adjacent connections in the warped mesh; and reducing weights on vertices on the boundary of the facial region. The input image may be captured by the camera at a first distance from the object; and the output image may appear as if the output image was captured by the camera at a distance greater than the first distance from the object. The method may also include storing the output image as the captured image. The camera may be integrated with a handheld electronic device.

[0005] According to another aspect, an electronic device is provided, comprising: a camera for capturing an image; a memory for storing the image; and one or more processors for executing instructions stored in the memory to implement the above method aspects.

[0006] According to another aspect, a computer readable medium storing processor readable instructions configured to cause one or more processors to perform the above method aspects is provided.

[0007] It will be appreciated that features described in the context of one aspect may be combined with features described in the context of another aspect.

[0008] This Summary is provided to introduce simplified concepts about foreshortening-distortion correction on the face, which is further described in the following Detailed Description and Figures. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Details of one or more aspects of perspective foreshortening distortion correction on the face are described in this document with reference to the following drawings. The same numbers are used throughout the drawings to reference the same features and components:

[0010] Figure 1 An example environment is shown in which techniques enabling correction of foreshortening distortion on faces may be implemented.

[0011] Figure 2 Example implementations of electronic devices that can implement foreshortening distortion correction on the face are shown.

[0012] Figure 3 Shown from Figure 1 and Figure 2 Example workflow for the Distortion Correction module.

[0013] Figure 4 Depicting an example implementation of facial reconstruction.

[0014] Figure 5Depicts an example warped mesh, which is Figure 3 Example of a warped mesh in .

[0015] Figure 6 An example comparison of reprojection results using different methods for an image of a subject wearing glasses is shown.

[0016] Figure 7 Depicting an example implementation of a warped mesh having one or more artifacts.

[0017] Figure 8 An example implementation of tilt correction for selfies is shown.

[0018] Fig. 9 Depicts an example method for correcting foreshortening distortion on faces in an image.

[0019] Fig.10 and Fig.11 Depicts a method for correcting foreshortening distortion on faces in an image.

[0020] Fig.12 An example computing system is shown, which may be implemented as shown in FIG. Figure 1-10 Any type of client, server, and / or electronic device described for implementing foreshortening correction on the face or in which techniques enabling foreshortening correction on the face may be implemented. DETAILED DESCRIPTION

[0021] Overview

[0022] This document describes techniques and systems that enable correction of foreshortening distortion on faces. Foreshortening distortion causes features or objects closer to the camera to appear larger, while other features have a fish-eye effect, producing an image of the face that looks different from the actual face. This foreshortening warp is due to the finite length of the user's arm holding the camera and the focal length of the camera. The techniques described herein provide a software-based solution that corrects foreshortening distortion and outputs an image that appears as if it were captured at a target focal length.

[0023] In various aspects, a method for correcting foreshortening distortion on a face in an image is disclosed. The method includes generating a depth map of an object in an input image captured by a camera. Additionally, the method includes generating a warping grid on the input image based on the depth map for correcting foreshortening distortion. The method also includes warping the input image according to the warping grid to provide an output image having reduced foreshortening distortion compared to the input image.

[0024] In various aspects, an electronic device is disclosed. The electronic device includes a camera for capturing an image and a memory for storing the image. The electronic device also includes one or more processors for executing instructions stored in the memory to implement a reconstruction module, a mesh module, and a warping module. The reconstruction module detects a set of landmarks on an object in an input image, adjusts a general three-dimensional face model to adopt the detected set of landmarks to provide an adjusted face model, thereby providing an adjusted 3D face model with a face of the object, and outputs the adjusted 3D face model. The reconstruction module also refines the adjusted 3D face model by removing one or more features outside the face area of ​​the adjusted 3D face model to provide a refined 3D face model, smoothes the refined 3D face model to remove sharp contours caused by the refinement, and rasterizes a depth map based on the smoothed 3D model. The mesh module places a two-dimensional (2D) grid on an input image, the 2D grid having a plurality of vertices, and for each vertex of the plurality of vertices corresponding to a facial region of an object in the input image, forward projects a corresponding point on the input image based on a depth map and camera geometry, and selects a center of the facial region as a reprojection center to maintain the shape of one or more objects worn by the object in the facial region. The mesh module also employs an energy function (e.g., a mesh optimization process) to evenly distribute the plurality of vertices locally, so that each vertex in the facial region conforms to a corresponding point, reduces warping of vertices adjacent to a contour of the facial region, and prevents warping of vertices outside the facial region. Then, the mesh module generates a warped mesh that maps each point on the input to a corrected output. The warping module warps the input image according to the warping mesh to provide an output image having reduced perspective foreshortening distortion compared to the input image.

[0025] These are just a few examples of how the described techniques and devices can be used to implement perspective foreshortening distortion correction on the face. Other examples and implementations are described throughout this document. This document now turns to an example operating environment, followed by a description of example devices, methods, and systems.

[0026] Operating Environment

[0027] Figure 1 An example environment 100 is shown in which techniques for enabling foreshortening distortion correction on a face may be implemented. The example environment 100 includes an electronic device 102 (e.g., a user device) that includes or is associated with a camera system 104, one or more computer-readable media 106, and a display device 108. As further described below, the electronic device 102 is configured to implement a distortion correction module 110.

[0028] In the illustrated environment 100, a user 112 holds an electronic device 102 and uses a camera system 104 to take a "selfie". As shown in 116-1, the camera system 104 captures an image 114 of the user 112, in which the user's face is distorted due to the proximity between the camera 118 of the camera system 104 and the user's face. For example, the user's nose appears too large, and the nose and mouth areas appear vertically elongated. Using the captured image 114 and the focal length of the camera as input, the distortion correction module 110 manipulates the captured image 114 to provide an output image 120 that looks like it was captured at a larger focal length. As described in further detail below, the image manipulation utilizes facial model reconstruction and grid-based image warping to correct foreshortening distortion. As shown in 116-2, the user's nose in the output image 120 appears smaller than the user's nose in the captured image 114, and the nose and mouth areas are less elongated. In addition, the eyes in the output image 120 are closer than in the captured image 114. The ratio between the eyes and the nose in output image 120 is different than the corresponding ratio in captured image 114. Using the techniques described herein, output image 120 is more representative of the human-perceived appearance of user 112, and therefore "looks" more like user 112 than captured image 114 due to the distortion in captured image 114. Subjectively, output image 120 appears more natural than captured image 114.

[0029] In more detail, consider Figure 2 , which shows an example implementation 200 of an electronic device 102 that can implement perspective foreshortening distortion correction on a face. Figure 2 The electronic device 102 is shown with various example devices, including a smart phone 102-1, a tablet computer 1022, a laptop computer 102-3, a desktop computer 102-4, a computing watch 102-5, computing glasses 102-6, a gaming system 102-7, a home automation and control system 102-8, and a microwave oven 102-9. The electronic device 102 may also include other devices, such as televisions, entertainment systems, audio systems, automobiles, drones, tracking pads, drawing tablets, netbooks, e-readers, home security systems, and other home appliances. Note that the electronic device 102 may be wearable, non-wearable but mobile, or relatively immobile (e.g., desktops and appliances).

[0030] The electronic device 102 also includes one or more computer processors 202 and one or more computer readable media 106, which include memory media and storage media. Applications and / or operating systems 204 implemented as computer readable instructions on the computer readable media 106 can be executed by the computer processor 202 to provide some or all of the functionality described herein. For example, the computer readable medium 106 can include a distortion correction module 110 that implements a reconstruction module 206, a mesh module 208, and a warping module 210.

[0031] The reconstruction module 206 is configured to generate a depth map on a facial region in an input image. To this end, the reconstruction module 206 first reconstructs a head model based on the face of the user in the input image. Specifically, the reconstruction module 206 fits a three-dimensional (3D) facial model based on the input image, and then uses the 3D facial model to render a depth map at the viewpoint (e.g., focal length) of the camera. In order to generate a 3D facial model, the reconstruction module 206 uses a facial detection algorithm to detect a set of landmarks on the input. Then, the reconstruction module 206 adopts the detected landmarks by adjusting a standard facial model (e.g., a universal facial model) to fit the detected landmarks to the standard facial model. In one example, the reconstruction module 206 minimizes the Euclidean norm (e.g., L2 distance) between the landmarks and corresponding points of the standard facial model in Euclidean space. Then, the reconstruction module 206 outputs a 3D facial model that looks like the object in the input image. The reconstruction module 206 refines the 3D facial model by removing the ears, the back of the face, and the neck from the 3D model. Then, the refined 3D facial model is smoothed to avoid sharp contours caused by model refinement. The reconstruction module 206 then rasterizes the depth map based on the smoothed 3D model.

[0032] The mesh module 208 is configured to generate a warped mesh that maps each point on the input image to a corrected output (e.g., a smoothed 3D model). The mesh module 208 first places a two-dimensional (2D) grid M on the input image. For each vertex on the grid that overlaps the facial region in the input image, the mesh module 208 forward projects the corresponding point relative to the face based on the depth map and camera geometry (e.g., focal length, orientation, angle). As part of the forward projection, the mesh module 208 selects the reprojection center to be at the center of the facial region to avoid compressing the glasses worn by the user in the image. The projected vertex is represented by the term M'. In some cases, the mesh including the projected vertex M' may be incompatible with the background. To address this incompatibility, the mesh module 208 employs a mesh optimization process by utilizing an energy / cost function to balance between the projected vertex M' and the background. The mesh optimization process includes an energy term that includes a facial term and a regularization term. For the face term, face vertices should be close to M', the face term should allow translation and scaling, and lower weights are assigned to vertices adjacent to the contour. The regularization term is based on |Vi-Vj|^2, where Vi and Vj are adjacent vertices on the 2D grid M. The regularization term is also based on a line-bending term to reduce the change in the orientation of adjacent connections in the warped mesh by penalizing the mesh deformation direction. The mesh optimization process is solved using any suitable solver, such as a nonlinear least squares solver.

[0033] The warping module 210 is configured to warp the input image according to the warping grid generated by the grid module 208. Any suitable image warping method may be used to warp the image. The warping module 210 then provides an output image that is a warped version of the input image. The output image is rendered for display via the display device 108 and is a more accurate representation of the human-perceivable appearance of the user's face.

[0034] The electronic device 102 may also include a network interface 212. The electronic device 102 may use the network interface 212 to communicate data over a wired, wireless, or optical network. By way of example and not limitation, the network interface 212 may communicate data over a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a wide area network (WAN), an intranet, the Internet, a peer-to-peer network, a point-to-point network, or a mesh network.

[0035] The camera system 104 is implemented to capture image data. The camera system 104 may include one or more front-facing cameras and rear-facing cameras. An example front-facing camera is Figure 11, camera system 104 has a wide field of view (FOV) (e.g., 64, 83, 90 degrees), which allows for group selfies. However, with the same wide FOV camera, user 112 may hold camera 118 closer to their face for selfies so that the user's face occupies a larger area in the captured image.

[0036] The electronic device 102 may also include one or more sensors 214. The one or more sensors 214 may include any of a variety of sensors, such as an audio sensor (e.g., a microphone), a touch input sensor (e.g., a touch screen), an image capture device (e.g., a camera or camcorder), a proximity sensor (e.g., a capacitive sensor), or an ambient light sensor (e.g., a photodetector).

[0037] The electronic device 102 may also include a display device 108. The display device 108 may include any suitable display device, such as a touch screen, a liquid crystal display (LCD), a thin film transistor (TFT) LCD, an in-plane switching (IPS) LCD, a capacitive touch screen display, an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED) display, a super AMOLED display, etc.

[0038] Figure 3 Show Figure 1 and Figure 2 3. An example workflow 300 of the distortion correction module 110 of the embodiment of the present invention is shown. The camera system 104 captures an original image 302. The reconstruction module 206 uses the original image 302 as an input image. Using the input image, the reconstruction module 206 reconstructs a head model from the input image by fitting the face of the object in the input image onto the 3D face model to generate a depth map. During fitting, the reconstruction module 206 also calculates the rotation and translation of the 3D model relative to the camera. The mesh module 208 uses the depth map to calculate a warp mesh 306 on the input image for correction using reprojection and regularization techniques. The warp module 210 warps the input image according to the warp mesh to provide an output image 308 with reduced perspective foreshortening distortion. For example, the face in the output image 308 is less elongated, the nose is narrower, and the ratio between the eyes and the nose is smaller compared to the ratio of the original image 302. Further details of this and other features are described below.

[0039] As described above, the reconstruction module 206 first detects landmarks on the face in the input image. Figure 4 Such an example is shown in , which describes an example implementation 400 of facial reconstruction. For example, given an image 402, the reconstruction module 206 detects a plurality of landmarks 404 corresponding to various features on a face 406 in the image 402. In addition, Figure 4An example reconstructed 3D face model 408 is shown, which is based on the distance and orientation of the face to the camera estimated from the image 402. In various aspects, the camera and head model are placed in a right-handed coordinate system in which the y-axis is opposite to gravity. The camera is placed in the positive z-axis direction and faces the origin of the coordinate system, with the up-vector parallel to the y-axis. The head model is placed approximately at the origin of the coordinate system, facing a direction approximately parallel to the positive z-axis.

[0040] return Figure 3 , the head model is rasterized with a camera, where the depth component is recorded in a 2D image (i.e., depth map 304). The depth map 304 has the same depth as the input image 302. As shown in the depth map 304, features outside the facial area (e.g., the ears and back of the model) are removed because they are not part of the facial part of the model. In addition, since the eyes and mouth are not covered by the model, there may be hollow parts in the resulting depth map. Due to inaccuracies in the fitting process (which often occur in images where the face is posed at an abnormal angle), the depth map may overflow the facial part, thereby covering parts of the background and introducing artifacts. In order to address these artifacts, portrait segmentation is calculated from the input image to further trim the depth map.

[0041] Using the camera and the head model, the virtual camera can be oriented toward the head model, but at a greater distance from the head model than the distance between the camera and the face, to capture (or recapture) the head model with less perspective foreshortening distortion. The distance of the virtual camera can be specified according to user input, or a predefined default value can be used. The FOV or focal length of the virtual camera is estimated so that the face occupies approximately the same area (e.g., the same size) in the new image as it occupies in the input image 302. Using the depth map, the mesh module 208 calculates 3D points on the head model that correspond to feature points in the input image 302. These 3D points are then projected onto the virtual camera, resulting in different pixel coordinates for each 3D point. In various aspects, these 3D points are taken uniformly on the image plane. The mesh module 208 can perform a projection process on the 3D points to determine corresponding warped points on the virtual camera, which can be organized into a warp grid (e.g., warp grid 306) defined on a rectilinear grid.

[0042] Figure 5 An example warping grid 500 is shown, which is Figure 3302, the warp grid 500 is used to store information for correcting foreshortening warping. The warp grid 500 has a uniform rectilinear grid 502 that covers substantially the entire image. The warp grid 500 can have any suitable number of cells 504, such as about 100 cells per side. Each vertex on the grid corresponds to an image coordinate in the input image 302. The warp grid 500 also associates each vertex 506 on the grid with another image coordinate that describes the warped position 508 of the vertex.

[0043] To calculate the warped position 508, the mesh module 208 determines whether the vertex is located on a face (the face is represented by the face boundary 510). If the vertex is on the face (e.g., within the face boundary 510), the depth map is used to calculate a 3D point on the face corresponding to the underlying image coordinates, and the point is projected onto the virtual camera to obtain new image coordinates. If the vertex is not on the face, there is no warping on the vertex, and the mesh module 208 avoids calculating the warped position of the vertex. For example, vertex 512 is determined to be within the face boundary 510, and therefore, the mesh module 208 determines the warped position 514 of vertex 512. The mesh module 208 determines that vertex 516 (and other vertices represented by solid circles) are outside the face boundary 510 and are therefore not on the face. Therefore, the mesh module 208 does not calculate the warped position of vertex 516 because only the face in the image will be warped. Vertices on or near the boundary 518 of the warping mesh, such as vertex 520 (represented by an open circle), are determined to be boundary vertices and are not warped but remain fixed. Additionally, vertex 522 is determined to be outside the mesh boundary and is processed similarly to vertex 516 .

[0044] Due to the nature of the reprojection process, the image content is compressed towards the center of the image, causing some artifacts in the resulting image. For example, Figure 6 An example 600 is shown that includes a comparison of images of a subject wearing glasses after reprojection using different methods. The example 600 shown includes an original image 602 of a face 604 wearing glasses 606-1. For comparison purposes, Figure 6 Including based on the naive projection ( projection) and a second processed image 610 centered on the face 604 based on the projection.

[0045] In the first processed image 608, the naive projection uses the image center 612 of the image 608 as the center for reprojection (e.g., the reprojection center indicated by arrow 614-1), which causes the image 608 to be compressed toward the image center 612. This compression causes some undesirable effects on portions of the image 608 that are not in the center of the image 608. For example, notice that the glasses 606-2 have been distorted compared to the original shape of the glasses 606-1 in the original image 602. This distortion is due to the compression toward the image center 612 of the image 608. To address this issue, the mesh module 208 uses the centroid 616 of the boundary of the face 604 (e.g., the approximate center of the facial region) for the reprojection center (e.g., the reprojection center indicated by arrow 614-2). Using the facial boundary centroid 616 for the reprojection center causes the warping to compress the face 604 toward the center of the face 604. Note that the shape of the glasses 606-3 is approximately the same as the shape of the glasses 606-1 in the original image 602.

[0046] Figure 7 An example implementation 700 of a warped mesh with one or more artifacts is depicted. In some instances, a resulting warped mesh (e.g., warped mesh 306) output by mesh module 208 may include one or more artifacts. Artifacts may include content discontinuities in the warped image caused by excessive changes in one or more vertices during warping, which results in overlaps between two or more quadrilaterals ("quads") defined by a subset of vertices in the warped mesh. Image 702 represents a processed image in which a warped mesh 704 (e.g., warped mesh 500) has been applied to original image 302. Artifact 706 is caused by overlapping quadrilaterals in warped mesh 704, such as quadrilateral 708 overlapping quadrilateral 710, and quadrilateral 712 overlapping both quadrilateral 710 and quadrilateral 714.

[0047] To address this problem, a numerical optimization process can be used to regularize the warped mesh. Padding vertices (e.g. Figure 5 Vertices 522 in 522 are added outside the original warped mesh and then clipped to reduce artifacts close to the border of the input image, such as artifacts introduced by faces touching the image border. In the example, multiple layers of vertices are filled on each of the four sides of the warped mesh 704. This ensures that facial vertices (e.g., grid vertices on the face or in the facial region, such as Figure 5 Vertices 512 in do not touch the border of the warped mesh. Vertices on the border of the grid (e.g., Figure 5 Vertex 520 in is fixed, and other vertices that do not fall on the face or on the grid boundary (e.g., Figure 5 The vertices 516 in can move freely.

[0048] To regularize the warped mesh, the mesh module 208 applies an energy function defined on the warped mesh and minimizes the energy to encourage uniform distribution of vertices. Figure 5 The vertices 506, 512, 516 and 522 in are subjected to regularization. The energy function is a nonlinear least squares system and is defined by the following equation:

[0049] In Equation 1, the term i refers to the index of the corresponding vertex of the warped mesh. The term F refers to the index of the corresponding face vertex of the warped mesh, and the term B refers to the index of the corresponding border vertex of the warped mesh. The term (i, j) represents the connection between adjacent vertices (because the grid is formed by quadrilaterals, a vertex can have at most four adjacent vertices). The term u i represents the position of the warped vertices and is optimized as the goal. The term W(i) refers to the projection-based image coordinates of the corresponding vertex. The term T refers to a data tensor, such as a rank-three data tensor that can be used to construct a bilinear face model with two attributes, namely identity and expression. The first summation term in Equation 1 calculates the energy on the facial vertices. By minimizing this term, the facial vertices are encouraged to conform to the projection-based warping. The second summation term in Equation 1 evaluates the energy on all pairs of adjacent vertices. By minimizing this term, the grid vertices are encouraged to be evenly distributed locally in their neighborhood. In addition, the term s scales the set of facial vertices as a whole and the term t translates the set of facial vertices as a whole.

[0050] The balance between the two summations is given by w F (i) and Λ ij Weighting. Vertices on the boundary of a face (e.g., a contour) are treated differently from vertices on the face. Facial vertices on the boundary of a face are given lower weights. In one example, if i is not on the face, then w F (i) is zero (0), or if i is on the face then w F (i) is one (1), and if i is on a contour on the face then w F (i) multiplied by a scaling factor (e.g., 0.5). Thus, the weight on the energy term that holds vertices in place is reduced to make them more susceptible to regularization. To identify whether a facial vertex is on a contour, the mesh module 208 may make a copy of the depth map, scale the copy of the depth map by a factor of, for example, 0.9, align its center with the center of the original depth map, and refer to the facial vertex as "on a contour" if it is not within the facial region of the resized copy. Scaling the depth map in this manner allows more scaling or compression toward the center of the facial region and less scaling or compression near the boundaries on the face.

[0051] Item Λij is called the line bending term and is introduced into the pair-wise energy so that the connections are kept with minimal changes to their orientation. ij are usually diagonal and positive, and are designed to make (u i ,u j ) in the x component is more penalized and the horizontal connection of (i, j) makes (u i ,u j )’s y-component is penalized more to prevent the line from bending.

[0052] Figure 8 An example implementation 800 of tilt correction for a face is shown. When a user takes a selfie, they typically look toward a display to see a preview image before pressing a button to capture the image. However, due to hardware constraints, the camera is typically offset from the display. Because the user typically looks at the display device 108 rather than the camera 118, when taking a selfie, the user's face may be tilted downward in the captured image. Figure 8 An example of such a tilt is shown in , where a side profile of a subject 802 is shown looking in a direction 804 toward display device 108 rather than toward camera 118 .

[0053] With the head model reconstructed and registered to the camera, 3D points on the head model can be moved around the x-axis (e.g., the horizontal axis relative to the camera's viewing angle, toward / away from the camera). Figure 8 The head model may be rotated by an angle within an example range of four to twelve degrees. The rotation may cause the focus of the eye to move upward relative to the electronic device 102, approximately falling on the camera 118, as shown by the object 808. By applying the tilt correction to the warp grid, the warping of the captured image in which the user's face is tilted downward is corrected, and the user's face appears to be looking directly at the camera.

[0054] return Figure 3 , the warping module 210 uses the warping grid 306 provided by the grid module 208 to warp the original image 302 to provide an output image 308. Any suitable warping technique may be used. By using the warping grid 306 that is based on the depth map and has been regularized, the output warped image has reduced perspective foreshortening distortion compared to the original image 302 and is a better representation of the object as perceived by others in reality.

[0055] Example Method

[0056] Fig. 9 An example method 900 for correcting foreshortening distortion on a face is shown. The method 900 may be performed by an electronic device 102 that uses a camera system 104 to capture an input image and uses a distortion correction module 110 to correct foreshortening distortion. The method 900 is shown as a set of blocks that specify operations performed, but is not necessarily limited to the order or combination of operations shown for performing the corresponding blocks. In addition, any of one or more operations may be repeated, combined, reorganized, or linked to provide a variety of additional and / or alternative methods. In the following discussion, reference may be made to the following. Figure 1 An example operating environment 100 or Figure 2-8 Entities or processes described in detail in the present disclosure are referenced only as examples. The techniques described are not limited to being performed by one entity or multiple entities operating on one device.

[0057] At 902, a semi-dense depth map is generated for an object in an input image captured by a camera. Figure 1 The distortion correction module 110 in the image processing module 110 determines landmarks on the face of the object in the input image and fits a three-dimensional (3D) face model to the face. The distortion correction module 110 then uses the fitted 3D face model to render a depth map at the camera's viewpoint.

[0058] At 904, based on the depth map, a warp mesh is generated on the input image to correct foreshortening distortion. For example, the distortion correction module 110 generates the warp mesh based on the depth map and camera geometric characteristics such as focal length, orientation, and rotation. In addition, the distortion correction module 110 selects the center of the face region instead of the image center as the reprojection center for forward projecting points on the input image based on the depth map. The distortion correction module 110 also performs a mesh optimization process to smooth the warp mesh in the area between the face region and the background region.

[0059] At 906, the input image is warped according to the warp grid to provide an output image having reduced perspective foreshortening distortion compared to the input image. The distortion correction module 110 may use any suitable warping mechanism to warp the input image (e.g., the original image 302) based on the warp grid 306. For example, each point on the input image corresponding to a vertex on an unwarped grid (e.g., the 2D grid 502) placed on the input image is warped (e.g., translated) to new coordinates that map to the corresponding vertex on the warp grid 306.

[0060] Fig.10 and Fig.11A method 1000 for correcting foreshortening distortion in a selfie photograph is shown. The method 1000 may be performed by an electronic device 102 that uses a camera system 104 to capture an input image and uses a distortion correction module 110 to correct foreshortening distortion. The method 1000 is shown as a set of blocks that specify operations performed, but is not necessarily limited to the order or combination of operations shown for performing the corresponding blocks. In addition, any of one or more operations may be repeated, combined, reorganized, or linked to provide a variety of additional and / or alternative methods. In the following discussion, reference may be made to the method 1000. Figure 1 An example operating environment 100 or Figure 2-8 Entities or processes described in detail in the present disclosure are referenced only as examples. The techniques described are not limited to being performed by one entity or multiple entities operating on one device.

[0061] At 1002, an input image including a person's face is received. The camera system 104 of the electronic device 102 may capture the input image, such as image 302. The input image may be captured by a user 112 holding the electronic device to take a self-portrait (e.g., a selfie photo). The input image may be captured via the front camera 118 or the rear camera of the electronic device 102.

[0062] At 1004, a set of landmarks are detected on a face in the input image. In various aspects, the reconstruction module 206 may use any suitable detection technique on the input image to detect a set of landmarks representing facial features on a person's face.

[0063] At 1006, the generic face model is adjusted to incorporate the detected set of landmarks. In an example, the reconstruction module 206 performs a model fitting process to match the generic 3D face or head model to the landmarks, in particular by modifying the generic 3D face model to incorporate the landmarks.

[0064] At 1008, an adjusted 3D facial model having the face of the subject from the input image is output. In various aspects, reconstruction module 206 provides the modified 3D facial model (modified to fit the landmarks) to mesh module 208.

[0065] At 1010, the adjusted 3D facial model is refined by removing one or more features outside the facial region to provide a refined 3D facial model. In one example, the reconstruction module 206 removes the ears, neck, and back of the face from the 3D facial model. This is to allow distortion correction to occur primarily on the facial region of the input image.

[0066] At 1012, the thinned 3D facial model is smoothed to remove sharp contours caused by the thinning. The reconstruction module 206 may perform smoothing of the 3D facial model. Any suitable smoothing technique may be used.

[0067] At 1014, the depth map is rasterized based on the smoothed 3D facial model. In various aspects, the reconstruction module 206 generates the depth map (such as Figure 3 The depth map 304 in FIG. 1 is taken and rasterized. The depth map is based on the focal length of the camera that captured the input image.

[0068] At 1016, a two-dimensional (2D) grid is placed on the input image, the 2D grid having a plurality of vertices. In various aspects, the mesh module 208 places the 2D grid on the input image. The 2D grid includes a plurality of quadrilaterals (e.g., quadrilaterals formed by four vertices on the 2D grid).

[0069] At 1018, for each of the plurality of vertices corresponding to the facial region of the object in the input image, forward project the corresponding point on the input image based on the depth map and the camera geometry to generate a warped mesh. Figure 2 As described, the mesh module 208 forward projects points on the input image based on the depth map and camera geometry.

[0070] At 1020, the center of the facial region is selected as a reprojection center to preserve the shape of one or more objects worn by the subject in the facial region. Figure 6 As depicted, the mesh module 208 selects the bounding box centroid 616 of the face 604 in the second processed image 610. Due to the location of the reprojected center, the shape of the eyeglass 606-3 is maintained.

[0071] At 1022, an energy function is employed to balance between the projected vertices and the background scene. In an example, the mesh module 208 applies the energy function described with respect to Equation 1 as a mesh optimization process to make the warped mesh look natural and smooth around the face region, the background region, and between the face region and the background region.

[0072] At 1024, the image is warped according to an energy function (eg, a grid optimization process) to provide an output image. For example, the warping module 210 warps an input image (eg, the original image 302) based on a warping grid to provide a warped image (eg, the output image 308).

[0073] At 1026, the output image is stored as the captured image. For example, the output image 308 is stored in a memory (such as a storage memory of the computer readable medium 106). In various aspects, the output image can be stored as the captured image to replace the input image. Alternatively, the input image can also be stored in the memory in addition to the output image. The output image can also be displayed via the display device 108.

[0074] In general, any components, modules, methods, and operations described herein may be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example methods may be described in the general context of executable instructions stored on a computer-readable storage memory local and / or remote to a computer processing system, and implementations may include software applications, programs, functions, and the like. Alternatively or in addition, any functions described herein may be performed, at least in part, by one or more hardware logic components, such as, but not limited to, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SoC), a complex programmable logic device (CPLD), and the like.

[0075] Example computing system

[0076] Fig.12 It can be implemented as shown in the previous reference Figure 1-11 Various components of an example computing system 1200 are described for any type of client, server, and / or electronic device for implementing foreshortening distortion correction on a face.

[0077] The computing system 1200 includes a communication device 1202 that enables wired and / or wireless communication of device data 1204 (e.g., radar data, authentication data, reference data, received data, data being received, data scheduled for broadcast, and data packets of data). The device data 1204 or other device content may include configuration settings for the device, media content stored on the device, and / or information associated with a user of the device (e.g., the identity of a person within a radar field or customized air gesture data). The media content stored on the computing system 1200 may include any type of radar, biometric, audio, video, and / or image data. The computing system 1200 includes one or more data inputs 1206, via which any type of data, media content, and / or input may be received, such as human speech, interaction with a radar field, touch input, user-selectable input or interaction (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.

[0078] The computing system 1200 also includes a communication interface 1208, which can be implemented as any one or more of the following: a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and any other type of communication interface. The communication interface 1208 provides a connection and / or communication link between the computing system 1200 and a communication network, through which other electronic, computing and communication devices communicate data with the computing system 1200.

[0079] The computing system 1200 includes one or more processors 1210 (e.g., any of a microprocessor, controller, or other controller) that can process various computer-executable instructions to control the operation of the computing system 1200 and implement techniques for facial authentication embedding migration and drift compensation, or in which facial authentication embedding migration and drift compensation techniques can be implemented. Alternatively or additionally, the computing system 1200 can be implemented in any one or combination of hardware, firmware, or fixed logic circuits implemented in conjunction with processing and control circuits, which are generally identified at 1212. Although not shown, the computing system 1200 may include a system bus or data transmission system that couples various components within the device. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures.

[0080] The computing system 1200 also includes computer-readable media 1214, such as one or more memory devices that implement persistent and / or non-transitory data storage (as opposed to signal transmission only), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and disk storage devices. The disk storage device can be implemented as any type of magnetic or optical storage device, such as a hard drive, a recordable and / or rewritable compact disk (CD), any type of digital versatile disk (DVD), etc. An example of the computer-readable medium 1214 includes Figure 1 The computing system 1200 may also include a mass storage media device (storage media) 1216.

[0081] The computer-readable medium 1214 provides a data storage mechanism to store device data 1204, as well as various device applications 1218 and any other type of information and / or data related to the operational aspects of the computing system 1200. For example, an operating system 1220 can be maintained as a computer application with the computer-readable medium 1214 and executed on the processor 1210. The device applications 1218 can include device managers, such as any form of control applications, software applications, signal processing and control modules, code local to a specific device, abstract modules, air gesture recognition modules, and other modules. The device applications 1218 can also include system components, engines, modules, or managers for implementing perspective foreshortening distortion correction on the face, examples of which include the distortion correction module 110 and the camera system 104. The computing system 1200 can also include or have access to one or more machine learning systems.

[0082] in conclusion

[0083] Although implementations of techniques and apparatus for implementing perspective shortening distortion correction on faces have been described in language specific to features and / or methods, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, these specific features and methods are disclosed as example implementations of perspective shortening distortion correction on faces.

Claims

1. A method for correcting foreshortening distortion on a face in an image, the method comprising: Generate a depth map of objects in an input image captured by a camera, including: fitting a three-dimensional facial model using the face of the subject from the input image, wherein a rotation and a translation of the three-dimensional facial model relative to the camera are calculated; and rendering the depth map at the camera's viewpoint using the fitted three-dimensional facial model; Based on the depth map, generating a warp grid on the input image for correcting the perspective foreshortening distortion; and Warping the input image according to the warping grid to provide an output image having reduced perspective foreshortening distortion compared to the input image, comprising: A tilt correction is applied to the warp grid to correct the warping of the input image where the face is tilted.

2. The method according to claim 1, wherein: Generating the depth map includes: detecting a set of landmarks on the object in the input image; adjusting a general three-dimensional face model to incorporate the detected set of landmarks to provide an adjusted face model, thereby providing an adjusted 3D face model having the face of the subject; The adjusted 3D facial model is output.

3. The method according to claim 2, further comprising: refining the adjusted 3D facial model by removing one or more features outside a facial region of the adjusted 3D facial model to provide a refined 3D facial model; smoothing the refined 3D facial model to remove sharp contours caused by the refinement; and The depth map is rasterized according to the smoothed 3D model.

4. The method according to claim 3, wherein: Removing the one or more features includes removing one or more of the following from the adjusted 3D facial model: ears, back of the face, and a neck.

5. The method according to claim 1, wherein: Generating the warped mesh includes: Placing a two-dimensional 2D grid on the input image, the 2D grid having a plurality of vertices; For each vertex of the plurality of vertices corresponding to a facial region of the object in the input image, forward projecting a corresponding point on the input image based on the depth map and camera geometry; and The center of the facial region is selected as a reprojection center to maintain the shape of one or more objects worn by the subject in the facial region.

6. The method according to claim 5, wherein: The forward projection comprises: determining a gaze direction of the object in the input image; placing the virtual camera at a position lower than the camera based on the gaze direction being below the camera; and The virtual camera is tilted upward such that a result of the warping changes the gaze direction toward the camera.

7. The method of claim 5, further comprising using an energy function for: Uniformly distributing the plurality of vertices locally; conforming each vertex in the facial region to the corresponding point; reducing warping of vertices adjacent to a contour of the facial region; and Prevents warping of vertices outside the facial region.

8. The method according to claim 7, wherein: The energy function includes a face term, and the method further includes determining the face term by: applying a lower weight to vertices adjacent to a contour of the facial region than other weights applied to vertices within the facial region that are not adjacent to the contour; as well as The vertices within the facial region are allowed to be translated and scaled uniformly as a group.

9. The method according to claim 7, wherein: The energy function includes a regularization term, and the method further includes determining the regularization term by: adding filled vertices outside the warped mesh; and The filled vertices are clipped to reduce artifacts near the border of the input image.

10. The method according to claim 9, wherein: The regularization term is also determined by the following operation: penalizing a deformation direction of the warp mesh to reduce changes in orientation of adjacent links in the warp mesh; and Reduce the weights on vertices on the border of the facial region.

11. The method according to any one of claims 1 to 10, wherein: The input image is captured by the camera at a first distance from the object; and The output image appears as if the output image was captured by the camera at a distance from the object greater than the first distance.

12. The method according to any one of claims 1 to 10, further comprising: The output image is stored as a captured image.

13. The method according to any one of claims 1 to 10, wherein: The camera is integrated with the handheld electronic device.

14. An electronic device comprising: A camera for capturing images; a memory for storing the image; One or more processors, configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 13.

15. A non-transitory computer-readable storage medium storing instructions which, when executed by a data processing device, cause the data processing device to perform the method according to any one of claims 1 to 13.

16. A computer program product comprising instructions which, when executed by a data processing apparatus, cause the data processing apparatus to perform the method according to any one of claims 1 to 13.

Images