A method for editing a facial image

Abstract

Provided is a method for editing a facial images in an image Performed is 3D GAN inversion for a distorted input facial image by optimizing the camera parameters and face latent code, which gives a generated image. A Triplanenet encoder is used to obtain face latent code. Besides, the obtained latent code and novel camera parameters are used to estimate depth and create a 3D mesh afterward, and render this mesh to obtain a warped portrait. Finally, applied is visibility-based blending so that visible regions are reprojected, and occluded parts are restored with a generative neural network. The proposed invention provides a desired head pose and / or exclude perspective distortions. 36

Description

DESCRIPTION Title of Invention A METHOD FOR EDITING A FACIAL IMAGE Technical Field

[0001] The present invention relates to image correction, namely to the correction of facial distortions and head pose in an image, for example, in a selfie photograph obtained from a camera. Background Art

[0002] Selfie are arguably the most common type of photos taken with a smartphone. Modern smartphone cameras provide high-quality images, yet they experience unwanted issues with facial geometry.

[0003] Camera distance plays a vital role in portrait perception. Selfie photos taken from close distances often suffer from perspective distortion, which reveals itself in malformed and asymmetrical traits, enormous nose and tiny or even hidden ears, providing unnatural and unflattering images.

[0004] Another 3D geometry-related issue is a misplaced head pose. Posing for a selfie requires practice, and selecting an appealing viewpoint is non-trivial: it either requires multiple attempts to adjust or can be even impossible in some scenarios. So, the ability to modify the head pose during the post-processing is a demanded feature in selfie editing.

[0005] Various approaches to face geometry manipulation have been proposed so far, either based on 2D and 3D warping, neural radiance fields, or generative models (neural networks) to synthesize a portrait with updated viewing conditions.

[0006] Known 2D warping-based undistortion methods rely on estimating a 2D flow map to warp an image. Such methods struggle with severe distortions due to inaccurate 3D geometry fitting.3D warping-based approaches perfectly preserve the details of the original image, yet, they are also unable to fill in occluded regions that arise inevitably.

[0007] NeRF-based methods provide a full control of camera parameters for novel view synthesis, but do not leverage face priors. Accordingly, the optimization is performed from scratch, thus being very slow: despite certain progress, these approaches are far from a real-time performance.

[0008] Another branch of face image manipulation technique relies on a usage of generative neural networks. In a generative pipeline, the original face image is encoded into a latent representation and further restored under novel viewing conditions and updated camera parameters with a pretrained 3D-aware GAN. The facial latent code, camera pose, and focal distance are estimated throughout a joint optimization procedure. However, fitting these parameters from a single distorted image is a challenging and ill-posed task, so existing methods struggle to restore the 3D geometry accurately. While this can be improved to a certain scale by imposing geometry constraints, there is an immanent major drawback: generative approaches cannot guarantee that the identity of a person is preserved. Besides, GANs tend to miss fine-details, which severely affects the image quality.

[0009] Known 2D GAN (Generative adversarial network) inversion methods do not maintain the facial structure during the transformation, so the multi-view consistency is not guaranteed. Recently introduced 3D GANs demonstrated their capability of generating consistent outputs based on implicit 3D representations.

[0010] To restore a face under novel viewing conditions, the original face should first be mapped into the latent space of a pretrained GAN. This technique is referred to as a GAN inversion.3D GAN inversion methods leverage 2D GAN in-version methods with pre-defined camera parameters, which can be estimated with another method and then fixed or further tuned along with the face latent code optimization.

[0011] Recent DisCO proposed a sophisticated 3D inversion scheme of optimizing the face latent code and camera parameters, which includes initializing with a short camera-to-face distance and reparametrizing the camera. The results achieved with a multi-stage optimization schedule with landmark and geometric regularization, look promising. Yet, an optimization-based GAN inversion is slow, and as a purely generative approach, DisCO does not preserve identity and outputs images lacking fine details.

[0012] Rather than by optimization, GAN inversion can be performed with encoder- based techniques, which transform an input image into the latent space with a single forward pass and are orders of magnitude faster. Up until recently, optimization- based methods have been superior in reconstruction quality, but the latest encoder- based methods are capable of providing as accurate and more view-consistent geometry [1].

[0013] In HFGI3D [2], training a GAN is supervised with images obtained by warping, making synthesized images more realistic. Hence, the output is entirely GAN-generated, while warping is used only as a guidance. On the contrary, in the suggested pipeline of GAN, warping directly contributes to the final image.

[0014] When changing head pose, disocclusions are inevitable and more prominent, since yet invisible parts of human head and face get revealed. Thus, warping-based methods are insufficient for head pose correction if applied solely, and trainable 3D- aware methods dominate the field.

[0015] One line of work implies using NeRF-based methods for single-view

[0016] or few-shot

[0017] novel view synthesis. However, these approaches may not preserve the identity of a person since no face prior is used.

[0016] Known GAN-based approaches aim at preserving person-specific details via conditioning on an input image or video

[0018] , latent code

[0019] or encoded facial attributes

[0020] . Still, maintaining identity cannot be guaranteed, and generated images exhibit slightly different traits.

[0017] Therefore, it is necessary limitation of different face manipulation paradigms, GAN / NeRF / warping-bases approaches, through performing combining warping from visible parts and generation for occluded parts. Description of Drawings

[0018] The above and / or other aspects will be more apparent from the following description of exemplary embodiments with reference to the accompanying drawings, in which:

[0019] Fig.1 illustrates examples of results of selfie editing according to the proposed invention.

[0020] Fig.2 illustrates an overview of the proposed pipeline.

[0021] Fig.3 illustrates first to the left: the HeRo capturing setup with smartphones assembled on a rig, other: series of photos of the same individuals, simultaneously captured by Front, Left, Right, and Top cameras.

[0022] Fig.4 illustrates qualitative comparisons on the CMDP dataset.

[0023] Fig.5 illustrates in-the-wild images with corrected distortion.

[0024] Fig.6 illustrates LPIPS, SSIM, ID and PSNR scores for different number of iterations.

[0025] Fig.7 illustrates examples of pose correction for a sample from the HeRo dataset. Best Mode Images, for example, self-portraits captured from a short distance might look unnatural or even unattractive due to heavy distortions causing facial features to be malformed, and ill-placed head poses. According to the proposed method performed is 3D GAN inversion for a distorted facial image by optimizing camera parameters and face latent code, which gives a generated image. Applied is the visibility-based blending so that visible regions are reprojected, and occluded parts are restored with a generative neural network. Experiments on face undistortion benchmarks and on the self-collected Head Rotation dataset (HeRo), demonstrate that the claimed method outperforms previous approaches both qualitatively and quantitatively, and thereby opens new possibilities for photorealistic selfie editing. Normalize portraits (face images) is giving a desired head pose and / or exclude perspective distortions. Proposed is a method for editing a facial images in image, the method comprising:а) selecting, by the user, an image for editing, comprising an image of at least one face; b) detecting face images in the selected image, segmenting one face image among detected face images; с) feeding the segmented image of the face to input of a neural network, that predicts camera position and parameters for the segmented image of the face, obtaining an initial camera position and parameters at the output; d) feeding the segmented face image of the face to input for a neural network, that predicts the face latent code, obtaining an a face latent code at the output of the neural network, that predicts the face latent code; steps (c) and (d) are implemented in parallel; e) implementing a iterative optimization process of the 3D GAN, for this purpose: feeding the predicted camera position and parameters and the predicted face latent code to input of 3D GAN (generative adversarial network), obtaining a generated image at the output of the 3D GAN, determining a loss function between the segmented face image and the generated image, iterative changing the predicted camera position and parameters and the predicted face latent code, feeding the changed camera position and parameters and the changed face latent code to input of 3D GAN, wherein, the iterative optimization process is conducted until the loss function reaches the minimum of the function, wherein a camera position and parameters anda face latent code satisfying this condition are the optimal camera position and parameters and the optimal face latent code; f) feeding the optimal face latent code to input of 3D GAN, feeding voluntary new camera position and parameters to input of 3D GAN; g) generating, by the 3D GAN a new generated image with face image corresponding to the segmented face image with a new facial angle corresponding to the new camera position and parameters, predicting a depth map of the new generated image; h) processing the predicted depth map for building a 3D mesh of the new generated image; i) projecting the built 3D mesh onto an image plane with the optimal camera position and parameters, generating a rendered image based on this projection; j) determining which 3D mesh vertices are visible and which are occluded after projection onto the image plane with the optimal camera position and parameters, obtaining visibility mask based on the said determination; steps (i) – (j) are implemented in parallel; k) blending the new generated image and the rendered image by use the visibility mask, obtaining an edited segmented face image showing the face, with new facial angle; l) passing the obtained the edited segmented face image into the selected image; the steps (b) – (k) are performed for at least one face in the selected image;m) displaying the selected image with at least one edited segmented face image on a screen to the user. At that the voluntary new camera position and parameters can be selected by the user. The proposed method can further comprise step of selecting by the user one face image for segmentation among detected face images in the selected image. At least one of the plurality of modules may be implemented through an AI model. A function associated with AI may be performed through the non-volatile memory, the volatile memory, and the processor. The above-described method performed by the electronic device may be performed using an artificial intelligence model. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or AI model of a desired characteristic is made. The learning may be performed in a device itself in which AI according to an embodiment is performed, and / or may be implemented through a separate server / system.The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks. The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. Detailed description Provided is a novel approach for face image (portrait) distortion and head- pose correction using perspective-aware 3D GAN inversion. The proposed method performs the functions of correcting face distortion and head pose, for example in selfie photos or any photos comprising face obtained from cameras. Using the proposed method, an initial image is processed and a corrected image is obtained. Using the proposed invention, determining the texture, geometry and detail of theface at different camera positions is carried out, resulting in selfie photographs with corrected facial distortion and corrected head pose. The proposed invention allows to correct the rotation of a person’s head or remove perspective distortions of the original image. The proposed method can effectively facilitate correction of camera portraits (for example selfies taken in real time or portraits selected by an user from a memory of the device, such as for example a smartphone) to aesthetically improve images. The proposed method can be used in applications for image to normalize portraits, for further processing in image / video analysis applications; to enhance videos streamed from the frontal camera, e.g., in applications for video conferencing or video blogging. The proposed invention can be used for smartphone, laptop, smart device or any electronic device able to perform computer-based image analysis, connected with an RGB camera, or obtaining RGB images from other source. The proposed method can be implemented for a computer with a GPU. The minimum required set of components includes a storage device, a processor for processing information, a graphics processor. The proposed method can be stored in the form of computer code on a computer-readable medium and can be implemented on the computer. The proposed invention integrates geometry-based and generative techniques, offering a robust solution for generating high-quality, identity-preserving images from novel views, for example, in the selfie context. Due to natural limitations of 3D data, known 3D Warping-based approaches (the 3D Warping Pipeline) tend to miss occluded regions. In the meantime, known Generativeapproaches (the Generative Pipeline) cannot guarantee that the identity of a person is preserved, as the generated face might differ from the original one. The proposed method combines the strengths of generative and warping paradigms, and leverages the power of a generative neural network while taking the most of a 3D-based warping approach (that is, when the deformation of the original frame occurs using some 3D information). Term “warping” indicates the deformation of the original frame, term “generative” indicates artificial image generation. As mentioned above, the proposed invention allows to correct the rotation of a person’s head or remove the perspective distortion of the original image. For example, in the original image the head is turned to the left, the user wants to get an image from this original image in which the head is turned to the right. With any such manipulation, new areas of the face that were not visible in the original image will be visible in the final image, that is previously occluded regions of the face will be visible in the image of the face from a new angle. The occluded regions are taken from a generated image, which allows restoring invisible parts of a face and head. Meanwhile, the visible part of the face is reprojected rather than generated, so the identity is preserved. Generation is obtaining a synthetical, novel image. Reprojected image is a «3D-warped» original image. The superior performance of the proposed method is proved on face undistortion benchmarks and the self-collected Head Rotation dataset (HeRo), dubbed HeRo, containing a number of identities with diverse head poses and camera-to-face distances. The HeRo dataset has been collected by the authors and can be used to evaluate methods aimed at changing the pose of the human head.Fig. 1 illustrates examples of results of selfie editing according to the proposed invention. The top line (original) shows the selfie images captured by the users, which are the original images that are fed to the GAN. The bottom line (corrected) shows the images obtained after processing according to the present invention. As follows from fig.1, according to the present invention it is possible to modify a head pose and remove perspective distortion seam-lessly, obtaining photorealistic and detailed corrected portraits. As will be described in detail below the novelty of the proposed invention consists in employing 3D GAN to generate a new generated image and a depth map, then constructing a 3D mesh from the depth map. Based on the 3D mesh, the original image is warped for obtaining a rendered image, and determined is which elements should be extracted from the new generated image and which from the rendered image by visibility analysis. Finally, the generated image and the rendered image are blended to form a final result. The known from the art pure 3D mesh warping methods cannot recover unseen parts of the face. Typically such areas include cheeks and ears in an undistortion case and the unseen side of a face in case of a head rotation. Depending on the required degree of portrait correction, the inpainted regions may range from just a few pixels to a noticeable portion of the image. The proposed invention uses GAN both to generate new generated (synthesized) image (to recover unseen facial details) and to recover depth information to 3D mesh warping. Thus, both new generated image and rendered image are aligned to each other, so it can be simply blended by visibility mask, as will be described below.The proposed method is shown schematically in Fig. 2. The proposed method consists of the following steps: 1) User selects an image, comprising at least one face image, for editing. Selection is implemented from the memory of the device or the user takes a photo with the camera and selects this photo for editing. The photo may be a selfie portrait or any other image containing at least one face. The user can select at least one face image from the selected image for editing and can also select a voluntary new camera position and parameters, i.e. new angle of the head in the selected image (as will be described in step 5). 2) Detection and segmentation of one face image in the selected image are conducted, for obtaining an input face image (“Input” in fig.2) representing a segmented face image. The detection process involves detecting face images in the selected image. The segmentation process involves cutting out the one face image from the selected image. The proposed method implements editing one face image. For editing all face images, it is necessary to apply the proposed method to each face image separately. The face image to be edited can be either selected by the user in step 1 or all face images in the selected image are edited one by one. Segmentation is carried out using a suitable preprocessing pipeline taken from the corresponding GAN, such pipeline are known from the art (for example [6]). 3) Initialization. For rendering images of the same face, but turned at a different angle (new face angle) with a 3D GAN, it is primarily needed to obtain an initial face latent code of the segmented face image and the initial camera position and parameters for an segmented face image.Camera position is a matrix which specifies the position of the camera in space. It can be represented as a matrix of rotation and translation, defining a projective transformation from point X in 3D space to point x (2D coordinates in image space). The camera parameters is a matrix characterizing the internal camera parameters, for example, in a simple case, the perspective camera has 4 parameters that determine the projective transformation on the image plane: - Cx and Cy, which are define projection of the camera optical center to the image plane - Fx and Fy – camera focal length in pixels. (Note that Fx and Fy may be different because the aspect ratio of a pixel is not guaranteed to be one. In case of exactly square pixels Fx = Fy). More complex camera models are known, which are the basis for more parameters that are known in the art and are not relevant to the development of the invention. The initial camera position and parameters are unknown initially. The initial camera position and parameters are determined using a neural network (as described below). Obtaining the initial face latent code of the segmented face image and the initial camera position and parameters for an segmented face image are implemented as follows: a) An initial camera position and parameters are determined according to the segmented face image. In particular, the segmented face image of the face is fed to the input of a neural network, that predicts the camera position and parameters (Deep3D FaceRecon in fig.2). After processing, the mentioned neural network produces the initial camera position and parameters c0 at the output. c0 isa tensor describing the camera position and parameters. Processes and suitable neural networks for determining the camera position and parameters are known from the prior art (for example, [7]). Figure 2 illustrates the known Deep3D FaceRecon neural network [7] as an example. b) An initial a face latent code (it is indicated as w0 in fig.2) are determined according the segmented face image of the face. The segmented face image of the face is fed to the input of a neural network, such as TriPlaneNet in fig.2, that predicts the face latent code. After processing, the mentioned neural network produces the initial a face latent code w0 (the TriPlanetNet neural network is illustrated as an example). The latent code is a tensor describing the image. The use of the latent codes is known from the prior art (see, for example,

[0021] ). Processes for determining the face latent code are known from the prior art (for example, [1]). TriPlanetNet is learned to predict the latent code for the 3D GAN so that by using this latent code and determined camera parameters it is possible to obtain an image close to the original one. Accordingly, it is trained on photographs of different people (real or synthetic) and learns to predict the latent code from the image. Steps (a) and (b) are implemented in parallel and is called the initialization process (initialization in fig.2). 4) Optimization. The part of 3D GAN (generative artificial neural network (Generative Painplane)) does not produce an output image sufficiently similar to the segmented face image when using the initial parameters w0 and c0. So it is necessary to make a few additional optimization steps in the 3D GAN to get a generated image output (Generated Image in fig.2) that is very similar to the inputsegmented face image. An optimization process (see fig.2 optimization) is used for this purpose: feeding the predicted camera position and parameters and the predicted face latent code to input of the 3D GAN, obtaining a generated image at the output of 3D GAN, determining a loss function (loss function in fig.2)between the segmented face image and the generated image, changing the predicted camera position and parameters and the predicted face latent code, feeding the changed camera position and parameters and the changed face latent code to input of 3D GAN. The iterative optimization process is conducted until the loss function reaches the minimum of the function. A camera position and parameters and a face latent code satisfying this condition are the optimal camera position andparameters and the optimal face latent code. When using the obtained parameters, the identity of the image obtained by 3D GAN to the original image willbe maintained, i.e. the optimal camera position and parameters and the optimal face latent codecorrespond to the initial segmented face image. It should be noted that during the iterative optimization process the 3D GAN weights are not changed, only w and c are changed, for example, by the gradient descent method using the loss function. Gradients are calculated using the backpropagation method through the 3D GAN network. In this way, it is calculatedwhich values of the vectors w and c make the greatest contribution to this error and they are changed to reduce this error. The loss function shows how one image is similar to another one. The larger this function, the less similar the images are to each other. In the proposed invention, the loss function consists of two components: - LPIPS loss is a measure of the similarity of one image to another one, which works close to human perception of images (known from the state of the art https: / / richzhang.github.io / PerceptualSimilarity / ), and - loss on landmarks, which is defined as the sum of the distances between key points of the face. The loss function can be calculated by any method known from the state of the art, the main thing is that it satisfies the main condition - it determines the degree of similarity of two images. Example of the optimization camera position and parameters (c0): Following DisCo [3], an initial camera translation(an initial distance from the camera to the center of the face (head)) is set to be reasonably small.is a distance from the camera to the center of the face with the help of which optimization the focal distance f of the camera is carried out, during optimizationis selected experimentally, and if too close (that is, the camera position is too close to the face) a distance is took, the face will go beyond the image; if too far (that is, the camera position is too far from the face) a distance is took, this also worsens the convergence, since convergence may remain in a local minimum that does not correspond to the real distance. Ideally, it is necessary to choose an initial distance that will ensure stable convergence of thealgorithm. Selecting an initial distance experimentally on a large number of photographs and the optimal initial approximation is selected. According to changing tz (the distance from the camera to the center of the face (head)) the focal distance f of the camera changes from the original f0 (f in this example represents camera position and parameters (c0)). Each time tz changes, a new value of f is obtained. At that, used is an estimated depth of eyes d0 in the initial face image and it remains unchanged. The estimated depth of eyes d0 is estimated for the segmented face image and represents the distance from the camera to the eyes in the segmented face image. When processing data containing a new value f, using 3D GAN, a new image is obtained. Based on the new image a loss function is calculated as will be described below. The focal distance f of the camera: whered0 is the estimated depth of eyes, tz0 is the initial distance from the camera to the center of the face (head). In details: after the initialization process, the face latent code and the camera position and parameters are adjusted jointly (an iterative process known from the prior art like a gradient descent method is used): Expression (1) below indicates that under the optimal face latent code and the optimal camera positionand parameters) the condition the minimum L ( ) is achieved. Thus,the optimalare those face latent code and optimal camera position and parameters for which loss function L reaches the minimum point of the function:(1) where is an image produced by a pre-trained 3D GAN, parameterized with weights over the latent w and camera parameters c; x — the input segmented face image. More precisely, the objective is given as a combination of the LPIPS loss, which compares the perceptual similarity of the input image x and the generated image G(w, c; ), and face landmark lose, the face landmarks are special image points (for example, face contour, eyes), that is, the sum of the distances between the key points of the face in the input segmented face and in the generated one is compared:L — losses, the sum of two losses (LPIPS loss and loss on landmarks) with coefficients, x — input segmented face image, G — 3D GAN, accepts as input w — face latent code, c — camera position and parameters; G depends on - GAN parameters;и — the weights with which the losses are added; LPIPS – loss, which compares the perceptual similarity of the input image x and the generated image G(w, c; ), f(x) calculates landmarks in the image, calculates landmarks in the generated image. The landmarks are determined using a neural network that performs optimization (for example, MediaPipe FaceMesh-V2 landmark estimation model [8]) on the segmented face image, a new image is generated using 3D GAN so that Landmarks in the generated image have the same locations as in the segmented face image. Landmark loss is the sum of the distances between the key points of the face in the original photo and the generated one is compared. In the formula below, the landmark loss is the sum of the squares of the distances:- a set of (x, y) coordinates of points on the generated and reference (original image) images, respectively, we have ||M|| number of sets of (x, y) coordinates of points. Lossconsiders the total deviation of all sets of (x, y) coordinates of points from the reference ones (that is, from the coordinates of the points of the original image (the greater the distance between key points, the higher the loss function will be) where m and are normalized 3D keypoints, andis a number of landmark points, no additional constraints are imposed based on landmark uncertainty. The latent code and camera parameters are optimized with a learning rate of 0.001 over 200 iterations. Instead of an exhaustive optimization-based GAN inversion exploited by DisCo [3], tackled is inversion problem with an encoder network E, that maps a real image into a latent code. A camera-conditioned TriPlaneNet encoder [1] is opted for, as it is able to separate geometry from camera effects, which is crucial in the case. During the optimization, alternated are computing gradients w.r.t a face latent code and camera, and they are optimized in a round-robin manner, that is, the parameters are optimized one by one. At the first step, the camera parameters are fixed and the latent code is optimized; at the second step, on the contrary, the latent code is fixed and the camera position and parameters are optimized, and so on until the optimal face latent code and the optimal camera position andparameters)are achieved. In general, this is just one optimization strategyand is not necessarily followed. Any other known method can be used. The experiments evidence that the alternating updating strategy provides superior results compared to the joint optimization employed by DisCO. Next steps relates to Novel view synthesis (see fig.2): 5)New Generated Image and Depth Map. The optimal face latent codeobtained at the previous step and voluntary new camera position and parameters and cnovel in fig. 2) are fed to input of the 3D GAN. The voluntary new cameraposition and parameters cnovel corresponds to new facial angle (new angle of the same face) and can be selected either arbitrarily or by the user when selecting animage for editing (step 1). For example, for selecting the new camera position and parameters cnovel the user can use sliders on the screen of an electronic device with which the user can set the head rotation to the sides, up and down, as well as the distance from the center of the head to the center of the camera. The 3D GAN generates new generated image (New Generated Image in fig.2) with face image corresponding to the segmented face image with new facial angle corresponding to the new camera position and parameters, i.e. from another different angle. The new generated image can have new previously invisible parts of the face (such as the right ear when turning left) without maintaining the quality (Original Resolution) of the original segmented face image. Also, the 3D GAN predicts a depth map (Depth Map in fig.2) of the new generated image. 6) 3D Mesh. The depth map is processed using the mesh-based deterministic 3D warping approach, for example, described in the document [9]. The 3D Mesh of the new generated image (representation of the face in 3D in the form of a 3D polygonal mesh) is built using the predicted depth map (3D Mesh in fig.2). 7) 3D mesh based warping, Rendered image. Then, this 3D mesh, which is a parametric 3D approximation of the shape and surface of a three-dimensional object, is projected onto an image plane with the optimal camera position and parameters. This projection gives an idea of how the 3D polygonal mesh (the geometric representation of the face in 3D space) would be seen if that face position is shot at the optimal camera position and parameters predicted for the initialsegmented face image (arrows from the 3D Mesh and to "3D mesh based warping” in fig.2). Based on this projection, a rendered image is generated (rendered image in fig.2). The rendered image is obtained only by using pixels of the segmented face image. For the rendered image, areas that were not visible on the segmented face image are not generated, but only pixels from the segmented face image are used. The rendered image displays how the face will be visible from a new angle, that is, with a given new camera position and parameters (cnovel). In other words, a rendered image is the image obtained by transforming the original image using mesh warping (a method known from the prior art). Figuratively speaking, the initial segmented face image is stretched onto the 3D MESH, and the 3D MESH becomes colored in the colors of the initial segmented face image. And then this colorized 3D MESH is displayed from a different angle (according to the new camera parameters cnovel). This may reveal parts that are not visible in the initial segmented image; in this case, “holes” are formed in the image, which are determined by analyzing the visible parts. These holes are filled with parts of the generated image during the blending step (see step 8). 8) Vertex visibility analysis, Visibility mask. Determined is the visibility (visible or invisible) of each vertex of a 3D polygonal mesh when 3D polygonal meshis projected onto an image plane with the optimal camera position and parameters (arrows to the “Vertex visibility analysis” in fig. 2). Particularly, first computed is the visibility for each vertex in the mesh using the z-buffer of arasterization method. Secondly, mesh faces that are almost orthogonal (comprise an angle exceeding 80o) are filtered out with the viewing direction. Since the face is represented as a 3D mesh (polygonal mesh), some vertices of this mesh are not visible from different angles. Based on such determination, visible vertices (not being occluded, for example, when a person does not look at the camera with his left side, the right side is not visible) are selected such that the corresponding normal of a polygonal mesh triangle looks into camera (the normal to the triangle is the direction perpendicular to the plane of the triangle). A visibility mask is generated according to the above analysis. The use of visibility masks in computer graphics and rendering is known in the prior art (see for example VM Visibility Mask (telecomtrainer.com) . The Visibility Mask (VM) is a technique used to optimize the rendering process by selectively rendering only the visible parts of a scene. The Visibility Mask is a buffer or data structure that keeps track of the visibility of individual pixels or fragments in the scene. By connecting neighbor vertices in the mesh, a coarse mesh is obtained. It is further refined though bilateral blur smoothing in order to avoid any unnatural sharp angles appearing in a rendered image. Used is a kernel size of 5. Finally, the mesh is projected using an estimated original camera pose to get texture coordinates, and the texture is resampled to get a novel view. 9) Blending, Result. The new generated image and the rendered image are blended using the visibility mask (Blending in fig.2). Accordingly, the two images are well-aligned by design, which allows composing them with a minimal effort. Image blending is a technique known in the art that allows to insert part of one image into another in such a way that the composition of the images looks natural,without seams at the borders of the insertion. Image blending is produced by the method of adding two images (the new generated image and the rendered image) with different weights, and the weights from the visibility masks are used, and used is a method of smoothing the boundaries of two added images. Such methods are known from the prior art, for example, Laplacian Pyramid Blending: http: / / graphics.cs.cmu.edu / courses / 15-463 / 2005_fall / www / Lectures / Pyramids.pdf. Through this operation, areas are synthesized that were not visible at the original facial angle, but became visible at the changed facial angle (Result in fig. 2). The described method is focused on the cropped face regions solely however, the method could easily be extended to handle full-frame images, e.g., by following the steps described in [3]. Faces are detected with [4] and masked using a MODNet [5]. EXPERIMENTS Datasets Evaluated is face undistortion on two publicly available distinct datasets. Caltech Multi-Distance Portraits (CMDP) [9] contains frontal portraits of 53 individuals with various face attributes, each photographed from seven distances. In the wild images [3] features in-the-wild severely distorted portraits, scraped from the web. Since there are no references or ground truth images, used are only these images for qualitative comparison. Head rotation datasetsCollected are the Head Rotation (HeRo) dataset, containing portraits of 19 people with varying attributes (glasses, facial hair, facial expressions). Overall, there are 68 series of four photos in each series. Portraits were taken using frontal cameras of four Samsung Galaxy S23FE smartphones fixed on a rig Fig.3. Figure 3 illustrates the installation on which the dataset was collected, and samples from it. Our invention was subsequently tested on this dataset. From the collected photos, we have what, for example, a person should look like from the left angle, having a front angle. Accordingly, the proposed method was tested on the front photo and head pose correction was made in left / top / right view, then these photos were compared with real photos. All the devices were synchronized with 1, so that series of 4 images were captured simultaneously. Metrics Used are four standard evaluation metrics to assess the portrait perspective correction. Specifically, calculated are photometric errors between the aligned output images and corresponding references, including PSNR, SSIM, and LPIPS

[0010] . Besides assessed is identity preservation with the ID score, which is the cosine distance between predicted and reference face features from ArcFace

[0011] . RESULTS Face Undistortion Presented is the evaluation of the claimed approach against competing face undistortion methods in Table 1. Evidently, the claimed method notably outperforms others in terms of most metrics, especially in identity preservation. Due to thesampling of pixels from the original image, the claimed solution keeps crucial details of identity, such as eye color, wrinkles, earrings, etc. Table 1: Quantitative comparison of face undistortion methods on CMDP [9]. The best scores are marked bold. table 1This table 1 compares our method with competitors.4 metrics are used that compare pictures in pairs. PSNR, SSIM compare the quality of photographs, LPIPS compares the perceptual similarity of images, ID compares the similarity of personalities. An up arrow means more is better, and a down arrow means less is better. The proposed method outperforms competitors in all metrics. The portraits corrected through a variety of approaches, including the claimed solution, are depicted in Fig.4. Illustrated is a comparison of our distortion correction method with other methods. The claimed method excels in addressing severely distorted faces. The neural network not only successfully restores occluded regions, but also preserves crucial identity details, as highlighted in crops. Two examples are used, the first column contains the input image, the last column contains what should be produced. The second and fourth lines show the detailsfrom lines 1 and 3 to show the effectiveness of our method. The proposed method preserves details, for example, we preserve earrings and eyes, but 3DP, HFGI3D, TriplaneNet, DisCO do not. At the same time, in lines 1 and 3 it is clear that the perspective distortion has also been corrected, compared to the Fried’s and Shih’s methods. Warping-based [9] and

[0012] do not seem to have a major effect on inputs. In contrast, 3DP introduces noticeably changes, yet amplifying distortions, so that the middle part of a face exhibits less distortion, but the head and chin are malformed. Generative TriplaneNet and DisCo make face looking noticeably different. HFGI3D [2] manages to preserve identity by combination of warping and generation, but produces visual artifacts and tends to oversmooths images. The claimed method generates faces with fewer perspective distortions while maintaining identity, as shown in Fig. 5. The proposed method (OUR in Fig. 5) shows a realistic output, generation of new, previously invisible areas, and preservation of the original quality. The claimed method successfully attains a balance between new facial parts generation and identity preserving. To identify a necessary and sufficient number of optimization steps, the number of iterations is varied and the quality achieved is assessed. As can be observed from Fig.6. The graphs show how the number of optimization iterations of face latent code and camera parameters affects the output quality. The x-axis shows the number of iterations, and the y-axis shows the quality metric of the algorithm. For all quality metrics, we need about 200 optimization steps, where the method reaches a plateau in quality. Optimal quality is achieved during the first 100 iterations, while further optimization brings a negligible growth of ID, yet does not improve LPIPS, SSIM and PSNR.PSNR peaks after 100 iterations and then decreases slightly as the number of iterations grows. Respectively, used are 200 iterations by default. Note that the baseline DisCorequires approximately 1200 iterations of a comparable complexity. Head Pose correction Presented is the evaluation of the approach against competing head pose correction methods in Table 2. Qualitative results are presented in Fig. 7. The proposed method is able to change the pose of a person’s head, while preserving the original image quality and the person’s similarity to himself better than anyone else. The major advantage of the suggested approach is identity preservation, as reflected in an exceptional ID score. Table 2. Quantitative comparison of head pose correction on the HeRo dataset. table 2This table compares our method with competitors.4 metrics are used that compare pictures in pairs. PSNR, SSIM compare the quality of photographs, LPIPS compares the perceptual similarity of images, ID compares the similarity of personalities. An up arrow means more is better, and a down arrow means less is better.CONCLUSION In this application, provided is a selfie editing method that eliminates face perspective distortion in a close-up face crop. The approach enriches solid, physically correct 3D warping with the flexibility and expressiveness of a 3D generative neural network. The resulting image is a blend of a warped image obtained through mesh-based rendering and another image produced with a 3D GAN. Applied is the visibility-based blending that visible regions are reprojected, and occlud-ed parts are restored with a generative model. Evaluation in face undistortion benchmarks and the novel Head Rotation dataset proved that the claimed method provides more realistic results with finer details and better preserved identity, compared to existing selfie editing techniques, and establishes a new state-of-the-art in face undistortion and head pose correction tasks. REFERENCES [1] Ananta R. Bhattarai, Matthias NieBner, and Artem Sev- astopolsky, "Triplanenet: An encoder for eg3d inversion," 2024. [2] Jiaxin Xie, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen, "High-fidelity 3d gan inversion by pseudo-multi-view optimization," 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 321-331, 2022. [3] Zhixiang Wang, Yu-Lun Liu, Jia-Bin Huang, Shin'ichi Satoh, Sizhuo Ma, Gurunandan Krishnan, and Jian Wang, "Disco: Portrait distortion correction with perspective-aware 3d gans," 2023.[4] Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann, "Blaze- face: Sub-millisecond neural face detection on mobile gpus," 2019. [5] Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, and Rynson W.H. Lau, "Modnet: Real-time trimap-free por-trait matting via objective decomposition," in AAAI, 2022. [6] Eric Chan, Connor Z. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, S. Khamis, Tero Karras, and Gordon Wetzstein, "Efficient geometry-aware 3d generative adversarial networks," 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.16102-16112, 2021. [7] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong, "Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set," 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.285-295, 2019. [8] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grund-mann, "Mediapipe: A framework for perceiving and processing reality," in Third Workshop on Computer Vision for AR / VR at IEEE Computer Vision and Pattern Recognition (CVPR) 2019, 2019. [9] Ohad Fried, Eli Shechtman, Dan B Goldman, and Adam Finkelstein, "Perspective-aware manipulation of portrait photos," ACM Transactions on Graphics (TOG), vol.35, no.4, pp.1-10, 2016.

[0010] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang, "The unreasonable effectiveness of deep features as a perceptual metric," 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp.586-595, 2018.

[0011] Jiankang Deng, J. Guo, J. Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou, "Arcface: Additive an-gular margin loss for deep face recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp.5962-5979, 2018.

[0012] YiChang Shih, Wei-Sheng Lai, and Chia-Kai Liang, "Distortion-free wide-angle portraits on camera phones," ACM Transactions on Graphics (TOG), vol.38, no.4, pp.1-12, 2019.

[0013] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia- Bin Huang, "3d photography using context-aware layered depth inpainting," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8028-8038.

[0014] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or, "Pivotal tuning for latent-based editing of real images," ACM Transactions on graphics (TOG), vol.42, no.1, pp.1-13, 2022.

[0015] Jaehoon Ko, Kyusun Cho, Daewon Choi, Kwang seok Ryoo, and Seung Wook Kim, "3d gan inversion with pose optimization," 2023 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV), pp.2966¬2975, 2022.

[0016] Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhigang Wang Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo and Xuelong Li, One-ShotHigh-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field, arXiv preprint, arXiv:2304.05097, 2023.

[0017] Jiawei Yang, Marco Pavone and Yue Wang, “FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization”, Conference on Computer Vision and Pattern Recognition (CVPR) 2023.

[0018] Xueting Li, Shalini De Mello, Sifei Liu, Koki Nagano, Umar Iqbal, and Jan Kautz, “Generalizable One-shot Neural Head Avatar”, arXiv preprint.

[0019] Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Ogras and Linjie Luo, “PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360 degree”, arXiv preprint, arXiv:2303.13071, 2023.

[0020] Yue Wu, Yu Deng, Jiaolong Yang, and Fangyun Wei, Qifeng Chen, and Xin Tong, “AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatars”, Advances in Neural Information Processing Systems (NeurIPS), 2022.

[0026]

[0021] Karras, T., Laine, S., & Aila, T. (2018). A Style-Based Generator Architecture for Generative Adversarial Networks.2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4396-4405. https: / / arxiv.org / abs / 1812.04948.

Claims

Claims

1. A method for editing a facial images in an image, the method comprising: а) selecting, by the user, an image for editing, comprising an image of at least one face; b) detecting face images in the selected image, segmenting one face image among detected face images; с) feeding the segmented image of the face to input of a neural network, that predicts camera position and parameters for the segmented image of the face, obtaining an initial camera position and parameters at the output; d) feeding the segmented face image of the face to input for a neural network, that predicts the face latent code, obtaining an a face latent code at the output of the neural network, that predicts the face latent code; e) implementing a iterative optimization process by the generative adversarial network (3D GAN), for this purpose: feeding the predicted camera position and parameters and the predicted face latent code to input of 3D GAN, obtaining a generated image at the output of the 3D GAN, determining a loss function between the segmented face image and the generated image, iterative changing the predicted camera position and parameters and the predicted face latent code, feeding the changed camera position and parameters and the changed face latent code to input of 3D GAN,wherein, the iterative optimization process is conducted until the loss function reaches the minimum of the function, wherein a camera position and parameters and a face latent code satisfying this condition are the optimal camera position and parameters and the optimal face latent code; f) feeding the optimal face latent code to input of the 3D GAN, feeding voluntary new camera position and parameters to input of the 3D GAN; g) generating, by the 3D GAN a new generated image with face image corresponding to the segmented face image with a new facial angle corresponding to the new camera position and parameters, predicting a depth map of the new generated image; h) processing the predicted depth map for building a 3D mesh of the new generated image; i) projecting the built 3D mesh onto an image plane with the optimal camera position and parameters, generating a rendered image based on this projection; j) determining which 3D mesh vertices are visible and which are occluded after projection onto the image plane with the optimal camera position and parameters, obtaining visibility mask based on the said determination; steps (i) – (j) are implemented in parallel; k) blending the new generated image and the rendered image by use the visibility mask, obtaining an edited segmented face image showing the face, with new facial angle;l) passing the obtained the edited segmented face image into the selected image; the steps (b) – (k) are performed for at least one face in the selected image; m) displaying the selected image with at least one edited segmented face image on a screen to the user.

2. The method according to claim 1, in which the voluntary new camera position and parameters are selected by the user.

3. The method according to any of claims 1, 2, further comprising step of selecting by the user one face image for segmentation among detected face images in the selected image.

4. The method according to any of claims 1-3, in which the steps (c) and (d) are implemented in parallel.

Images