A face editing method and related apparatus

Abstract

Embodiments of the present application disclose a face editing method and related device. When performing face editing, a face image to be edited and a corresponding face editing style can be acquired first, and then an initial face image is generated, the initial face image including a target face corresponding to the face to be edited under the face editing style. A processing device can determine a face contour boundary point corresponding to the initial face image, the face contour boundary point being used to identify a face contour of the target face. Thus, according to the face contour boundary point, the face to be edited can be adjusted, so that a difference between a face contour of the adjusted face to be edited and the face contour of the target face is less than a preset threshold. When generating an edited target face image according to the adjusted face to be edited image and the initial face image, the face part and the background part in the target face image can be more natural, the realism of face editing can be improved, and the user's face editing experience can be improved.

Description

Technical Field

[0001] This application relates to the field of face processing technology, and in particular to a face editing method and related apparatus. Background Technology

[0002] Facial effects are one of the most popular interactive effects methods today. For example, through the facial effects function provided in software applications, users can apply effects to their own facial images, such as making them look older, younger, or wearing glasses.

[0003] In related technologies, although the edited face can produce a refined face effect, the area occupied by the face in the face image may differ significantly from that before editing, resulting in a poor face image quality that fails to meet the user's editing needs. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a face editing method that makes the face image after face editing more closely match the face position in the original face image, thereby improving the face editing effect.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] In a first aspect, embodiments of this application disclose a face editing method, the method comprising:

[0007] Acquire the face image to be edited, and determine the face editing style corresponding to the face image to be edited;

[0008] An initial face image is generated based on the face image to be edited and the face editing style, wherein the initial face image includes the target face corresponding to the face to be edited in the face image to be edited under the face editing style;

[0009] Determine the face contour boundary points corresponding to the initial face image, and the face contour boundary points are used to identify the face contour of the target face;

[0010] Based on the facial contour boundary points, the face to be edited is adjusted so that the difference between the facial contour of the adjusted face to be edited and the facial contour of the target face is less than a preset threshold.

[0011] Based on the adjusted face image to be edited and the initial face image, a target face image corresponding to the face image to be edited under the face editing style is generated.

[0012] Secondly, embodiments of this application disclose a face editing device, the device comprising an acquisition unit, a first generation unit, a first determination unit, an adjustment unit, and a second generation unit:

[0013] The acquisition unit is used to acquire the face image to be edited and to determine the face editing style corresponding to the face image to be edited;

[0014] The first generation unit is configured to generate an initial face image based on the face image to be edited and the face editing style, wherein the initial face image includes the target face corresponding to the face to be edited in the face image to be edited under the face editing style;

[0015] The first determining unit is used to determine the face contour boundary points corresponding to the initial face image, wherein the face contour boundary points are used to identify the face contour of the target face.

[0016] The adjustment unit is used to adjust the face to be edited according to the face contour boundary points, so that the difference between the face contour of the adjusted face to be edited and the face contour of the target face is less than a preset threshold.

[0017] The second generation unit is used to generate a target face image corresponding to the face image to be edited under the face editing style, based on the adjusted face image to be edited and the initial face image.

[0018] In one possible implementation, the device further includes a second determining unit:

[0019] The second determining unit is used to determine the face region distribution parameters corresponding to the adjusted face image to be edited. The face region distribution parameters are used to identify the distribution position of each facial component in the adjusted face image to be edited.

[0020] The second generation unit is specifically used for:

[0021] Based on the facial region distribution parameters, each facial component in the adjusted face image to be edited is replaced with the target facial component in the target face to generate the target face image. The target facial component and the facial component have the same semantic meaning on the face.

[0022] In one possible implementation, the first generating unit is specifically used for:

[0023] The style vector corresponding to the face image to be edited is determined, and the style editing parameters corresponding to the face editing style are determined. The style vector is used to identify the face to be edited, and the style editing parameters are used to adjust the face to be edited to the face editing style.

[0024] The initial face image is generated based on the style editing parameters and the style vector.

[0025] In one possible implementation, the style editing parameters include an editing degree parameter and a style editing vector, wherein the style editing vector is used to identify the vector editing direction corresponding to the face editing style, and the editing degree parameter is used to identify the degree to which the face to be edited is edited according to the face editing style;

[0026] The first generation unit is specifically used for:

[0027] Based on the style editing vector and the editing degree parameter, determine the style change vector;

[0028] Based on the style change vector and the style vector, determine the target style vector;

[0029] The initial face image is generated based on the target style vector.

[0030] In one possible implementation, the edit level parameter and the style edit vector are generated in the following way:

[0031] Obtain the undetermined style editing vector, attribute classifier, and multiple undetermined editing degree parameters corresponding to the face editing style. The attribute classifier is used to determine whether the face image generated based on the style vector changes in the target face attribute. The target face attribute is the face attribute corresponding to the face editing style.

[0032] In response to a change in the classification result of the attribute classifier caused by the undetermined style editing vector under a target undetermined editing degree parameter, the target undetermined editing degree parameter is determined as the editing degree parameter, and the undetermined style editing vector is determined as the style editing vector.

[0033] In one possible implementation, obtaining the undetermined style editing vector corresponding to the face editing style includes:

[0034] Obtain a target face image set, wherein multiple face images in the target face image set all possess the target face attributes;

[0035] Generate style vectors corresponding to the multiple face images respectively;

[0036] Determine the average style vector of the style vectors corresponding to the multiple face images respectively;

[0037] The undetermined style editing vector is generated based on the average style vector.

[0038] In one possible implementation, the style edit vector is generated in the following way:

[0039] Obtain the initial style editing vector and the interference style editing vector corresponding to the face editing style;

[0040] Determine the orthogonal vector component and parallel vector component of the interference style editing vector under the vector direction of the initial style editing vector, wherein the interference style editing vector is different from the face editing style corresponding to the initial style editing vector;

[0041] Remove the parallel vector component from the initial style editing vector to generate the style editing vector.

[0042] In one possible implementation, the first generating unit is specifically used for:

[0043] Based on the face image to be edited, an initial style vector is generated using a style vector generation model;

[0044] Generate an initial face image based on the initial style vector;

[0045] Based on the difference between the initial face image and the face image to be edited, the initial style vector is adjusted to obtain the style vector corresponding to the face image to be edited.

[0046] Thirdly, embodiments of this application disclose a computer device, the device including a processor and a memory:

[0047] The memory is used to store program code and transmit the program code to the processor;

[0048] The processor is configured to execute the face editing method described in any one of the first aspects according to the instructions in the program code.

[0049] Fourthly, embodiments of this application disclose a computer-readable storage medium for storing a computer program for executing the face editing method described in any one of the first aspects.

[0050] As can be seen from the above technical solution, when performing face editing, the image of the face to be edited can be acquired first, and the face editing style corresponding to the image of the face to be edited can be determined. Then, an initial face image is generated, which includes the target face corresponding to the face to be edited in the image of the face to be edited under the face editing style. In order to generate a face editing image with good effect, the face contour boundary points corresponding to the initial face image can be determined. These face contour boundary points are used to identify the face contour of the target face. Therefore, based on these face contour boundary points, the face to be edited can be adjusted so that the difference between the face contour of the adjusted face to be edited and the face contour of the target face is less than a preset threshold. Thus, when generating the edited target face image based on the adjusted face image to be edited and the initial face image, the transition between the face part and the background part in the target face image can be made more natural, improving the realism of face editing and improving the user's face editing experience. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A schematic diagram illustrating a face editing method in a practical application scenario provided by an embodiment of this application;

[0053] Figure 2 A flowchart illustrating a face editing method provided in this application embodiment;

[0054] Figure 3 A schematic diagram illustrating a face editing method provided in an embodiment of this application;

[0055] Figure 4 A schematic diagram illustrating a face editing method provided in an embodiment of this application;

[0056] Figure 5 A schematic diagram illustrating a face editing method provided in an embodiment of this application;

[0057] Figure 6 A schematic diagram illustrating a face editing method provided in an embodiment of this application;

[0058] Figure 7 A model architecture diagram provided for an embodiment of this application;

[0059] Figure 8 A comparison chart of effects provided for an embodiment of this application;

[0060] Figure 9 A structural block diagram of a face editing device provided in an embodiment of this application;

[0061] Figure 10 A structural diagram of a computer device provided in an embodiment of this application;

[0062] Figure 11 This is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0063] The embodiments of this application will now be described with reference to the accompanying drawings.

[0064] In related technologies, when editing faces, the focus is only on whether the desired style change effect is achieved, such as making the face look older or younger. However, the differences in facial shape between the edited face and the original face are not taken into account. This results in an excessive difference in facial shape between the edited face and the original face, leading to poor realism and failing to meet users' needs for face editing effects.

[0065] To address the aforementioned technical issues, this application provides a face editing method that adjusts the face image to be edited based on the face contour boundary points, making the face image after face editing more closely match the face position in the original face image. This allows for a more natural transition between the face image and the background image when combining the face image to be edited with the initial face image to generate the target face image, thus improving the face editing effect.

[0066] Understandably, this method can be applied to processing devices capable of face editing, such as terminal devices or servers with face editing capabilities. This method can be executed independently by the terminal device or server, or it can be applied in network scenarios where the terminal device and server communicate, executing in cooperation. The terminal device can be a computer, mobile phone, or similar device. The server can be an application server or a web server; in actual deployment, this server can be a standalone server or a cluster of servers.

[0067] To facilitate understanding of the technical solutions provided in the embodiments of this application, a face editing method provided in the embodiments of this application will be introduced next in conjunction with a practical application scenario.

[0068] See Figure 1 , Figure 1 This is a schematic diagram of a face editing method in a practical application scenario provided by an embodiment of this application. In this practical application scenario, the processing device can be a server 101 with face editing function.

[0069] After acquiring the image of the face to be edited and the corresponding face editing style, server 101 can perform preliminary face editing to obtain an initial face image. This initial face image includes the target face corresponding to the face to be edited in the image of the face to be edited under the specified face editing style. In this practical application scenario, the face editing style can be a preschool style, meaning the face can be edited to resemble a child's face. As shown in the figure, due to the generally small size of preschool faces, the edited target face is smaller under this face editing style, resulting in a large difference between the target face and the face to be edited. Since the target face needs to replace the face to be edited in the image of the face to be edited when generating the target face image, a large difference in face shape will lead to a large difference between the replaced face image and the background image, resulting in a poor sense of naturalness in the image.

[0070] To improve the face editing effect, server 101 can determine the face contour boundary points corresponding to the initial face image. These face contour boundary points are used to identify the face contour of the target face. Based on these face contour boundary points, server 101 can adjust the face image to be edited so that the face contour of the adjusted face to be edited is consistent with the face contour of the target face. Thus, when generating the target face image based on the adjusted face image to be edited and the initial face image, the face image portion in the target face image can transition more naturally with the background image portion, improving the realism of face editing.

[0071] Next, with reference to the accompanying drawings, a face editing method provided by an embodiment of this application will be described. See also Figure 2 , Figure 2 A flowchart of a face editing method provided in this application embodiment, the method including:

[0072] S201: Obtain the face image to be edited and determine the face editing style corresponding to the face image to be edited.

[0073] The image to be edited can be any image containing a face. The face editing style refers to the style that the face will be processed in this face editing, such as aging style, youthful style, black hair style, glasses style, etc. The face editing style can be the default setting or determined based on the user's selection.

[0074] S202: Generate an initial face image based on the face image to be edited and the face editing style.

[0075] Based on this face editing style, the processing device can learn how to perform face editing on the face to be edited in the face image to obtain the initial face image. The initial face image includes the target face corresponding to the face to be edited in the face image under the face editing style. That is, the target face is the face obtained after the face to be edited is changed under the face editing style.

[0076] S203: Determine the boundary points of the face contour corresponding to the initial face image.

[0077] Understandably, during the generation of the target face, the processing device only focuses on how to edit the face to be edited according to the desired face editing style, but it doesn't consider the differences in facial shape between the resulting target face and the original face to be edited. This can lead to excessively large differences in facial shape between the target face and the face to be edited. However, a face image includes not only the face but also some background image. Therefore, when the shape of the face differs significantly, it can cause anomalies in the background, resulting in an unnatural transition between the face and the background and a poor sense of realism in the face editing. To avoid these problems, the processing device can determine the facial contour boundary points corresponding to the initial face image. These boundary points are used to identify the facial contour of the target face.

[0078] S204: Adjust the face to be edited based on the face contour boundary points so that the difference between the face contour of the adjusted face to be edited and the face contour of the target face is less than a preset threshold.

[0079] The processing device can adjust the face to be edited based on the face outline marked by the face outline boundary points, so that the face outline of the adjusted face to be edited is close to or consistent with the face outline of the target face. The preset threshold can be adjusted based on the actual face editing effect requirements. The smaller the threshold, the closer the face outline of the target face is to the face outline of the face to be edited.

[0080] S205: Generate the target face image corresponding to the face image to be edited under the face editing style based on the adjusted face image to be edited and the initial face image.

[0081] Since the target face in the adjusted face image to be edited is relatively consistent with the face to be edited in terms of facial contour, the image to be edited can reflect how to achieve a natural transition between the target face and the background image under the contour of the target face. The initial face image includes the target face edited under the face editing style. Therefore, by combining the adjusted face image to be edited and the initial face image, the target face image corresponding to the face image to be edited under the face editing style can be generated.

[0082] As can be seen from the above technical solution, when performing face editing, the image of the face to be edited can be acquired first, and the face editing style corresponding to the image of the face to be edited can be determined. Then, an initial face image can be generated, which includes the target face corresponding to the face to be edited in the image of the face to be edited under the face editing style. In order to generate a face editing image with good effect, the face contour boundary points corresponding to the initial face image can be determined. These face contour boundary points are used to identify the face contour of the target face. Therefore, based on these face contour boundary points, the face to be edited can be adjusted so that the difference between the face contour of the adjusted face to be edited and the face contour of the target face is less than a preset threshold. Thus, when generating the edited target face image based on the adjusted face image to be edited and the initial face image, the transition between the face and the background in the target face image can be made more natural, improving the realism of face editing and enhancing the user's face editing experience.

[0083] As mentioned above, to further improve the face editing effect, the processing device can also perform additional processing on the face and background parts of the image. Understandably, during face editing, since a face image may include not only the face but also other background image components such as the environment, failure to differentiate between the face and background may result in some degree of distortion in the background image of the edited face image. Figure 3 As shown, although it is possible to effectively rejuvenate the face, the background image around the face in the edited face image is severely distorted.

[0084] Therefore, in one possible implementation, to avoid background image distortion caused by face editing, the processing device can determine the facial region distribution parameters corresponding to the adjusted face image to be edited. These parameters identify the distribution positions of each facial component in the adjusted face image, such as the positions of facial features, hair, face, glasses, and neck. Based on these parameters, the processing device can replace each facial component in the adjusted face image with a target facial component, generating the target face image. The target facial component and the facial component have the same semantic meaning on the face; that is, they correspond to the same parts of the face. For example, the processing device can replace the facial features in the adjusted face to be edited with the facial features in the adjusted target face, and replace the hair in the adjusted face to be edited with the hair in the adjusted target face, etc. This method allows for the replacement of only the face-related parts of the image, with minimal impact on the background, thus effectively protecting the background image and improving the face editing results.

[0085] For example, the processing device can perform facial semantic segmentation on the adjusted face image to be edited, obtaining a facial region mask. Facial semantic segmentation refers to using a neural network to segment various elements (including hair, face, facial features, glasses, neck, clothing, background, etc.) of a face image, with the same semantic meaning having the same pixel value. Then, the processing device can overlay the initial face image onto the face to be edited in the adjusted face image to obtain the target face image.

[0086] In one possible implementation, when determining the initial face image, the processing device can determine the style vector corresponding to the face image to be edited, and the style editing parameters corresponding to the face editing style. The style vector is used to identify the face to be edited, and the style editing parameters are used to adjust the face to be edited to the face editing style. Then, the processing device can generate the initial face image according to the style editing parameters and the style vector, that is, adjust the style vector according to the style editing parameters so that the adjusted style vector can identify the target face under the face editing style.

[0087] As shown in the formula below, Image real Let w be the face image to be edited, w be the style vector corresponding to the face image to be edited, and w′ be the style vector after editing.

[0088]

[0089] Specifically, in one possible implementation, the style editing parameters may include an editing degree parameter and a style editing vector. The style editing vector identifies the style editing direction corresponding to the face editing style; that is, when the style vector is adjusted according to the style editing direction, the face identified by the style vector will change under that face editing style. The editing degree parameter identifies the degree to which the face to be edited is edited according to the face editing style. For example, when the face editing style is an aging style, the editing degree parameter can identify the degree to which the face to be edited is aged.

[0090] The processing device can determine a style change vector based on the style editing vector and the editing degree parameter. This style change vector identifies the specific way the style vector is adjusted. Based on the style change vector and the style vector, the processing device can determine a target style vector. This target style vector identifies the target face obtained under the face editing style. Therefore, the processing device can generate the initial face image based on this target style vector.

[0091] Since this application generates edited face images based on vector editing, it can generate diverse style change vectors and editing degree parameters for various face editing styles, thereby improving the diversity of face editing.

[0092] As shown in the formula below, w is the style vector corresponding to the face image to be edited, ψ is the editing degree parameter, a is the style editing vector, and w′ is the target style vector.

[0093] w′=w+ψa

[0094] In one possible implementation, the edit level parameter and style edit vector are generated in the following way:

[0095] The processing device can acquire the pending style editing vector, attribute classifier, and multiple pending editing degree parameters corresponding to the face editing style. The attribute classifier is used to determine whether the face image generated based on the style vector changes in the target face attributes, such as whether the face looks older, younger, or whether the face wears glasses. The target face attribute is the face attribute corresponding to the face editing style. For example, when the face editing style is to look older or younger, the face attribute can be age; when the face editing style is to wear glasses, the face attribute can be whether the face wears glasses.

[0096] Understandably, when the pending style edit vector and the pending edit level parameter can effectively cause attribute changes in the face obtained by editing the style vector under that face editing style, it indicates that the pending style edit vector and the pending edit level parameter can be used as style edit parameters for editing the face under that face editing style. Therefore, the processing device can use this method to determine whether the pending style edit vector is effective and whether the pending edit level parameter is sufficient to cause changes in the target face attributes of the face image to be edited.

[0097] If the classification result of the attribute classifier changes under the target undetermined editing degree parameter, it indicates that the target undetermined editing degree parameter and the undetermined style editing vector can be used to edit the face image under the face editing style. Therefore, the processing device can determine the target undetermined editing degree parameter as the editing degree parameter and the undetermined style editing vector as the style editing vector.

[0098] For example, the processing device can train an attribute classifier for each face editing style based on real face images, and use this attribute classifier to verify the style editing vector and the editing degree parameter. As shown in the formula below, where a is the style editing vector to be verified, ψ is the editing degree parameter to be verified, and w... editLet G be the style vector used for validation, G be the generative model that generates face images based on the style vector, and C be the style vector. s For an attribute classifier of a certain face attribute s, s edit The classification result output by the attribute classifier is as follows: for example, the output result for wearing glasses is 1, and the output result for not wearing glasses is 0. For the facial attribute of glasses, if ψa can be used to classify w... edit If the corresponding classification results change, it means that they can be used as style editing vectors and editing degree parameters.

[0099] s edit =C s (G(w edit +ψa))

[0100] In one possible implementation, the undetermined style editing vector can be obtained as follows: The processing device can acquire a set of target face images, where multiple face images in the set all possess the target face attribute, such as multiple face images showing people wearing glasses. The processing device can generate style vectors corresponding to each of the multiple face images, and then determine the average style vector of these style vectors. Since multiple face images possess the target attribute, and style vectors can identify facial features in face images, the average style vector can reflect the characteristics of the target attribute to a certain extent. Therefore, the processing device can generate the undetermined style editing vector based on the average style vector. For example, the processing device can acquire a total average style vector corresponding to face images of multiple styles, which can reflect the characteristics of the basic face. Thus, by comparing the total average style vector with the average style vector, the characteristics of the style vector corresponding to the target face attribute can be reflected, thereby determining the corresponding style editing vector.

[0101] It's understandable that some face editing styles might be quite similar. For example, older people generally wear glasses, so the "aging" face editing style and the "wearing glasses" face editing style would have some similarity. In one possible implementation, to avoid the simultaneous alteration of multiple facial attributes when editing a face using a particular style due to similarity, the processing device can differentiate and process the style editing vectors for different face editing styles.

[0102] The processing device can acquire the initial style editing vector and the interfering style editing vector corresponding to the face editing style. The interfering style editing vector is a vector that will interfere with face editing of the initial style editing vector. In the vector dimension, two completely orthogonal vectors do not affect each other, that is, the interfering style editing vector is a style editing vector that is not orthogonal to the initial style editing vector.

[0103] The processing device can determine the orthogonal and parallel vector components of the interfering style editing vector under the vector direction of the initial style editing vector. This interfering style editing vector corresponds to a different face editing style than the initial style editing vector. Understandably, the orthogonal vector components will not affect face editing when using the initial style editing vector, while the parallel vector components, being parallel to the vector direction of the initial style editing vector, will affect face editing. Therefore, the processing device can remove the parallel vector components from the initial style editing vector to generate the style editing vector, thereby removing the influence of other face editing styles and obtaining a more accurate and targeted face editing result.

[0104] like Figure 4 As shown, since "aging" and "wearing glasses" have certain similarities in face editing styles, when editing a face to "age", the edited face will have some traces of glasses before removing interference, while the face after removing interference will have few traces of glasses.

[0105] Specifically, in one possible implementation, when determining the style vector corresponding to the face to be edited, the processing device can generate an initial style vector based on the face image to be edited using a style vector generation model. Then, an initial face image is generated based on this initial style vector. The difference between the initial face image and the face image to be edited reflects the accuracy of the initial style vector; that is, the smaller the difference between the two images, the more accurately the initial style vector represents the face image to be edited. Based on this, the processing device can adjust the initial style vector according to the difference between the initial face image and the face image to be edited, thus obtaining the style vector corresponding to the face image to be edited. In other words, this embodiment can first obtain a relatively accurate initial style vector through a style vector generation model, thereby shortening the adjustment time required when adjusting based on image differences, and obtaining a more accurate style vector.

[0106] like Figure 5 As shown, Figure 5 The diagram illustrates a face editing method that addresses two key challenges. Challenge A is that the background of the edited portrait can change significantly, making it impossible to directly replace the original face image with the target face image. Challenge B is that the overall face shape can be significantly altered after face editing.

[0107] See Figure 6 First, the processing device can perform face detection and registration on the input image or video frame containing the face, and then use the obtained registration points to crop and align to obtain the face image to be edited. crop Then, image I can be... crop The input style vector encoder obtains an initial style vector w0. The processing device can then process the style vector w0. i The input style vector portrait generator produces a restored face image, and the difference between the restored image and the original face image is calculated to guide the style vector w. i+1 Optimization. The processing device can repeat this optimization process approximately three times to ensure that the face image generated from the style vector is essentially identical to the original face image, thereby obtaining the style vector w corresponding to the original face image.

[0108] The processing device can adjust the style vector w based on the style edit vector obtained from sampling and classification to obtain the attribute-edited style vector w'. The processing device can then input w' into the style vector portrait generator to obtain the corresponding generated image I. edit The processing equipment performs 3DMM modeling and 3D registration on the original and edited images to obtain facial contour key points. Then, it uses the obtained facial contour key points to stretch the original face image into triangular patches, so that the face contour of the stretched image is consistent with the model generation result.

[0109] Then, the processing device can perform facial semantic segmentation on the stretched original image to obtain the facial region mask, and then superimpose the generated face image onto the facial region of the stretched original image according to the facial region mask to obtain the final target face image.

[0110] In generating face images based on style vectors, the processing device can use a Style Generative Adversarial Network (StyleGAN). Figure 7 This paper presents a StyleGAN model architecture. In StyleGAN, a non-linear mapping network (8 fully connected layers) is first used to map z to w space. Then, an Adaptive Instance Normalization (AdaIN) model is used to achieve layer-by-layer style transfer. Therefore, the w vector is also called the style vector (Style Code). The training objective of StyleGAN is only to generate high-quality, more realistic face images, so it can be trained in an unsupervised manner.

[0111] To further improve the efficiency of face image editing, the processing device can encode the number of convolutional channels in each layer of the StyleGAN generator (three variable scales were designed here: 1x / 0.75x / 0.5x), and pass this encoded information to the discriminator D for training. This results in a generative model usable at multiple scales. A comparison of model performance and effects at different scales can be found in [link to relevant documentation]. Figure 8 As shown, at the same output resolution of 1024, the execution speed (i.e., the delay) is nearly 3 times faster. Meanwhile, the fidelity index (FID) of the generated image only decreases slightly.

[0112] Based on the face editing method provided in the above embodiments, this application also provides a face editing device, see [link to relevant documentation]. Figure 9 , Figure 9 This application provides a structural block diagram of a face editing device 900, which includes an acquisition unit 901, a first generation unit 902, a first determination unit 903, an adjustment unit 904, and a second generation unit 905.

[0113] The acquisition unit 901 is used to acquire the face image to be edited and to determine the face editing style corresponding to the face image to be edited;

[0114] The first generation unit 902 is configured to generate an initial face image based on the face image to be edited and the face editing style, wherein the initial face image includes the target face corresponding to the face to be edited in the face image to be edited under the face editing style;

[0115] The first determining unit 903 is used to determine the face contour boundary points corresponding to the initial face image, and the face contour boundary points are used to identify the face contour of the target face.

[0116] The adjustment unit 904 is used to adjust the face to be edited according to the face contour boundary points, so that the difference between the face contour of the adjusted face to be edited and the face contour of the target face is less than a preset threshold.

[0117] The second generation unit 905 is used to generate a target face image corresponding to the face image to be edited under the face editing style based on the adjusted face image to be edited and the initial face image.

[0118] In one possible implementation, the device further includes a second determining unit:

[0119] The second determining unit is used to determine the face region distribution parameters corresponding to the adjusted face image to be edited. The face region distribution parameters are used to identify the distribution position of each facial component in the adjusted face image to be edited.

[0120] The second generation unit 905 is specifically used for:

[0121] Based on the facial region distribution parameters, each facial component in the adjusted face image to be edited is replaced with the target facial component in the target face to generate the target face image. The target facial component and the facial component have the same semantic meaning on the face.

[0122] In one possible implementation, the first generating unit 902 is specifically used for:

[0123] The style vector corresponding to the face image to be edited is determined, and the style editing parameters corresponding to the face editing style are determined. The style vector is used to identify the face to be edited, and the style editing parameters are used to adjust the face to be edited to the face editing style.

[0124] The initial face image is generated based on the style editing parameters and the style vector.

[0125] In one possible implementation, the style editing parameters include an editing degree parameter and a style editing vector, wherein the style editing vector is used to identify the vector editing direction corresponding to the face editing style, and the editing degree parameter is used to identify the degree to which the face to be edited is edited according to the face editing style;

[0126] The first generation unit 902 is specifically used for:

[0127] Based on the style editing vector and the editing degree parameter, determine the style change vector;

[0128] Based on the style change vector and the style vector, determine the target style vector;

[0129] The initial face image is generated based on the target style vector.

[0130] In one possible implementation, the edit level parameter and the style edit vector are generated in the following way:

[0131] Obtain the undetermined style editing vector, attribute classifier, and multiple undetermined editing degree parameters corresponding to the face editing style. The attribute classifier is used to determine whether the face image generated based on the style vector changes in the target face attribute. The target face attribute is the face attribute corresponding to the face editing style.

[0132] In response to a change in the classification result of the attribute classifier caused by the undetermined style editing vector under a target undetermined editing degree parameter, the target undetermined editing degree parameter is determined as the editing degree parameter, and the undetermined style editing vector is determined as the style editing vector.

[0133] In one possible implementation, obtaining the undetermined style editing vector corresponding to the face editing style includes:

[0134] Obtain a target face image set, wherein multiple face images in the target face image set all possess the target face attributes;

[0135] Generate style vectors corresponding to the multiple face images respectively;

[0136] Determine the average style vector of the style vectors corresponding to the multiple face images respectively;

[0137] The undetermined style editing vector is generated based on the average style vector.

[0138] In one possible implementation, the style edit vector is generated in the following way:

[0139] Obtain the initial style editing vector and the interference style editing vector corresponding to the face editing style;

[0140] Determine the orthogonal vector component and parallel vector component corresponding to the vector direction of the initial style editing vector, wherein the interference style editing vector is different from the face editing style corresponding to the initial style editing vector;

[0141] Remove the parallel vector component from the initial style editing vector to generate the style editing vector.

[0142] In one possible implementation, the first generating unit 902 is specifically used for:

[0143] Based on the face image to be edited, an initial style vector is generated using a style vector generation model;

[0144] An initial face image is generated based on the initial style vector;

[0145] Based on the difference between the initial face image and the face image to be edited, the initial style vector is adjusted to obtain the style vector corresponding to the face image to be edited.

[0146] This application also provides a computer device, which will be described below with reference to the accompanying drawings. Please refer to... Figure 10 As shown in the figure, this application provides a device, which can also be a terminal device. The terminal device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0147] Figure 10 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 10 The mobile phone includes components such as a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless Fidelity (WiFi) module 770, a processor 780, and a power supply 790. Those skilled in the art will understand that... Figure 10 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0148] The following is combined Figure 10 A detailed introduction to each component of a mobile phone:

[0149] RF circuit 710 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 780; additionally, it transmits uplink data to the base station. Typically, RF circuit 710 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, RF circuit 710 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0150] The memory 720 can be used to store software programs and modules. The processor 780 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 720. The memory 720 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0151] The input unit 730 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732. The touch panel 731, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 731), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 731 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 780, and can also receive and execute commands sent by the processor 780. In addition, the touch panel 731 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 731, the input unit 730 may also include other input devices 732. Specifically, other input devices 732 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0152] The display unit 740 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 740 may include a display panel 741, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel. Further, a touch panel 731 may cover the display panel 741. When the touch panel 731 detects a touch operation on or near it, it transmits the information to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides corresponding visual output on the display panel 741 based on the type of touch event. Although in Figure 10 In this embodiment, the touch panel 731 and the display panel 741 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 731 and the display panel 741 can be integrated to realize the input and output functions of the mobile phone.

[0153] The mobile phone may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 741 according to the ambient light level, and the proximity sensor can turn off the display panel 741 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0154] Audio circuit 760, speaker 761, and microphone 762 provide an audio interface between the user and the mobile phone. Audio circuit 760 converts received audio data into electrical signals and transmits them to speaker 761, where speaker 761 converts them into sound signals for output. On the other hand, microphone 762 converts collected sound signals into electrical signals, which are received by audio circuit 760, converted into audio data, and then processed by processor 780 before being transmitted via RF circuit 710 to, for example, another mobile phone, or the audio data can be output to memory 720 for further processing.

[0155] WiFi is a short-range wireless transmission technology. Through the WiFi module 770, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 10 The WiFi module 770 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0156] The processor 780 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 720, and calls data stored in the memory 720 to perform various functions and process data, thereby performing overall detection of the phone. Optionally, the processor 780 may include one or more processing units; preferably, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 780.

[0157] The mobile phone also includes a power supply 790 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 780 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0158] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0159] In this embodiment, the processor 780 included in the terminal device also has the following functions:

[0160] Acquire the face image to be edited, and determine the face editing style corresponding to the face image to be edited;

[0161] An initial face image is generated based on the face image to be edited and the face editing style, wherein the initial face image includes the target face corresponding to the face to be edited in the face image to be edited under the face editing style;

[0162] Determine the face contour boundary points corresponding to the initial face image, and the face contour boundary points are used to identify the face contour of the target face;

[0163] Based on the facial contour boundary points, the face to be edited is adjusted so that the difference between the facial contour of the adjusted face to be edited and the facial contour of the target face is less than a preset threshold.

[0164] Based on the adjusted face image to be edited and the initial face image, a target face image corresponding to the face image to be edited under the face editing style is generated.

[0165] This application also provides a server; please refer to [link to relevant documentation]. Figure 11 As shown, Figure 11 This is a structural diagram of a server 800 provided in an embodiment of this application. The server 800 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 822 (e.g., one or more processors) and a memory 832, and one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 842 or data 844. The memory 832 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 822 may be configured to communicate with the storage media 830 and execute the series of instruction operations in the storage media 830 on the server 800.

[0166] Server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0167] The steps performed by the server in the above embodiments can be based on Figure 11 The server structure shown.

[0168] This application also provides a computer-readable storage medium for storing a computer program that executes any one of the face editing methods described in the foregoing embodiments.

[0169] This application also provides a computer program product including instructions that, when run on a computer, cause the computer to execute the face editing method provided in any of the above embodiments.

[0170] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0171] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0172] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A face editing method, characterized in that, The method includes: Acquire the face image to be edited, and determine the face editing style corresponding to the face image to be edited; An initial face image is generated based on the face image to be edited and the face editing style. The initial face image includes the target face corresponding to the face to be edited in the face image to be edited under the face editing style. The initial face image is generated based on style editing parameters and a style vector. The style vector is obtained by adjusting the initial style vector based on the difference between the initial face image and the face image to be edited. The style editing parameters include an editing degree parameter and a style editing vector. The style editing vector is generated by: obtaining the initial style editing vector and the interference style editing vector corresponding to the face editing style; determining the orthogonal vector component and the parallel vector component corresponding to the interference style editing vector under the vector direction of the initial style editing vector, wherein the interference style editing vector is different from the face editing style corresponding to the initial style editing vector; removing the parallel vector component from the initial style editing vector to generate the style editing vector. Determine the face contour boundary points corresponding to the initial face image, and the face contour boundary points are used to identify the face contour of the target face; Based on the facial contour boundary points, key facial contour points are obtained through 3DMM modeling and 3D registration. The obtained key facial contour points are used to stretch the face to be edited into triangular patches so that the difference between the adjusted facial contour of the face to be edited and the facial contour of the target face is less than a preset threshold. Based on the adjusted face image to be edited and the initial face image, a target face image corresponding to the face image to be edited under the face editing style is generated.

2. The method according to claim 1, characterized in that, The method further includes: Determine the facial region distribution parameters corresponding to the adjusted face image to be edited. The facial region distribution parameters are used to identify the distribution position of each facial component in the adjusted face image to be edited. The step of generating the target face image corresponding to the face image to be edited under the face editing style based on the adjusted face image to be edited and the initial face image includes: Based on the facial region distribution parameters, each facial component in the adjusted face image to be edited is replaced with the target facial component in the target face to generate the target face image. The target facial component and the facial component have the same semantic meaning on the face.

3. The method according to claim 1, characterized in that, The step of generating an initial face image based on the face image to be edited and the face editing style includes: The style vector corresponding to the face image to be edited is determined, and the style editing parameters corresponding to the face editing style are determined. The style vector is used to identify the face to be edited, and the style editing parameters are used to adjust the face to be edited to the face editing style. The initial face image is generated based on the style editing parameters and the style vector.

4. The method according to claim 3, characterized in that, The style editing vector is used to identify the vector editing direction corresponding to the face editing style, and the editing degree parameter is used to identify the degree to which the face to be edited is edited according to the face editing style; The step of generating the initial face image based on the style editing parameters and the style vector includes: Based on the style editing vector and the editing degree parameter, determine the style change vector; Based on the style change vector and the style vector, determine the target style vector; The initial face image is generated based on the target style vector.

5. The method according to claim 4, characterized in that, The edit level parameter and the style edit vector are generated in the following way: Obtain the undetermined style editing vector, attribute classifier, and multiple undetermined editing degree parameters corresponding to the face editing style. The attribute classifier is used to determine whether the face image generated based on the style vector changes in the target face attribute. The target face attribute is the face attribute corresponding to the face editing style. In response to a change in the classification result of the attribute classifier caused by the undetermined style editing vector under a target undetermined editing degree parameter, the target undetermined editing degree parameter is determined as the editing degree parameter, and the undetermined style editing vector is determined as the style editing vector.

6. The method according to claim 5, characterized in that, The step of obtaining the undetermined style editing vector corresponding to the face editing style includes: Obtain a target face image set, wherein multiple face images in the target face image set all possess the target face attributes; Generate style vectors corresponding to the multiple face images respectively; Determine the average style vector of the style vectors corresponding to the multiple face images respectively; The undetermined style editing vector is generated based on the average style vector.

7. The method according to claim 3, characterized in that, Determining the style vector corresponding to the face image to be edited includes: Based on the face image to be edited, an initial style vector is generated using a style vector generation model; Generate an initial face image based on the initial style vector; Based on the difference between the initial face image and the face image to be edited, the initial style vector is adjusted to obtain the style vector corresponding to the face image to be edited.

8. A face editing device, characterized in that, The device includes an acquisition unit, a first generation unit, a first determination unit, an adjustment unit, and a second generation unit. The acquisition unit is used to acquire the face image to be edited and to determine the face editing style corresponding to the face image to be edited; The first generation unit is configured to generate an initial face image based on the face image to be edited and the face editing style. The initial face image includes the target face corresponding to the face to be edited in the face image to be edited under the face editing style. The initial face image is generated based on style editing parameters and a style vector. The style vector is obtained by adjusting the initial style vector based on the difference between the initial face image and the face image to be edited. The style editing parameters include an editing degree parameter and a style editing vector. The style editing vector is generated by: obtaining the initial style editing vector and the interference style editing vector corresponding to the face editing style; determining the orthogonal vector component and the parallel vector component corresponding to the interference style editing vector under the vector direction of the initial style editing vector, wherein the interference style editing vector is different from the face editing style corresponding to the initial style editing vector. Remove the parallel vector component from the initial style editing vector to generate the style editing vector; The first determining unit is used to determine the face contour boundary points corresponding to the initial face image, wherein the face contour boundary points are used to identify the face contour of the target face. The adjustment unit is used to obtain key points of the facial contour through 3DMM modeling and 3D registration based on the boundary points of the facial contour, and to stretch the face to be edited by triangle using the obtained key points of the facial contour so that the difference between the facial contour of the face to be edited and the facial contour of the target face is less than a preset threshold. The second generation unit is used to generate a target face image corresponding to the face image to be edited under the face editing style, based on the adjusted face image to be edited and the initial face image.

9. A computer device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the face editing method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the face editing method according to any one of claims 1-7.

11. A computer program product comprising instructions that, when run on a computer, causes the computer to perform the face editing method according to any one of claims 1-7.

Images