Image generation method and device, electronic equipment and storage medium

Abstract

This application proposes an image generation method, apparatus, electronic device, and storage medium. The method includes: acquiring key points of a region to be edited in a face image; acquiring a directional semantic editing space of the region to be edited based on the key points; and editing the region to be edited based on the directional semantic editing space to generate a target face image corresponding to the face image. In this application, the directional semantic editing space allows for directional processing of different regions of the face image, resulting in a high degree of decoupling. This enables controllable and interpretable semantic editing of the face image, thereby achieving different degrees of semantic editing and optimizing the semantic editing effect of the face image.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of generative adversarial networks, the demand for the quality of samples generated by generators is increasing. In related technologies, the generator edits the data to generate sample images, resulting in poor generation and editing effects and low decoupling. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, the first aspect of this application proposes a method for generating images.

[0005] The second aspect of this application also proposes an image generation apparatus.

[0006] The third aspect of this application proposes an electronic device.

[0007] The fourth aspect of this application proposes a computer-readable storage medium.

[0008] The fifth aspect of this application proposes a computer program product.

[0009] The first aspect of this application proposes an image generation method, comprising: obtaining key points of a region to be edited in a face image; obtaining a directional semantic editing space of the region to be edited based on the key points of the region to be edited; and editing the region to be edited based on the directional semantic editing space to generate a target face image corresponding to the face image.

[0010] In addition, the image generation method proposed in the first aspect of this application may also have the following additional technical features:

[0011] According to one embodiment of this application, obtaining the directional semantic editing space of the region to be edited based on key points of the region to be edited includes: obtaining the first Jacobian matrix of the region to be edited based on the key points of the region to be edited; and generating the directional semantic editing space based on the first Jacobian matrix.

[0012] According to one embodiment of this application, obtaining the directional semantic editing space of the region to be edited based on key points of the region to be edited includes: obtaining a first Jacobian matrix and a second Jacobian matrix of the region to be edited based on key points of the region to be edited; and generating the directional semantic editing space based on the first Jacobian matrix and the second Jacobian matrix.

[0013] According to one embodiment of this application, a directional semantic editing space is generated based on a first Jacobian matrix and a second Jacobian matrix, including: obtaining a semantic editing space for key points based on the first Jacobian matrix; obtaining a semantic suppression space for key points based on the second Jacobian matrix; and obtaining the intersection of the semantic editing space and the semantic suppression space as the directional semantic editing space.

[0014] According to one embodiment of this application, obtaining the semantic editing space of key points based on the first Jacobian matrix includes: obtaining the first transpose Jacobian matrix of the first Jacobian matrix, and obtaining the first matrix based on the first Jacobian matrix and the first transpose Jacobian matrix; solving for eigenvalues ​​and eigenvectors of the first matrix, and obtaining the first eigenvectors corresponding to the first eigenvalues ​​in the first matrix that satisfy the first set conditions, so as to constitute the semantic editing space of key points.

[0015] According to one embodiment of this application, obtaining the feature vector corresponding to the first feature value that satisfies the first set condition in the first matrix to form the semantic editing space of the key point includes: obtaining the first feature vector corresponding to the first feature value that is greater than or equal to the set threshold to form the semantic editing space of the key point; or, sorting the first feature vectors from high to low according to the first feature value, and sequentially obtaining a first set number of first feature vectors to form the semantic editing space of the key point.

[0016] According to one embodiment of this application, obtaining the semantic suppression space of key points based on the second Jacobian matrix includes: obtaining the second transpose Jacobian matrix of the second Jacobian matrix, and obtaining the second matrix based on the second Jacobian matrix and the second transpose Jacobian matrix; solving for eigenvalues ​​and eigenvectors of the second matrix, and obtaining the second eigenvectors corresponding to the second eigenvalues ​​of the second matrix under the second set conditions, so as to constitute the semantic suppression space of key points.

[0017] According to one embodiment of this application, obtaining the feature vector corresponding to the second feature value of the second set condition in the second matrix to form the semantic suppression space of the key point includes: obtaining the second feature vector corresponding to the second feature value that is equal to zero to form the semantic suppression space of the key point; or, sorting the second feature vectors from low to high according to the second feature value, and sequentially obtaining a second set number of second feature vectors to form the semantic suppression space of the key point.

[0018] According to one embodiment of this application, editing a region to be edited based on a directional semantic editing space to generate a target face image corresponding to a face image includes: obtaining offset vectors of key points based on the directional semantic editing space and offset parameters; and editing the key points of the region to be edited based on the offset vectors to generate the target face image.

[0019] According to one embodiment of this application, after generating a target face image corresponding to a face image, the method further includes: inputting the target face image into a face attribute classifier for classification and recognition to obtain the classification labels of multiple attributes of the target face image and the classification probabilities under the classification labels; and generating attribute classification explanation information of the target face image based on the classification labels of multiple attributes and the classification probabilities under the classification labels.

[0020] According to one embodiment of this application, before obtaining the key points of the region to be edited in the face image, the method further includes: obtaining a normally distributed noise vector, performing an affine transformation on the noise vector to generate a style vector in the style space; and inputting the style vector into an image generator to obtain the face image.

[0021] According to one embodiment of this application, the method further includes: taking the inverse gradient derivative of the feature vector of the key point for either the first Jacobian matrix or the second Jacobian matrix to generate any Jacobian matrix.

[0022] A second aspect of this application also proposes an image generation apparatus, comprising: a detection module for acquiring key points of a region to be edited in a face image; an acquisition module for acquiring a directional semantic editing space of the region to be edited based on the key points of the region to be edited; and a generation module for editing the region to be edited based on the directional semantic editing space to generate a target face image corresponding to the face image.

[0023] The image generation apparatus proposed in the second aspect of this application may also have the following additional technical features:

[0024] According to one embodiment of this application, the acquisition module is further configured to: acquire a first Jacobian matrix of the region to be edited based on key points of the region to be edited; and generate a directional semantic editing space based on the first Jacobian matrix.

[0025] According to one embodiment of this application, the acquisition module is further configured to: acquire a first Jacobian matrix and a second Jacobian matrix of the region to be edited based on key points of the region to be edited; and generate a directional semantic editing space based on the first Jacobian matrix and the second Jacobian matrix.

[0026] According to one embodiment of this application, the acquisition module is further configured to: acquire the semantic editing space of key points based on a first Jacobian matrix; acquire the semantic suppression space of key points based on a second Jacobian matrix; and acquire the intersection of the semantic editing space and the semantic suppression space as the directional semantic editing space.

[0027] According to one embodiment of this application, the acquisition module is further configured to: acquire the first transpose Jacobian matrix of the first Jacobian matrix, and acquire the first matrix based on the first Jacobian matrix and the first transpose Jacobian matrix; solve for the eigenvalues ​​and eigenvectors of the first matrix, and acquire the first eigenvectors corresponding to the first eigenvalues ​​in the first matrix that satisfy the first set conditions, so as to constitute the semantic editing space of the key points.

[0028] According to one embodiment of this application, the acquisition module is further configured to: acquire a first feature vector corresponding to a first feature value that is greater than or equal to a set threshold, so as to constitute a semantic editing space for key points; or, sort the first feature vectors from high to low according to the first feature value, and sequentially acquire a first set number of first feature vectors to constitute a semantic editing space for key points.

[0029] According to one embodiment of this application, the acquisition module is further configured to: acquire the second transpose Jacobian matrix of the second Jacobian matrix, and acquire the second matrix based on the second Jacobian matrix and the second transpose Jacobian matrix; solve for the eigenvalues ​​and eigenvectors of the second matrix, and acquire the second eigenvectors corresponding to the second eigenvalues ​​of the second matrix under the second set conditions, so as to constitute the semantic suppression space of the key points.

[0030] According to one embodiment of this application, the acquisition module is further configured to: acquire the second feature vector corresponding to the second feature value equal to zero, thereby constituting the semantic suppression space of the key point; or, sort the second feature vectors from low to high according to the second feature value, and sequentially acquire a second set number of second feature vectors, thereby constituting the semantic suppression space of the key point.

[0031] According to one embodiment of this application, the generation module is further configured to: obtain the offset vector of the key points based on the directional semantic editing space and the offset parameter; and edit the key points of the region to be edited based on the offset vector to generate a target face image.

[0032] According to one embodiment of this application, the apparatus further includes: a classification module, configured to input a target face image into a face attribute classifier for classification and recognition, to obtain classification labels of multiple attributes of the target face image and classification probabilities under the classification labels; and to generate attribute classification explanation information of the target face image based on the classification labels of multiple attributes and classification probabilities under the classification labels.

[0033] According to one embodiment of this application, the detection module is further configured to: obtain a normally distributed noise vector, perform an affine transformation on the noise vector to generate a style vector in the style space; and input the style vector into an image generator to obtain a face image.

[0034] According to one embodiment of this application, the acquisition module is further configured to: perform back gradient differentiation on the feature vector of the key point for either the first Jacobian matrix or the second Jacobian matrix to generate any Jacobian matrix.

[0035] A third aspect of this application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the image generation method described in the first aspect above.

[0036] A fourth aspect of this application provides a computer-readable storage medium, wherein computer instructions are provided to cause a computer to perform the image generation method proposed in the first aspect.

[0037] The fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the image generation method according to the first aspect above.

[0038] The image generation method and apparatus proposed in this application obtain a directional semantic editing space corresponding to the region to be edited based on key points within the region to be edited in a face image, and then edit the region to be edited in the face image according to the directional semantic editing space, thereby generating the corresponding target face image after editing. In this application, different regions of the face image can be processed directionally through the directional semantic editing space, resulting in a high degree of decoupling. This enables controllable and interpretable semantic editing of the face image, optimizing the semantic editing effect of the face image.

[0039] It should be understood that the description herein is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0040] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0041] Figure 1 This is a schematic flowchart illustrating an image generation method according to an embodiment of this application;

[0042] Figure 2 This is a schematic flowchart illustrating an image generation method according to another embodiment of this application;

[0043] Figure 3 This is a schematic flowchart illustrating an image generation method according to another embodiment of this application;

[0044] Figure 4 This is a schematic flowchart illustrating an image generation method according to another embodiment of this application;

[0045] Figure 5 This is a schematic flowchart illustrating an image generation method according to another embodiment of this application;

[0046] Figure 6 This is a schematic flowchart illustrating an image generation method according to another embodiment of this application;

[0047] Figure 7 This is a schematic flowchart illustrating an image generation method according to another embodiment of this application;

[0048] Figure 8 This is a schematic flowchart illustrating an image generation method according to another embodiment of this application;

[0049] Figure 9 This is a schematic diagram of the structure of an image generation apparatus according to an embodiment of this application;

[0050] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0051] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0052] The following description, with reference to the accompanying drawings, outlines a method, apparatus, electronic device, and storage medium for generating images according to embodiments of this application.

[0053] Figure 1 This is a schematic flowchart illustrating an image generation method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0054] S101, Obtain the key points of the area to be edited in the face image.

[0055] In adversarial training of adversarial networks, different samples need to be generated based on the generator, and the discriminator needs to judge the samples to achieve adversarial training between the generator and the discriminator. In this way, a generator that can generate samples close to real samples and a discriminator that can accurately identify samples can be obtained.

[0056] Among them, adversarial networks targeting facial images can generate facial image samples by editing facial features to different degrees.

[0057] In this embodiment of the application, the face image has multiple features, and the attribute recognition of the face can be achieved based on different features. For the same feature, different feature manifestations have a certain degree of influence on the attribute recognition result of the face.

[0058] To determine the impact of different regions of a face image on attribute recognition results, features of a specific region can be processed. Based on the processed face features and the corresponding recognition results, the correlation between the features of different regions of the face image and the attribute recognition results can be obtained.

[0059] In this embodiment of the application, the area that needs to be processed can be used as the area to be edited in the face image. For example, if feature processing is needed on the facial contour of the face image, the facial contour part of the face image can be used as the area to be edited.

[0060] Furthermore, key points of the region to be edited are obtained from all key point information of the face image using relevant setting tools. Optionally, the face image can be input into a face key point detector, and the key points of the face image can be obtained based on the detector's output, thereby obtaining the key point information of the region to be edited in the face image.

[0061] S102, based on the key points of the region to be edited, obtain the directional semantic editing space of the region to be edited.

[0062] To achieve controllable and interpretable attribute recognition of facial images, semantic editing of the editable region of the facial image can be performed based on a directional semantic editing space.

[0063] In some implementations, after obtaining the edit space corresponding to the face image, the edit space can be segmented according to different regions of the face image, thereby generating directional semantic edit spaces corresponding to different regions of the face image.

[0064] In other implementations, the semantic editing space corresponding to different regions of a face image can be obtained, and the semantic editing space corresponding to each region can be used as the directional semantic editing space corresponding to that region.

[0065] Furthermore, a directional semantic editing space can be obtained based on the key points of the region to be edited. For example, a directional semantic editing space can be obtained for key points in the eye region, and semantic editing can only be performed on the eye region of the face image. Similarly, a directional semantic editing space can be obtained for key points in the facial contour region, and semantic editing can only be performed on the facial contour region of the face image.

[0066] Optionally, the key points of the area to be edited in the face image can be solved based on the relevant set algorithm, and the directional semantic editing space corresponding to the area to be edited can be obtained according to the solution result.

[0067] It should be noted that strong decoupled editing of face images can be achieved based on the directional semantic editing space, so that only the region to be edited is semantically edited during the semantic editing process of face images, while ensuring that other regions are not edited.

[0068] S103, based on the directional semantic editing space, edit the region to be edited to generate the target face image corresponding to the face image.

[0069] In this embodiment of the application, key points of the area to be edited can be input into the corresponding directional semantic editing space, and semantic editing can be performed in the directional semantic editing space.

[0070] Optionally, semantic editing can be performed based on the attribute dimensions of key points in the area to be edited, and the semantic editing amount can be obtained for different attribute dimensions, thereby obtaining the edited key point information.

[0071] For example, to enable semantic editing of the eye region in a facial image, key points of the eye region can be input into the corresponding directional semantic editing space.

[0072] The saturation of key points in the eye area is set for semantic editing. The amount of editing corresponding to each key point in the saturation dimension is obtained. Based on the current saturation and the corresponding amount of editing, the saturation of key points in the eye area after targeted editing is obtained.

[0073] Furthermore, based on the output of the directional semantic editing space, the key point information of the region to be edited after directional editing is obtained.

[0074] In some implementations, the key points of the edited area can be integrated with the key points of other unedited areas of the face image to obtain the key points of the complete face image after directional editing, thereby generating the target face image after face image editing.

[0075] In other implementations, an edited image corresponding to the region to be edited can be generated based on the key points after editing the region to be edited. This edited image can then be stitched together with face images from other regions to generate the target face image after face image editing.

[0076] The image generation method proposed in this application obtains the directional semantic editing space corresponding to the region to be edited based on key points within the region to be edited in the face image, and then edits the region to be edited in the face image according to the directional semantic editing space, thereby generating the corresponding target face image after editing. In this application, different regions of the face image can be processed directionally through the directional semantic editing space, with a high degree of decoupling, enabling controllable and interpretable semantic editing of the face image, thereby achieving different degrees of semantic editing of the face image and optimizing the semantic editing effect of the face image.

[0077] As one possible implementation method, it can be combined with Figure 2 To further understand the acquisition of the directional semantic editing space, Figure 2 This is a flowchart illustrating an image generation method according to another embodiment of this application, as shown below. Figure 2 As shown, the method includes:

[0078] S201, based on the key points of the region to be edited, obtain the first Jacobian matrix of the region to be edited.

[0079] In implementation, a normally distributed noise vector can be obtained, and an affine transformation can be performed on the noise vector to generate a style vector in the style space.

[0080] In this embodiment, a corresponding face image can be generated based on a normally distributed noise vector, wherein the normally distributed noise vector is input to the space (Z∈R). 512 Then, a mapping network is used to map the normally distributed noise vector to the latent space, where the latent space is a 512-dimensional space (W∈R). 512 This process involves obtaining the corresponding latent variable W, and then performing an affine transformation on the latent variable W based on the affine transformation module of each layer of the generator, mapping it to the generator's style space. The style space is a 9088-dimensional space (S∈R). 9088 ), and then obtain the corresponding style vector S.

[0081] Furthermore, the style vector is input into the image generator to obtain the face image.

[0082] In this embodiment, a style vector can be input into an image generator, and the image generator performs imaging processing on the style vector to obtain a face image. Specifically, if the style vector S is set to a 9088-dimensional vector, then the resolution of the generated face image I is 1024×1024 (I∈R). 1024×1024×3 ).

[0083] Alternatively, the mapping from style vector S to face image I can be labeled as I = G(s).

[0084] Furthermore, all key points corresponding to the face image I can be obtained, and the key points of the region to be edited can be determined from them. In this embodiment, the directional semantic editing space of the region to be edited can be constructed based on a set number of feature vectors. Therefore, feature vectors that can construct the directional semantic editing space can be obtained based on a set algorithm, thereby realizing the acquisition of the directional semantic editing space.

[0085] Optionally, the back gradient derivative can be calculated based on the key points of the region to be edited to obtain the corresponding Jacobian matrix, which is then determined as the first Jacobian matrix of the region to be edited, as shown in the following formula:

[0086]

[0087] Among them, J v Let H0(s) be the first Jacobian matrix of the H0 function, and H0(s) be the key points of the region to be edited.

[0088] Furthermore, the feature vectors corresponding to the directional semantic editing space of the region to be edited are obtained through the first Jacobian matrix.

[0089] S202 generates a directional semantic editing space based on the first Jacobian matrix.

[0090] In this embodiment of the application, the first Jacobian matrix can be further calculated, and based on the calculation result, the feature vector of the directional semantic editing space corresponding to the region to be edited can be obtained, thereby realizing the construction of the directional semantic editing space corresponding to the region to be edited.

[0091] Among them, a semantic editing space can be constructed based on the key points of the region to be edited, which can be used to perform semantic editing on the key points, and this space can be used as the directional semantic editing space of the region to be edited.

[0092] Optionally, the semantic editing space of key points can be obtained based on the first Jacobian matrix.

[0093] Specifically, the first transpose Jacobian matrix of the first Jacobian matrix can be obtained, and the first matrix can be obtained based on the first Jacobian matrix and the first transpose Jacobian matrix.

[0094] In this embodiment of the application, the first Jacobian matrix can be set to J. v Its transpose matrix is ​​determined as the first transpose Jacobian matrix J. v If T, then the product of the first Jacobian matrix and the first transpose Jacobian matrix can be determined as the first matrix J. v J vT .

[0095] Furthermore, the first matrix is ​​solved for eigenvalues ​​and eigenvectors to obtain the first eigenvectors corresponding to the first eigenvalues ​​that satisfy the first set conditions in the first matrix, so as to form the semantic editing space of the key points.

[0096] In this embodiment of the application, the first matrix can be solved, and feature vectors that can be used to construct the directional semantic editing space corresponding to the region to be edited can be obtained from multiple solutions of the first matrix.

[0097] Optionally, the first matrix can be solved based on a set algorithm. For example, all eigenvalues ​​of the first matrix can be obtained by calculating the characteristic polynomial corresponding to the first matrix, and then the eigenvector corresponding to each eigenvalue can be solved. Based on the eigenvalue and its corresponding eigenvector, the solution of the first matrix can be generated.

[0098] To ensure that the directional semantic editing space can achieve the effect of semantic editing on the region to be edited in the face image, a first feature vector that satisfies the set criteria can be obtained from all solutions of the first matrix, where the set criteria can be used as the first set condition.

[0099] In implementation, each eigenvector in all solutions of the first matrix can perform semantic editing of key points in the region to be edited. The magnitude of the first eigenvalue is positively correlated with the editing ability of the first eigenvector for key points. It can be understood that the first eigenvector with a higher first eigenvalue has a higher semantic editing ability for key points than the first eigenvector with a lower first eigenvalue.

[0100] To ensure the effectiveness of semantic editing of key points in the region to be edited in the directional semantic editing space, the corresponding first set conditions can be obtained based on the first feature value, thereby enabling the filtering of feature vectors.

[0101] In some implementations, feature vectors corresponding to the first feature value that is greater than or equal to a set threshold can be obtained to form the semantic editing space of the key points.

[0102] In this embodiment of the application, a set threshold corresponding to the first feature value can be obtained, and a value greater than or equal to the set threshold can be determined as the first set condition.

[0103] When the first feature value is greater than or equal to the set threshold, it can be determined that the first feature value meets the first set condition, and the corresponding first feature vector can effectively perform semantic editing on the key points of the area to be edited.

[0104] Furthermore, based on the first feature vector that satisfies the first set condition, the semantic editing space of the key points of the region to be edited is constructed, and this semantic editing space is used as the directional semantic editing space of the key points of the region to be edited.

[0105] In other implementations, the first feature vectors can be sorted from high to low according to the first feature value, and a first set number of first feature vectors can be obtained sequentially to form the semantic editing space of the key points.

[0106] In this embodiment of the application, the first feature value is different. In order to ensure the semantic editing effect of the region to be edited in the directional semantic editing space, the first feature vector can be sorted based on the first feature value.

[0107] Among all solutions to the first matrix, the largest first eigenvalue can be obtained based on the following formula:

[0108]

[0109] In the above formula, H0(s) represents the key points of the region to be edited, and J... v Let α be the first Jacobian matrix of the function H0, and α be a fixed step size.

[0110] In implementation, there is a set number of feature vectors for constructing the directional semantic editing space, which can be used as the first set number.

[0111] Furthermore, starting from the first feature vector with the highest first feature value, a first set number of first feature vectors are collected from the sorting results of the first feature vectors. Based on this part of the first feature vectors, a semantic editing space for key points is constructed, and this semantic editing space is used as the directional semantic editing space for key points in the region to be edited.

[0112] The image generation method proposed in this application obtains the first Jacobian matrix corresponding to the region to be edited based on the key points of the region to be edited, and generates a directional semantic editing space for the region to be edited based on the first Jacobian matrix. In this application, the directional semantic editing space enables regionalized directional processing of face images, allowing for controllable and interpretable semantic editing of face images.

[0113] As another possible implementation, it can be combined with Figure 3 To further understand the acquisition of the directional semantic editing space, Figure 3 This is a flowchart illustrating an image generation method according to another embodiment of this application, as shown below. Figure 3 As shown, the method includes:

[0114] S301, based on the key points of the region to be edited, obtain the first Jacobian matrix and the second Jacobian matrix of the region to be edited.

[0115] In this embodiment of the application, when performing semantic editing on the area to be edited in a face image, it is possible that it will affect the surrounding non-editable areas.

[0116] Optionally, the regions of face images that require semantic editing can be defined as the regions to be edited in the face image, and the regions of face images that do not require semantic editing can be defined as the regions to be suppressed in the face image.

[0117] Then, the feature vectors for semantic editing of key points in the region to be edited can be combined with the feature vectors for protecting key points in the region to be suppressed from being edited. Based on the combined feature vectors, a directional semantic editing space for the region to be edited can be generated.

[0118] Optionally, the first and second Jacobian matrices of the region to be edited can be obtained by performing back gradient differentiation based on the key points of the region to be edited.

[0119] When key points within the area to be edited need to be edited, feature vectors for editing the key points within the area to be edited can be obtained based on the first Jacobian matrix.

[0120] Accordingly, when the key points in the region to be edited do not need to be edited, the region to be edited can be identified as the region to be suppressed in the face image, and the feature vector that protects the key points in the region from being edited can be obtained based on the second Jacobian matrix.

[0121] like Figure 4 As shown, a face image G(s) can be input into a face keypoint detector H(·), and the keypoints H(G(s)) of the face image can be output. The corresponding first Jacobian matrix J can be obtained based on the keypoints H(G(s)). v Second Jacobian matrix J w .

[0122] Furthermore, we can modify formula f v = Perform the back gradient derivative of H0(s) to obtain the first Jacobian matrix J. v Accordingly, formula f can be modified. w = Perform the back gradient derivative of F(s) to obtain the second Jacobian matrix J. w .

[0123] Where H0(s) is the keypoint editing function and F(s) is the keypoint suppression function.

[0124] S302 generates a directional semantic editing space based on the first and second Jacobian matrices.

[0125] In this embodiment of the application, feature vectors for editing different regions of a face image can be obtained based on the first Jacobian matrix, and feature vectors for suppressing different regions of a face image can be obtained based on the second Jacobian matrix.

[0126] Furthermore, the semantic editing space of keypoints can be obtained based on the first Jacobian matrix. This part is detailed in the aforementioned related content and will not be repeated here.

[0127] Accordingly, the semantic suppression space of key points can be obtained based on the second Jacobian matrix.

[0128] In implementation, the semantic suppression space of key points can protect the key points in the region to be suppressed, thus preventing them from being edited. At the same time, it can protect the texture information of the key points in the region to be edited, so that the image quality does not change when the region to be edited is edited.

[0129] This allows for the segmentation of a face image according to a defined format, and the determination of the index and centroid coordinates of each segment. For example, a face image can be triangulated (Delaunay), and the index and centroid coordinates of each triangular segment can be determined.

[0130] Regarding the suppression and protection of keypoints in the region to be suppressed, since the semantic information of keypoints in a face image is consistent, keypoints in the region to be suppressed can be uniformly sampled in each segment of the divided face image. While editing the keypoints in the region to be edited, the semantic information and relative positions of the keypoints in that part of the region to be suppressed will not change.

[0131] Optionally, the face image can be triangulated to generate triangular segments. By uniformly sampling each segment in the region to be suppressed and protecting the semantic information and relative position of the key points obtained from the sampling, the suppression and protection of key points in the region to be suppressed can be achieved.

[0132] Accordingly, while suppressing and protecting the region to be suppressed, it is also necessary to protect the texture information of the region to be edited. Optionally, the texture information of the region to be edited can be protected by constraining the attribute parameters of the pixels corresponding to the key points of the region to be edited to remain unchanged.

[0133] Optionally, the semantic suppression space can be implemented using a suppression function, which can be obtained based on the following formula:

[0134] F(s) = α1H1(s) + α2G(s) [P]

[0135] In the above formula, P = {i, c}, where i is the index of the segment after the face image is divided, and c is the centroid coordinate of segment i.

[0136] Furthermore, in order to construct the semantic suppression space, the second Jacobian matrix corresponding to the key point can be obtained, and the feature vector of the semantic suppression space can be obtained through the second Jacobian matrix.

[0137] Specifically, the second transpose Jacobian matrix of the second Jacobian matrix is ​​obtained, and the second matrix is ​​obtained based on the second Jacobian matrix and the second transpose Jacobian matrix.

[0138] As can be seen from the above example, the second Jacobian matrix can be set to J. w Then its transpose matrix and second transpose Jacobian matrix are The product of these two matrices can then be used to determine the second matrix.

[0139] The second Jacobian matrix can be obtained based on the following formula:

[0140]

[0141] In the above formula, J w Let F(s) be the second Jacobian matrix of the function F(s).

[0142] Furthermore, the eigenvalues ​​and eigenvectors of the second matrix are solved to obtain the second eigenvectors corresponding to the second eigenvalues ​​under the second set conditions in the second matrix, so as to construct the semantic suppression space of the key points.

[0143] In this embodiment of the application, the second matrix can be solved, and the feature vectors corresponding to the semantic suppression space of the key points can be obtained from multiple solutions of the second matrix.

[0144] Optionally, the second matrix can be solved based on a set algorithm. For example, all eigenvalues ​​of the second matrix can be obtained by calculating the characteristic polynomial corresponding to the second matrix, and then the eigenvector corresponding to each eigenvalue can be solved. Based on the eigenvalue and its corresponding eigenvector, the solution of the second matrix can be generated.

[0145] In order to achieve effective suppression of key points in the semantic suppression space, a second feature vector that meets the set criteria can be obtained from all solutions of the second matrix. The set criteria can be used as the second set condition.

[0146] In implementation, each eigenvector in all solutions of the second matrix can protect the key points of the region to be suppressed from being edited, and each eigenvector in all solutions of the second matrix can realize the semantic editing of the key points of the region to be edited. The magnitude of the second eigenvalue is negatively correlated with the ability of the second eigenvector to suppress key points. It can be understood that the second eigenvector with a high second eigenvalue has a lower semantic suppression ability for key points than the second eigenvector with a low second eigenvalue.

[0147] To ensure that key points in the suppressed region, which is not to be edited, are not edited in the directional semantic editing space, the corresponding second set conditions can be obtained based on the second feature value, thereby achieving the filtering of feature vectors.

[0148] In some implementations, the second feature vector corresponding to the second feature value that is equal to zero can be obtained to form the semantic suppression space of the key point.

[0149] In this embodiment of the application, a second eigenvector with a second eigenvalue equal to zero can be obtained from all solutions of the second matrix. When the second eigenvalue is zero, it can be determined that the corresponding second eigenvector can effectively protect the key points of the region to be suppressed.

[0150] Furthermore, a semantic suppression space for key points is constructed based on this second feature vector.

[0151] In other implementations, the second feature vectors can be sorted from low to high according to the second feature value, and a second set number of second feature vectors can be obtained sequentially to form the semantic suppression space of the key point.

[0152] In this embodiment of the application, the feature vectors for constructing the semantic suppression space of key points have a set number, which can be determined as a second set number.

[0153] Furthermore, the second feature vectors are sorted based on the second feature values, and a second set number of second feature vectors are obtained from the sorting results from low to high. The semantic suppression space of the key points is constructed based on this part of the second feature vectors.

[0154] Among all solutions to the second matrix, the largest second eigenvalue can be obtained based on the following formula:

[0155]

[0156] In the above formula, J w Let F(s) be the second Jacobian matrix of the function F, α be the fixed step size, and F(s) be the suppression function of the key points.

[0157] Furthermore, the intersection of the semantic editing space and the semantic suppression space is obtained as the directional semantic editing space.

[0158] In this embodiment, the feature vectors of the semantic editing space and the feature vectors of the semantic suppression space can be intersected to construct the directional semantic editing space corresponding to the region to be edited.

[0159] like Figure 5 As shown, the semantic editing space is set as span(v), and the semantic suppression space is... Then, the directional semantic editing space of key points is obtained based on the intersection of the two. Among them, the directional semantic editing space is

[0160] Among them, the directional semantic editing space can perform semantic editing on the key points of the region to be edited, and can protect the key points of the region to be suppressed from being edited. At the same time, it constrains the image quality of the key points of the region to be edited to remain unchanged.

[0161] It should be noted that the first feature vector for constructing the semantic editing space is a set of linearly independent linear basis vectors, which allows for the editing of keypoints. Correspondingly, the second feature vector for constructing the semantic suppression space is also a set of linearly independent linear basis vectors, which ensures that the suppression function remains unchanged, thereby achieving the suppression and protection of keypoints.

[0162] The image generation method proposed in this application obtains the semantic editing space corresponding to key points through a first Jacobian matrix and the semantic suppression space corresponding to key points through a second Jacobian matrix, thereby generating a directional semantic editing space corresponding to the region to be edited. In this application, the directional semantic editing space allows for directional processing of different regions of a face image, resulting in a high degree of decoupling. This enables controllable and interpretable semantic editing of face images, thus optimizing the semantic editing effect of face images.

[0163] In the above embodiments, the generation of the target face image can be combined with... Figure 6 To understand further, Figure 6 This is a flowchart illustrating an image generation method according to another embodiment of this application, as shown below. Figure 6 As shown, the method includes:

[0164] S601, obtain the offset vector of the key point based on the directional semantic editing space and offset parameters.

[0165] In this embodiment, the offset vector of the key points in the region to be edited can be obtained using the key points and corresponding offset parameters. The offset vector can be obtained based on a directional semantic editing space.

[0166] Optionally, corresponding offset parameters can be set, and the offset vector corresponding to the key point can be calculated based on the offset parameters.

[0167] Set the offset parameters to λ1, λ2, λ3, ..., λ n Based on the semantic editing space, the constituent vectors r1, r2, r3, ..., r n and the set offset parameters λ1, λ2, λ3, ..., λ n The offset vector corresponding to the key point is obtained as λ1r1+λ2r2+λ3r3+…+λ n r n .

[0168] S602, based on the offset vector, edits the key points of the region to be edited to generate the target face image.

[0169] In this embodiment of the application, the key points can be edited based on the obtained offset vector. The vector obtained after editing is the key point after the key point offset. Based on the key point information after the offset, the target face image after face image editing can be generated.

[0170] For example, such as Figure 7 As shown, let the face image be G(s), the edited target face image be G(s'), and the offset vector be λ1r1+λ2r2+λ3r3+…+λ n r n The formula for calculating the offset keypoint s' is as follows:

[0171] s'=s+λ1r1+λ2r2+λ3r3+…+λ n r n

[0172] It should be noted that the offset parameter and the offset vector are related to a certain extent. Therefore, by adjusting the offset parameter, the offset vector can be adjusted, thereby enabling control over the semantic editing of key points.

[0173] The offset parameter, which allows for maximum editing of key points, can be obtained based on the following formula:

[0174] [λ * 1,λ * 2,λ * 3,…,λ * n ]=argmaxC(G(s+λ1r1+λ2r2+λ3r3+…+λ n r n ))

[0175] In the above formula, [λ * 1,λ *2,λ * 3,…,λ * n [] is the offset parameter that maximizes the change in keypoints, G(s+λ1r1+λ2r2+λ3r3+…+λ n r n () represents the target face image corresponding to the edited key points.

[0176] Therefore, the vector of the offset face image obtained based on the offset vector and the key points of the region to be edited can be input into the set image generator. Based on the processing of the image generator, the target face image after face image editing can be generated.

[0177] In some implementations, the edited key points of the region to be edited can be integrated with other key points of the region to be suppressed to generate the key points corresponding to the target face image, thereby realizing the generation of the target face image.

[0178] The image generation method proposed in this application obtains the offset vectors of key points in the region to be edited based on the directional semantic editing space and set offset parameters. Furthermore, it edits the key points in the region to be edited based on the offset vectors, thereby generating the target face image corresponding to the face image. In this application, the directional semantic editing space allows for directional processing of different regions of the face image, resulting in a high degree of decoupling. This enables controllable and interpretable semantic editing of the face image, optimizing the semantic editing effect.

[0179] Furthermore, after generating the target face image, attribute recognition can be performed on the target face image, which can be combined with... Figure 8 understand, Figure 8 This is a flowchart illustrating an image generation method according to another embodiment of this application, as shown below. Figure 8 As shown, the method includes:

[0180] S801, input the target face image into the face attribute classifier for classification and recognition, so as to obtain the classification labels of the target face image and the classification probabilities under the classification labels.

[0181] In this embodiment of the application, the target face image can be classified and identified by a face attribute classifier. The classification result can be determined as the classification label of the target face image, and the probability corresponding to each classification label can be determined as the corresponding classification probability.

[0182] For example, the face attribute classifier can be set as a binary classifier, and the attribute classification can be set to whether the person in the target face image is wearing glasses. Based on the attribute dimension corresponding to whether glasses are worn, the attribute classifier can output the classification label of "wearing glasses" and "not wearing glasses". The probability of the person in the target face image wearing glasses is 0.9, and the probability of not wearing glasses is 0.1.

[0183] The classification labels for the target face image are "wearing glasses" and "not wearing glasses". The classification probability of the target face image under the label "wearing glasses" is 0.9, and the classification probability under the label "not wearing glasses" is 0.1.

[0184] It should be noted that the face attribute classifier can classify target face images by multiple attributes. This can be understood as classifying and recognizing different attribute dimensions of the target face image simultaneously, and outputting the corresponding classification labels and the classification probability of the target face image under each classification label.

[0185] S802, Generate attribute classification explanation information for the target face image based on the multi-attribute classification labels and the classification probabilities under the classification labels.

[0186] In this embodiment of the application, classification explanation information for classifying the target face image by attribute can be obtained based on the classification probability corresponding to the classification label of the target face image.

[0187] like Figure 7 As shown, the classification probability of a target face image under the classification label can be obtained using the following formula:

[0188] P = C(G(s'))

[0189] Here, G(s') represents the key points of the target face image.

[0190] It should be noted that, based on the offset parameter corresponding to the currently obtained classification probability, the offset parameter can be adjusted for the next round, thereby controlling the facial attribute classification result. This can be understood as using the classification probability to iteratively optimize the offset parameter, thus achieving optimized targeted editing of the facial image.

[0191] For example, a target face image can be classified according to the dimension of attractiveness. The face image is semantically edited based on the directional semantic editing space corresponding to the eyes, facial contours and mouth, and the corresponding target face image is generated.

[0192] The target face image is generated by magnifying the eyes of a facial image. In the attractiveness dimension attribute classification, the probability of the target face image corresponding to the "attractive" category label output by the facial attribute classifier is 0.99. Therefore, for this category label and corresponding probability, enlarging the eyes can increase the attractiveness of the face, which can be used as the attribute classification explanation information for the target face image in this classification result.

[0193] The target face image is generated by shrinking the facial contours of a face image to reduce its area. In the attractiveness dimension attribute classification, this target face image has a classification probability of 0.97 for the "attractive" label output by the face attribute classifier. Therefore, for this label and corresponding classification probability, reducing the face area can be considered as increasing the face's attractiveness, serving as the attribute classification explanation information for the target face image in this classification result.

[0194] The target face image is generated by adjusting the mouth of a person's face to enhance their smile by raising the corners of their mouth. In the attractiveness dimension attribute classification, the target face image has a classification probability of 1.0 under the label "attractive" output by the face attribute classifier. Therefore, this label and corresponding classification probability can be used as explanation information for the attribute classification of the target face image in this classification result, suggesting that a smile can increase facial attractiveness.

[0195] The image generation method proposed in this application generates attribute classification labels and corresponding classification probabilities for a target face image based on a face attribute classifier. Based on these labels and probabilities, attribute classification interpretation information for the target face image is generated. This application utilizes a directional semantic editing space to achieve directional editing of the face image, thereby making the attribute classification results of the target face image interpretable and controllable.

[0196] Corresponding to the image generation methods proposed in the above embodiments, an embodiment of this application also proposes an image generation apparatus. Since the image generation apparatus proposed in this application corresponds to the image generation methods proposed in the above embodiments, the implementation methods of the above image generation methods are also applicable to the image generation apparatus proposed in this application, and will not be described in detail in the following embodiments.

[0197] Figure 9 This is a schematic diagram of the structure of an image generation apparatus according to an embodiment of this application, as shown below. Figure 9 As shown, the image generation apparatus 900 includes a detection module 91, an acquisition module 92, a generation module 93, and a classification module 94, wherein:

[0198] The detection module 91 is used to obtain the key points of the area to be edited in the face image;

[0199] Module 92 is used to obtain the directional semantic editing space of the region to be edited based on the key points of the region to be edited;

[0200] The generation module 93 is used to edit the region to be edited based on the directional semantic editing space and generate the target face image corresponding to the face image.

[0201] In this embodiment of the application, the acquisition module 92 is further configured to: acquire the first Jacobian matrix of the region to be edited based on the key points of the region to be edited; and generate a directional semantic editing space based on the first Jacobian matrix.

[0202] In this embodiment of the application, the acquisition module 92 is further configured to: acquire the first Jacobian matrix and the second Jacobian matrix of the region to be edited based on the key points of the region to be edited; and generate a directional semantic editing space based on the first Jacobian matrix and the second Jacobian matrix.

[0203] In this embodiment of the application, the acquisition module 92 is further configured to: acquire the semantic editing space of key points based on the first Jacobian matrix; acquire the semantic suppression space of key points based on the second Jacobian matrix; and acquire the intersection of the semantic editing space and the semantic suppression space as the directional semantic editing space.

[0204] In this embodiment of the application, the acquisition module 92 is further configured to: acquire the first transpose Jacobian matrix of the first Jacobian matrix, and acquire the first matrix based on the first Jacobian matrix and the first transpose Jacobian matrix; solve for the eigenvalues ​​and eigenvectors of the first matrix, and acquire the first eigenvectors corresponding to the first eigenvalues ​​that satisfy the first set conditions in the first matrix, so as to constitute the semantic editing space of the key points.

[0205] In this embodiment of the application, the acquisition module 92 is further configured to: acquire the first feature vector corresponding to the first feature value that is greater than or equal to a set threshold, so as to form the semantic editing space of the key point; or, sort the first feature vectors from high to low according to the first feature value, and sequentially acquire a first set number of first feature vectors to form the semantic editing space of the key point.

[0206] In this embodiment of the application, the acquisition module 92 is further configured to: acquire the second transpose Jacobian matrix of the second Jacobian matrix, and acquire the second matrix based on the second Jacobian matrix and the second transpose Jacobian matrix; solve for the eigenvalues ​​and eigenvectors of the second matrix, and acquire the second eigenvectors corresponding to the second eigenvalues ​​of the second set conditions in the second matrix, so as to constitute the semantic suppression space of the key points.

[0207] In this embodiment of the application, the acquisition module 92 is further configured to: acquire the second feature vector corresponding to the second feature value that is equal to zero, thereby forming the semantic suppression space of the key point; or, sort the second feature vectors from low to high according to the second feature value, and sequentially acquire a second set number of second feature vectors to form the semantic suppression space of the key point.

[0208] In this embodiment of the application, the generation module 93 is further configured to: obtain the offset vector of the key points according to the directional semantic editing space and the offset parameter; and edit the key points of the region to be edited based on the offset vector to generate the target face image.

[0209] In this embodiment of the application, the image generation device 900 further includes a classification module 94, which is used to input the target face image into a face attribute classifier for classification and recognition, so as to obtain the classification labels of multiple attributes of the target face image and the classification probabilities under the classification labels; and generate attribute classification explanation information of the target face image based on the classification labels of multiple attributes and the classification probabilities under the classification labels.

[0210] In this embodiment of the application, the detection module 91 is further configured to: obtain a normally distributed noise vector, perform an affine transformation on the noise vector to generate a style vector in the style space; and input the style vector into an image generator to obtain a face image.

[0211] In this embodiment of the application, the acquisition module 92 is further configured to: perform back gradient differentiation on the feature vector of the key point for any one of the first Jacobian matrix and the second Jacobian matrix, so as to generate any Jacobian matrix.

[0212] The image generation apparatus proposed in this application obtains the directional semantic editing space corresponding to the region to be edited based on key points within the region to be edited in a face image, and then edits the region to be edited in the face image according to the directional semantic editing space, thereby generating the corresponding target face image after editing. In this application, different regions of the face image can be processed directionally through the directional semantic editing space, with a high degree of decoupling, enabling controllable and interpretable semantic editing of the face image and optimizing the semantic editing effect of the face image.

[0213] To achieve the above embodiments, this application also proposes an electronic device, a computer-readable storage medium, and a computer program product.

[0214] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 10 As shown, the device 1000 includes a memory 101, a processor 102, and a computer program stored on the memory 101 and executable on the processor 102. When the processor 102 executes program instructions, it implements the image generation method proposed in the above embodiments.

[0215] The electronic device of this application obtains the directional semantic editing space corresponding to the area to be edited within a face image based on key points within that area, and edits the area to be edited based on the directional semantic editing space to generate the corresponding target face image after editing. In this application, the directional semantic editing space allows for directional processing of different regions of the face image, resulting in a high degree of decoupling and enabling controllable and interpretable semantic editing of the face image, thus optimizing the semantic editing effect of the face image.

[0216] This application provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the image generation method proposed in the above embodiments.

[0217] The computer-readable storage medium of this application embodiment obtains the directional semantic editing space corresponding to the area to be edited within a face image based on key points within that area, and edits the area to be edited within the directional semantic editing space to generate the corresponding target face image after editing. In this application, the directional semantic editing space allows for directional processing of different regions of the face image, resulting in a high degree of decoupling and enabling controllable and interpretable semantic editing of the face image, thus optimizing the semantic editing effect of the face image.

[0218] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0219] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0220] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0221] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0222] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0223] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0224] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0225] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0226] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0227] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for generating an image, characterized in that, include: Key points for obtaining the editable region of a face image; Based on the key points of the region to be edited, obtain the directional semantic editing space of the region to be edited; The region to be edited is edited based on the directional semantic editing space to generate the target face image corresponding to the face image; Before obtaining the key points of the editable region of the face image, the process also includes: Obtain a normally distributed noise vector and perform an affine transformation on the noise vector to generate a style vector in the style space; The style vector is input into the image generator to obtain the face image; The step of editing the region to be edited based on the directional semantic editing space to generate the target face image corresponding to the face image includes: Based on the directional semantic editing space and offset parameters, obtain the offset vector of the key point; The key points of the region to be edited are edited based on the offset vector to generate the target face image.

2. The method according to claim 1, characterized in that, The step of obtaining the directional semantic editing space of the region to be edited based on the key points of the region to be edited includes: Based on the key points of the region to be edited, obtain the first Jacobian matrix of the region to be edited; Based on the first Jacobian matrix, the directional semantic editing space is generated.

3. The method according to claim 1, characterized in that, The step of obtaining the directional semantic editing space of the region to be edited based on the key points of the region to be edited includes: Based on the key points of the region to be edited, obtain the first Jacobian matrix and the second Jacobian matrix of the region to be edited. The directional semantic editing space is generated based on the first Jacobian matrix and the second Jacobian matrix.

4. The method according to claim 3, characterized in that, The generation of the directional semantic editing space based on the first Jacobian matrix and the second Jacobian matrix includes: Based on the first Jacobian matrix, the semantic editing space of the key points is obtained; Based on the second Jacobian matrix, the semantic suppression space of key points is obtained; Obtain the intersection of the semantic editing space and the semantic suppression space, and use it as the directional semantic editing space.

5. The method according to claim 4, characterized in that, The step of obtaining the semantic editing space of key points based on the first Jacobian matrix includes: Obtain the first transpose Jacobian matrix of the first Jacobian matrix, and obtain the first matrix based on the first Jacobian matrix and the first transpose Jacobian matrix; The first matrix is ​​solved by calculating eigenvalues ​​and eigenvectors to obtain the first eigenvector corresponding to the first eigenvalue that satisfies the first set condition in the first matrix, so as to form the semantic editing space of the key point.

6. The method according to claim 5, characterized in that, The step of obtaining the feature vector corresponding to the first feature value in the first matrix that satisfies the first set condition, so as to construct the semantic editing space of the key point, includes: Obtain the first feature vector corresponding to the first feature value that is greater than or equal to a set threshold, so as to construct the semantic editing space of the key point; or, The first feature vectors are sorted from high to low according to the first feature value, and a first set number of the first feature vectors are obtained sequentially to form the semantic editing space of the key point.

7. The method according to claim 4, characterized in that, The step of obtaining the semantic suppression space of key points based on the second Jacobian matrix includes: Obtain the second transpose Jacobian matrix of the second Jacobian matrix, and obtain the second matrix based on the second Jacobian matrix and the second transpose Jacobian matrix; The second matrix is ​​solved by calculating eigenvalues ​​and eigenvectors, and the second eigenvectors corresponding to the second eigenvalues ​​under the second set conditions in the second matrix are obtained to form the semantic suppression space of the key points.

8. The method according to claim 7, characterized in that, The step of obtaining the feature vector corresponding to the second feature value of the second set condition in the second matrix to construct the semantic suppression space of the key point includes: Obtain the second feature vector corresponding to the second feature value that is equal to zero, and construct the semantic suppression space of the key point; or, The second feature vectors are sorted from low to high according to the second feature value, and a second set number of the second feature vectors are obtained sequentially to form the semantic suppression space of the key point.

9. The method according to any one of claims 1-8, characterized in that, After generating the target face image corresponding to the face image, the process further includes: The target face image is input into a face attribute classifier for classification and recognition to obtain the classification labels of the target face image for multiple attributes and the classification probabilities under the classification labels; Based on the classification labels of the multiple attributes and the classification probabilities under the classification labels, attribute classification explanation information of the target face image is generated.

10. The method according to claim 3, characterized in that, The method further includes: For any one of the first and second Jacobian matrices, the inverse gradient derivative of the feature vector of the key point is calculated to generate the arbitrary Jacobian matrix.

11. An image generation apparatus, characterized in that, include: The detection module is used to obtain key points of the area to be edited in the face image; The acquisition module is used to acquire the directional semantic editing space of the region to be edited based on the key points of the region to be edited; The generation module is used to edit the region to be edited based on the directional semantic editing space to generate a target face image corresponding to the face image; The detection module is also used for: Obtain a normally distributed noise vector and perform an affine transformation on the noise vector to generate a style vector in the style space; The style vector is input into the image generator to obtain the face image; The generation module is further configured to: Based on the directional semantic editing space and offset parameters, obtain the offset vector of the key point; The key points of the region to be edited are edited based on the offset vector to generate the target face image.

12. The apparatus according to claim 11, characterized in that, The acquisition module is also used for: Based on the key points of the region to be edited, obtain the first Jacobian matrix of the region to be edited; Based on the first Jacobian matrix, the directional semantic editing space is generated.

13. The apparatus according to claim 11, characterized in that, The acquisition module is also used for: Based on the key points of the region to be edited, obtain the first Jacobian matrix and the second Jacobian matrix of the region to be edited. The directional semantic editing space is generated based on the first Jacobian matrix and the second Jacobian matrix.

14. The apparatus according to claim 13, characterized in that, The acquisition module is also used for: Based on the first Jacobian matrix, the semantic editing space of the key points is obtained; Based on the second Jacobian matrix, the semantic suppression space of key points is obtained; Obtain the intersection of the semantic editing space and the semantic suppression space, and use it as the directional semantic editing space.

15. The apparatus according to claim 14, characterized in that, The acquisition module is also used for: Obtain the first transpose Jacobian matrix of the first Jacobian matrix, and obtain the first matrix based on the first Jacobian matrix and the first transpose Jacobian matrix; The first matrix is ​​solved by calculating eigenvalues ​​and eigenvectors to obtain the first eigenvector corresponding to the first eigenvalue that satisfies the first set condition in the first matrix, so as to form the semantic editing space of the key point.

16. The apparatus according to claim 15, characterized in that, The acquisition module is also used for: Obtain the first feature vector corresponding to the first feature value that is greater than or equal to a set threshold, so as to construct the semantic editing space of the key point; or, The first feature vectors are sorted from high to low according to the first feature value, and a first set number of the first feature vectors are obtained sequentially to form the semantic editing space of the key point.

17. The apparatus according to claim 14, characterized in that, The acquisition module is also used for: Obtain the second transpose Jacobian matrix of the second Jacobian matrix, and obtain the second matrix based on the second Jacobian matrix and the second transpose Jacobian matrix; The second matrix is ​​solved by calculating eigenvalues ​​and eigenvectors, and the second eigenvectors corresponding to the second eigenvalues ​​under the second set conditions in the second matrix are obtained to form the semantic suppression space of the key points.

18. The apparatus according to claim 17, characterized in that, The acquisition module is also used for: Obtain the second feature vector corresponding to the second feature value that is equal to zero, and construct the semantic suppression space of the key point; or, The second feature vectors are sorted from low to high according to the second feature value, and a second set number of the second feature vectors are obtained sequentially to form the semantic suppression space of the key point.

19. The apparatus according to any one of claims 11-18, characterized in that, The device further includes: The classification module is used to input the target face image into a face attribute classifier for classification and recognition, so as to obtain the classification labels of the target face image and the classification probabilities under the classification labels; Based on the classification labels of the multiple attributes and the classification probabilities under the classification labels, attribute classification explanation information of the target face image is generated.

20. The apparatus according to claim 13, characterized in that, The acquisition module is also used for: For any one of the first and second Jacobian matrices, the inverse gradient derivative of the feature vector of the key point is calculated to generate the arbitrary Jacobian matrix.

21. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

23. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images