Facial image generation method and apparatus

Abstract

A method and apparatus for generating facial images are provided. The method includes: encoding an input image including a face to determine a first feature vector related to pose and a second feature vector related to identity; flipping the first feature vector relative to an axis in a corresponding space to determine a flipped first feature vector; determining an auxiliary feature vector based on the flipped first feature vector and rotation information corresponding to the input image; determining a final feature vector based on the first feature vector and the auxiliary feature vector; and generating an output image including a rotated face by decoding the final feature vector and the second feature vector based on the rotation information.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0103447, filed on August 18, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes, and also claims the benefit of Korean Patent Application No. 10-2020-0161545, filed on November 26, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and devices for generating facial images. Background Technology

[0003] Facial recognition is a technique in computer vision, which models and helps understand the real world. Large datasets are available for facial recognition using deep learning. A typical facial recognition system using deep learning includes an encoder configured to extract features from an input image and a decoder configured to generate an output image from the feature vectors. The accuracy of facial recognition can be improved through the application of deep learning. However, the recognition rate can still be low due to various factors, including external environmental factors (such as illumination) and other factors associated with image synthesis (such as facial pose in the image). Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, the selection of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a processor-implemented method for generating a face image includes: encoding an input image including a face to determine a first feature vector related to pose and a second feature vector related to identity; determining a flipped first feature vector by flipping the first feature vector relative to an axis in a corresponding space; determining an auxiliary feature vector based on the flipped first feature vector and rotation information corresponding to the input image; determining a final feature vector based on the first feature vector and the auxiliary feature vector; and generating an output image including the rotated face by decoding the final feature vector and the second feature vector based on the rotation information.

[0006] The steps for determining the auxiliary feature vector may include: applying the flipped first feature vector, the positional information of the main facial features in the input image corresponding to the size of the flipped first feature vector, and the rotation information corresponding to the size of the flipped first feature vector to a convolutional neural network (CNN) to determine the auxiliary feature vector.

[0007] The steps of determining the first feature vector and the second feature vector may include: determining the positional information of the main facial features in the input image from the input image; and determining the first feature vector and the second feature vector by encoding the positional information and the input image. The steps of determining the auxiliary feature vector may include: transforming the positional information by flipping the positional information relative to the axis in the corresponding space; and determining the auxiliary feature vector based on the flipped first feature vector, the rotational information corresponding to the input image, and the transformed positional information.

[0008] The steps of transforming the position information may include: adjusting the size of the position information to correspond to the size of the flipped first feature vector.

[0009] Location information may include a key point heatmap corresponding to a face in the input image.

[0010] Location information may include: for each pixel in each input image, a value between 0 and 1 representing the probability of the main facial features in the input image.

[0011] The rotation information may include information indicating the rotation direction used to generate the output image from the input image, and the magnitude of the rotation information may correspond to the magnitude of the flipped first feature vector.

[0012] The steps for determining auxiliary feature vectors based on rotation information may include: transforming the rotation information to indicate the opposite rotation direction; and determining the auxiliary feature vectors based on the transformed rotation information.

[0013] Rotation information may also include information indicating the degree of rotation used to generate the output image from the input image.

[0014] Rotation information can be determined by comparing the positional information of the main facial features in the input image with the preset facial pose in the output image.

[0015] The step of generating an output image may further include: transmitting the output image as an input image for subsequent iterations; and performing subsequent iterations based on the rotation degree included in the rotation information, including determining a first feature vector and a second feature vector, determining a flipped first feature vector, determining an auxiliary feature vector, determining a final feature vector, and generating an output image.

[0016] The steps of encoding the input image may include: encoding the input image using an encoder, and the neural network of the encoder may include: an input layer corresponding to the input image; and an output layer corresponding to a first feature vector and a second feature vector.

[0017] The steps of decoding the final feature vector and the second feature vector may include: using a decoder to decode the final feature vector and the second feature vector based on rotation information, and the neural network of the decoder may include: an input layer corresponding to the final feature vector, the second feature vector and the rotation information; and an output layer corresponding to the output image.

[0018] The step of determining the first feature vector of the flip may include: determining the flipped input image by flipping the input image relative to the axis in the corresponding space; and determining the first feature vector of the flip related to the pose by encoding the flipped input image.

[0019] The method may include: extracting features for face recognition based on the output image; and recognizing faces based on the extracted features.

[0020] The method may include: generating multiple output images corresponding to an input image by changing the degree of rotation included in rotation information; and recognizing a face based on the generated multiple output images.

[0021] The method may include: training a neural network for an encoder for encoding, a neural network for a decoder for decoding, and a neural network for determining auxiliary feature vectors based on an output image and a target image corresponding to an input image.

[0022] A non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, configure the one or more processors to perform the method.

[0023] In another general aspect, a processor-implemented method for generating a face image includes: determining a pose-related first feature vector and an identity-related second feature vector by applying an input image including a face to an encoder; determining a flipped first feature vector by flipping the first feature vector relative to an axis in a corresponding space; determining an auxiliary feature vector by applying the flipped first feature vector and rotation information corresponding to the input image to a first neural network; determining a final feature vector based on the first feature vector and the auxiliary feature vector; generating an output image including a rotated face by applying the final feature vector, the second feature vector, and the rotation information to a decoder; and training a neural network of the encoder, a neural network of the decoder, and the first neural network based on the output image and a target image corresponding to the input image.

[0024] The training steps may include: training based on loss, which includes any one or any combination of the following losses: a loss generated by comparing features extracted from the output image and features extracted from the target image; a loss generated by comparing the output image and the target image pixel by pixel; a loss generated by comparing the identity of the output image and the identity of the target image; a loss based on the difference in pixel values ​​between pixels in the output image; and a generative adversarial network (GAN) loss based on the output image and the target image.

[0025] In another general aspect, a facial image generation apparatus includes: one or more processors configured to: determine a pose-related first feature vector and an identity-related second feature vector by encoding an input image including a face; determine a flipped first feature vector by flipping the first feature vector relative to an axis in a corresponding space; determine an auxiliary feature vector based on the flipped first feature vector and rotation information corresponding to the input image; determine a final feature vector based on the first feature vector and the auxiliary feature vector; and generate an output image including a rotated face by decoding the final feature vector and the second feature vector based on the rotation information.

[0026] To determine the auxiliary feature vector, the one or more processors may be configured to determine the auxiliary feature vector by applying a flipped first feature vector, positional information of the main facial features in the input image corresponding to the size of the flipped first feature vector, and rotation information corresponding to the size of the flipped first feature vector to a convolutional neural network (CNN).

[0027] The one or more processors may be configured to: determine the positional information of the main facial features in the input image from the input image in order to determine the first feature vector and the second feature vector, and determine the first feature vector and the second feature vector by encoding the positional information and the input image; and transform the positional information by flipping the positional information relative to the axis in the corresponding space and adjusting the size of the positional information to a size corresponding to the size of the flipped first feature vector in order to determine the auxiliary feature vector based on the flipped first feature vector, the rotation information corresponding to the input image and the transformed positional information.

[0028] The rotation information may include information indicating the rotation direction and degree of rotation used to generate the output image from the input image, and the magnitude of the rotation information may correspond to the magnitude of the first feature vector being flipped.

[0029] To generate an output image, the one or more processors may be configured to: transmit the output image as an input image for subsequent iterations; and, based on the rotation degree included in the rotation information, perform subsequent iterations to determine a first feature vector and a second feature vector, determine a flipped first feature vector, determine an auxiliary feature vector, determine a final feature vector, and generate the output image.

[0030] In order to encode the input image, the one or more processors may be configured to encode the input image using an encoder, and the neural network of the encoder may include: an input layer corresponding to the input image; and an output layer corresponding to a first feature vector and a second feature vector.

[0031] In order to decode the final feature vector and the second feature vector, the one or more processors may be configured to: use a decoder to decode the final feature vector and the second feature vector based on rotation information, and the neural network of the decoder may include: an input layer corresponding to the final feature vector, the second feature vector and the rotation information; and an output layer corresponding to the output image.

[0032] To determine the first feature vector of the flip, the one or more processors may be configured to: determine the flipped input image by flipping the input image relative to the axis in the corresponding space; and determine the first feature vector of the flip, which is related to the pose, by encoding the flipped input image.

[0033] The one or more processors may be configured to: extract features for face recognition based on the output image; and recognize faces based on the extracted features.

[0034] In another general aspect, a processor-implemented method for generating a face image includes: determining a pose-related first feature vector based on an input image including a face; determining a flipped feature vector as a symmetric transformation of the first feature vector; determining an auxiliary feature vector based on the flipped feature vector and rotation information corresponding to the input image; determining a final feature vector based on the first feature vector and the auxiliary feature vector; and generating an output image including the rotated face based on the final feature vector and the rotation information.

[0035] Other features and aspects will become clear from the following detailed description, the accompanying drawings, and the claims. Attached Figure Description

[0036] Figure 1 An example of a facial image generation method is shown.

[0037] Figure 2 An example of a facial image generation model is shown.

[0038] Figure 3An example of a model that repeatedly executes a facial image generation method is shown.

[0039] Figure 4 An example of the flip module of the face image generation model is shown.

[0040] Figure 5 An example of an auxiliary feature extraction module for a facial image generation model is shown.

[0041] Figure 6A and Figure 6B An example of the flip operation of the flip module is shown.

[0042] Figure 7 An example of an auxiliary feature extraction module is shown.

[0043] Figure 8 An example of applying a symmetric transformation to an input image is shown.

[0044] Figure 9 An example of a training method for a facial image generation model is shown.

[0045] Figure 10 An example is shown where a model is trained using a loss generated by comparing the identity of the output image with the identity of the target image.

[0046] Figure 11 An example is shown of training a model using a generative adversarial network (GAN) loss based on the output image and the target image.

[0047] Figures 12A to 12C An example using an image generation model is shown.

[0048] Figure 13 An example of a facial image generation device is shown.

[0049] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0050] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of known features may be omitted.

[0051] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will become clear upon understanding the disclosure of this application.

[0052] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any one and any combination of any two or more of the associated listed items. As used herein, the terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, elements, components, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, elements, components, and / or combinations thereof. The use of the term “may” herein with respect to examples or embodiments (e.g., regarding what an example or embodiment may include or implement) indicates the presence of at least one example or embodiment that includes or implements such features, while all examples are not limited thereto.

[0053] Although the terms “first” or “second” are used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as a second component, second assembly, second region, second layer, or second part may also be referred to as a second component, second assembly, second region, second layer, or second part.

[0054] Throughout this specification, when an element such as a layer, region, or substrate is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" that other element, directly "connected to," or "bonded to" that other element, or there may be one or more other elements in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, there may be no other elements in between. Similarly, expressions such as "between" and "immediately between," and "adjacent to" and "closely adjacent to" can also be interpreted as described above.

[0055] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and as commonly understood after understanding this disclosure. Unless clearly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalized sense.

[0056] Furthermore, in the description of the exemplary embodiments, such descriptions will be omitted when it is believed that a detailed description of the structure or function learned after understanding the disclosure of this application would lead to a vague interpretation of the exemplary embodiments.

[0057] In the following description, examples will be described in detail with reference to the accompanying drawings, and the same reference numerals in the drawings always denote the same elements.

[0058] Figure 1 An example of a facial image generation method is shown.

[0059] Reference Figure 1 A facial image generation method may include an operation 110 of encoding an input image, an operation 120 of performing a symmetric transformation on a feature vector obtained through encoding, and an operation 130 of generating an output image by decoding the feature vector obtained through encoding and the feature vector obtained through the symmetric transformation. One or more embodiments of the facial image generation method may improve the accuracy of facial recognition, regardless of adverse factors including external environmental factors (such as illumination) and other factors related to image synthesis (such as facial pose in the image).

[0060] For example, the encoding operation 110 may include encoding an input image including a face to obtain a pose-related first feature vector and an identity-related second feature vector. Operation 110 may be performed by an encoding module (e.g., an encoder) including a neural network. In one example, the encoder's neural network may include an input layer corresponding to the input image and an output layer corresponding to the first and second feature vectors. The encoding module may discriminately output the pose-related and identity-related feature vectors by learning or training pose-related and identity-related features from the input image in a disentangled manner. The pose-related feature vector may correspond to a feature vector based on the pose of the face (e.g., the angle and / or orientation of the face in the input image). The identity-related feature vector may correspond to a feature vector independent of the face's pose. In the output of the encoding module, the pose-related feature vector is referred to herein as the first feature vector, and the identity-related feature vector independent of the face's pose is referred to herein as the second feature vector.

[0061] The encoding module may be a module configured to receive an input image and location information of the main facial features in the input image as input, and output a first feature vector and a second feature vector. Hereinafter, the location information of the main facial features will also be referred to as the location information of the input image. The location information of the input image may be information indicating the location of the main facial features used to verify the pose of the face in the input image. For example, referring to… Figure 4 and Figure 5 Each of the location information 402 of input image 401 and the location information 502 of input image 501 may include a landmark heatmap, which represents the probability of the principal facial feature of each pixel in each of input images 401 and 501 as a value between 0 and 1. Here, as a non-limiting example, the principal facial feature or main facial feature may include any facial feature (such as any one of the eyes, nose, and lips).

[0062] Prior to operation 110, the face image generation method may include obtaining positional information of the input image from the input image. The positional information of the input image may be input to an encoding module along with the input image, and a first feature vector and a second feature vector may subsequently be output. The positional information of the input image can be used to determine rotational information for generating the output image and to obtain auxiliary feature vectors. Non-limiting examples of rotational information and auxiliary feature vectors will be described in detail below.

[0063] The symmetry transformation operation 120 may include obtaining a flipped first feature vector by flipping a first feature vector relative to an axis in the corresponding space, obtaining an auxiliary feature vector based on the flipped first feature vector and rotation information corresponding to the input image, and obtaining a final feature vector based on the first feature vector and the auxiliary feature vector. Operation 120 may be performed by a symmetry transformation module. Non-limiting examples of symmetry transformation modules will be described in detail below.

[0064] Decoding operation 130 may include generating an output image of a rotated face by decoding the final feature vector obtained in operation 120 and the second feature vector obtained in operation 110 based on rotation information corresponding to the input image. Operation 130 may be performed by a decoding module (e.g., a decoder) including a neural network. In one example, the neural network of the decoder may include an input layer corresponding to the final feature vector, the second feature vector, and the rotation information, and an output layer corresponding to the output image. The decoding module may generate an output image of a face in the input image rotated based on a preset facial pose in the output image. The preset facial pose may include a frontal face, a side face, etc. For example, the preset facial pose may include a pose rotated left or right by a preset angle from a frontal face. The rotation information corresponding to the input image may be determined based on the position information of the face in the input image according to the preset facial pose. That is, the rotation information may be determined by comparing the position information of the input image and the preset facial pose. The rotation information may include information about whether an output image corresponding to the preset facial pose is generated based on the direction and degree of rotation of the face in the input image. That is, the rotation information may include information indicating the direction of rotation and information indicating the degree of rotation for generating the output image from the input image. In one example, the magnitude of the rotation information can correspond to the magnitude of the first feature vector that has been flipped.

[0065] The decoding module can generate an output image in which the face in the input image has been rotated by a preset rotation angle. Therefore, the decoding module can generate an output image in which the face in the input image has been rotated based on information indicating the rotation degree. For example, to generate an output image in which the face in the input image has been rotated by a preset rotation angle, the decoding module can repeat the face image generation method based on the information indicating the rotation degree to generate an output image in which the face in the input image has been rotated to correspond to a preset facial pose.

[0066] In other words, in a facial image generation method, the operation of generating an output image may include passing the generated output image as an input image for subsequent iterations and performing iterations based on the rotation degrees included in the rotation information. Here, iteration means repeating the facial image generation method and includes obtaining a first feature vector and a second feature vector, obtaining a flipped first feature vector, obtaining an auxiliary feature vector, obtaining a final feature vector, and generating an output image.

[0067] Figure 2 An example of a facial image generation model is shown.

[0068] A facial image generation model may include one or more modules that perform the facial image generation methods described herein, and includes one or more processors and / or one or more memories. For ease of description, the facial image generation model will be described below by each module based on its functionality. However, the facial image generation model is not limited to the structure shown. The facial image generation model may also include memory. Hereinafter, the facial image generation model will also be simply referred to as the model.

[0069] Reference Figure 2 The model may include modules that perform various operations of the facial image generation method described herein. For example, the model may include an encoding module 210, a decoding module 230, and a symmetry transformation module 220, wherein the encoding module 210 is configured to encode an input image and extract feature vectors, the decoding module 230 is configured to decode the feature vectors and generate an output image, and the symmetry transformation module 220 is configured to transform the feature vectors output from the encoding module 210 into feature vectors that will be input to the decoding module 230.

[0070] As described above, the encoding module 210 can encode an input image including a face and output a pose-related first feature vector and an identity-related second feature vector. The encoding module 210 can receive the position information of the input image and the input image as input to encode the input image.

[0071] As described above, the decoding module 230 can decode the final feature vector and the second feature vector based on the rotation information corresponding to the input image, and generate an output image including the rotated face. For example, through decoding, the decoding module 230 can generate an output image in which the face in the input image is rotated by a preset rotation angle, or generate an output image in which the face in the input image is rotated based on information indicating the degree of rotation included in the rotation information.

[0072] like Figure 3 As shown, when the decoding module 230 generates an output image in which the face in the input image is rotated by a preset rotation angle, the model can generate an output image in which the face in the input image is rotated to correspond to a preset facial pose. Figure 3 An example of a model that repeatedly executes a facial image generation method is shown. (See reference...) Figure 3Based on the information indicating the degree of rotation, the facial image generation method executed by the model can be repeated twice, and then an output image corresponding to a preset facial pose can be generated. In this case, the facial image generation method can be repeatedly executed by the model using the output of the decoding module 310 as the input of the encoding module 320, and an output image can be generated. For example, in the case of generating an output image of a face rotated 30 degrees (°) in the input image based on the position information of the input image and the preset facial pose information in the output image, the output image of the face rotated by 30° can be generated by the model executing the facial image generation method twice to generate an image of the face rotated by 15° each time. In one example, information indicating the direction of rotation can also be included in the rotation information, and the decoding module can decode the output image based on the information indicating the direction of rotation.

[0073] Although the model is Figure 3 The model is shown as comprising multiple physically identical modules, but in another example, the operations performed by the model can be repeated by inputting the output of the decoding module into the encoding module. For example, in Figure 2 In the example, the face image generation method can be repeatedly executed by the model using the output of the decoding module 230 as a new input to the encoding module 210, and an output image can be generated.

[0074] Return to reference Figure 2 The symmetry transformation module 220 can receive a first pose-related feature vector from the output of the encoding module 210, and the output will be input to the final feature vector of the decoding module 230. The symmetry transformation module 220 can generate a flipped first feature vector by flipping the input first feature vector relative to an axis in the space corresponding to the first feature vector, generate an auxiliary feature vector based on the flipped first feature vector and rotation information, and generate the final feature vector based on the auxiliary feature vector and the first feature vector. The final feature vector can be obtained by calculating the auxiliary feature vector and the first feature vector. The calculation of the final feature vector may include element-wise addition of the auxiliary feature vector and the first feature vector.

[0075] The symmetry transformation module 220 may include a flip module 240, which is configured to flip the first feature vector of the input relative to an axis in the corresponding space to output a flipped first feature vector. A non-limiting example of the flip module 240 in the model may be as follows: Figure 4 The operation is performed as shown. Figure 4 This shows an example of the flip module of a face image generation model. (See reference...) Figure 4 The flipping module 420 can receive the first feature vector 421 from the output of the encoding module 410 as input, and output the flipped first feature vector 422 by flipping the first feature vector 421. In the following text, reference will be made to... Figure 6A and Figure 6B Provide a detailed description of a non-limiting example of flipping.

[0076] Return to reference Figure 2 The symmetry transformation module 220 may include an auxiliary feature extraction (FE) module 250, which is configured to generate auxiliary feature vectors based on the flipped first feature vector and rotation information. A non-limiting example of the auxiliary FE module 250 in the model may be as follows: Figure 5 The operation is performed as shown. Figure 5 This illustrates an example of an auxiliary feature extraction module in a facial image generation model. (See reference...) Figure 5 The auxiliary FE module 530 can receive the flipped first feature vector 522, the transformed position information 503 and rotation information 504 of the input image as input, and output an auxiliary feature vector by applying the input to the neural network 531. In the following text, reference will be made to... Figure 7 A non-limiting example describing the detailed structure of the auxiliary FE module 530.

[0077] The transformed position information 503 can be information obtained by transforming the position information obtained from the input image, which will be applied to the neural network 531. The transformed position information 503 may include information obtained by performing a transformation to flip the position information obtained from the input image relative to an axis in the corresponding space and / or a transformation to adjust the size of the position information to correspond to the size of the flipped first feature vector. In other words, the transformed position information 503 can correspond to information where the position information extracted from the input image has been flipped or resized.

[0078] Additionally, the rotation information 504 to be applied to the neural network 531 may include information indicating a rotation direction opposite to the rotation information input to the decoding module 540. For example, if the rotation information 504 input to the decoding module 540 indicates that the face in the input image needs to be rotated to the left to generate the output image, the rotation information can be transformed to include information indicating a right rotation, and then the transformed rotation information can be input to the neural network 531 of the auxiliary FE module 530. That is, before being input to the neural network 531, the rotation information 504 can be transformed by the model's processor to have information indicating the opposite rotation direction.

[0079] Figure 6A and Figure 6B An example of the flip operation of the flip module is shown.

[0080] Reference Figure 6AThe first feature vector 601 may include a specific number and size of channels. For example, the first feature vector 601 may include C channels of size W×H, where the C channels of W×H can be represented by W×H×C. The flipped first feature vector 602, as the output of the flipping module, may include the same size and number of channels as the first feature vector 601, and can be obtained by flipping each channel of the first feature vector 601 relative to the vertical axis in the corresponding space.

[0081] Reference Figure 6B The first feature vector may include C channels 610, 620, and 630 of size W × H. A flipped first feature vector including C channels 640, 650, and 660 of size W × H can be obtained by flipping each channel of the first feature vector relative to a specific axis in space. For example, channel 640 of the flipped first feature vector can be obtained by flipping channel 610 of the first feature vector relative to a height axis 603 passing through the center of two-dimensional (2D) space. In this example, as a result of the flip relative to axis 603, element 611 in channel 610 of the first feature vector can be moved to element 641 in channel 640 of the flipped first feature vector. For example, if each channel of the first feature vector corresponds to an image, each channel of the flipped first feature vector may correspond to an image in which each channel of the first feature vector is flipped horizontally.

[0082] Figure 7 An example of an auxiliary FE module is shown. (See reference...) Figure 7 The auxiliary FE module may include a neural network 720 configured to output auxiliary feature vectors, and may output auxiliary feature data 730 from input data 710, which includes any one or any combination of a flipped first feature vector 711, position information 712 of the input image (e.g., transformed position information), and rotation information 713 (e.g., transformed rotation information). The neural network 720 of the auxiliary FE module may be a convolutional neural network (CNN). Although the neural network 720 in... Figure 7 The example shown is a CNN architecture including convolutional layers (Conv), but the structure of the neural network 720 with auxiliary FE modules is not limited to the example shown. For example, the neural network 720 configured to output auxiliary feature vectors can be implemented by various neural networks with various structures (e.g., CNNs and deep neural networks (DNNs)).

[0083] In the case that the neural network 720 is a CNN, each input channel can be implemented with the same dimension and the same size. For example, the input of the neural network 720 may include a first feature vector 711 with C channels of size W×H that has been flipped, position information 712 of the input image with one channel of size W×H, and rotation information 713 of one channel of size W×H.

[0084] As described above, the positional information of the input image obtained from the input image can be transformed for input into the neural network 720. That is, the positional information of the input image to be input into the neural network 720 can correspond to positional information transformed by operations performed by the model's processor. Here, the transformation of the input image's positional information can include a symmetric transformation relative to a specific axis in space. Additionally, the transformation of the positional information can include resizing. For example, if the size of each channel of the first feature vector is W×H, the positional information of the input image can be resized to have a size of W×H. That is, the positional information 712 can correspond to information where the positional information of the input image is flipped and / or resized.

[0085] Rotation information 713 can be information of size W×H that includes information indicating the direction of rotation. Rotation information 713 can correspond to data that sets the value of each element in a W×H 2D matrix to a first or second value based on the rotation direction. For example, the first value could be 1, indicating a left rotation direction, and the second value could be -1, indicating a right rotation direction. Alternatively, rotation information 713 can be set to values ​​indicating both the rotation direction and the degree of rotation. For example, if the rotation direction is left and the degree of rotation is 15°, the value of rotation information 713 can be set to 1. For example, if the rotation direction is left and the degree of rotation is 30°, the value of rotation information 713 can be set to 2. For example, if the rotation direction is right and the degree of rotation is 30°, the value of rotation information 713 can be set to -2.

[0086] As described above, the rotation information 713 input to the neural network 720 of the auxiliary FE module may include information indicating the direction opposite to the direction in which the input image needs to be rotated to have a preset facial pose. For example, if the input image needs to be rotated 30° to the left to have a preset facial pose, the rotation information 713 input to the neural network 720 of the auxiliary FE module may include information indicating a 30° rotation to the right.

[0087] Figure 8 An example of applying a symmetric transformation to an input image is shown.

[0088] Reference Figure 8The flipping module 820 can flip the input image 801 relative to an axis in space to output a flipped input image 802. The input image 801 and the flipped input image 802 can be input to encoding modules 810 and 811 respectively, and a first feature vector related to the poses of the input image 801 and the flipped input image 802 respectively can be output. In one example, the first feature vector corresponding to the flipped input image 802 (e.g., the flipped first feature vector) can be input to the auxiliary FE module 830, and an auxiliary feature vector can be output. The auxiliary FE module 830 can receive rotation information 803 and / or position information of the flipped input image 802, as well as the first feature vector corresponding to the flipped input image 802, as input. Here, the size of the position information of the flipped input image 802 can be adjusted to have a size corresponding to the size of the first feature vector of the flipped input image 802. That is, the position information of the flipped input image 802 can correspond to the transformed position information obtained by flipping the position information of the input image 801 and adjusting the size of the position information of the input image 801. In one example, the final feature vector to be input to the decoding module 840 can be obtained based on the auxiliary feature vector output from the auxiliary FE module 830 and the first feature vector corresponding to the input image 801 output from the encoding module 810. The decoding module 840 can decode the final feature vector and the second feature vector, which is independent of the pose corresponding to the input image 801, based on the rotation information 803 to generate the output image.

[0089] In other words, the facial image generation method of one or more embodiments described herein may include: an operation of encoding an input image including a face to obtain a pose-related first feature vector and a pose-independent identity-related second feature vector; an operation of flipping the input image relative to an axis in a corresponding space to obtain a flipped input image; an operation of encoding the flipped input image to obtain a pose-related flipped first feature vector; an operation of obtaining an auxiliary feature vector based on the flipped first feature vector and rotation information corresponding to the input image; an operation of obtaining a final feature vector based on the first feature vector and the auxiliary feature vector; and an operation of generating an output image including a rotated face by decoding the final feature vector and the second feature vector based on rotation information.

[0090] Figure 9 An example of a training method for a facial image generation model is shown.

[0091] Reference Figure 9Model 910 can be trained based on input image 901 and target image 902 corresponding to input image 901. That is, the neural network in model 910 can be trained to generate output image 903 similar to target image 902, and can be trained using various training or learning algorithms. The neural network in model 910 may include a neural network of encoding module 920 for extracting first and second feature vectors from the input image, a neural network of decoding module 940 for generating the output image, and a neural network of auxiliary FE module 930. Training of model 910 can be performed by a training module including a processor and / or memory.

[0092] In one example, a face image generation model can be trained based on various loss functions. These loss functions can be determined based on a single loss or a combination of multiple losses. The face image generation model can be trained by optimizing the parameters of the neural network within the model based on such loss functions.

[0093] Here, the loss used to generate the loss function may include, for example, perceptual loss between the output image and the target image, pixel-wise loss generated by comparing the output image and the target image pixel by pixel, identity preservation loss generated by comparing the identity of the output image and the identity of the target image, loss based on the difference in pixel values ​​between pixels of the output image, and generative adversarial network (GAN) loss based on the output image and the target image, etc.

[0094] The perceptual loss between the output and target images, and the pixel-wise loss generated by comparing the output and target images pixel-wise, correspond to the loss determined by comparing the output and target images. The perceptual loss between the output and target images can be generated by comparing features extracted from the output image and features extracted from the target image. The pixel-wise loss generated by comparing the output and target images pixel-wise corresponds to the loss generated based on the difference between the pixel values ​​of the output image and the corresponding pixel values ​​of the target image. The face image generation model can be trained to generate output images similar to the target image by using the loss determined by comparing the output and target images.

[0095] The loss based on the difference in pixel values ​​between pixels in the output image can be equivalent to the loss based on the difference between the pixel value of a pixel in the output image and the pixel value of its neighboring pixels. A face image generation model can be trained to generate output images in which neighboring pixels are naturally connected by using a loss determined based on the pixel values ​​of pixels in the output image.

[0096] The identity preservation loss, generated by comparing the identities in the output image and the target image, can be determined based on the identity-related second feature vector obtained from the input image and the second feature vector obtained from the target image. The identity preservation loss can correspond to the loss in synthesizing or generating a face in the output image while preserving the same identity of the face in the input image. In one example, reference will be made below. Figure 10 This section describes in detail the module used to train a model using a loss generated by comparing the identity of the output image with the identity of the target image. Figure 10 This illustrates an example of training a model using a loss generated by comparing the identity of the output image with the identity of the target image. (See reference...) Figure 10 The identity preservation module 1020 can perform model training based on a loss generated by comparing the identity of the output image 1003 with the identity of the target image 1002. The identity preservation module 1020 can train the neural network in the model based on a loss determined by comparing the second feature vector of the target image 1002 output from the encoding module 1010 with the second feature vector of the input image 1001 output from the encoding module 1011. That is, the neural network of the facial image training model is trained to generate an output image 1003 from the input image 1001 that maintains the same identity as either the input image 1001 or the target image 1002.

[0097] Figure 11 An example is shown where a model is trained using a GAN loss based on the output and target images (e.g., GAN loss 1104). See reference. Figure 11 The discriminator 1120 can be trained to determine that the output image 1103 is fake and that the target image 1102 is real, and the face image generation model 1110 (or, generator 1110 hereinafter) can be trained to generate the output image 1103 that is determined to be real by the discriminator 1120 using the GAN loss 1104. The generator 1110 can be trained to generate an output image 1103 that is similar to the target image 1102 by using the GAN loss 1104.

[0098] Figures 12A to 12C An example of using an image generation model (e.g., a generator) is shown.

[0099] Reference Figures 12A to 12C A facial image generation model (or, the generator below) can be used for user verification by matching a user’s previously stored gallery images (Gallery IMG) with a newly input test image (Probe IMG).

[0100] Figure 12AThe diagram illustrates a process of inputting a previously stored image (Gallery IMG) and a target image (Probe IMG) into a generator, generating output images 1201 and 1202 respectively, each containing a face with the same facial pose, and performing user verification based on the generated output images 1201 and 1202. Here, the facial pose to be included in the output images can be preset, and the model can obtain rotation information for generating the output images based on the preset facial pose.

[0101] Figure 12B The diagram illustrates a process of inputting a previously stored image (Gallery IMG) into a model, generating an output image 1211 containing a face with the same facial pose as the target image (Probe IMG), and performing user verification based on the generated output image 1211 and the target image. Here, the rotation information used by the model to generate the output image is obtained by detecting facial pose information in the target image and comparing it with positional information from the previously stored image. By detecting facial pose information in the previously stored image and inputting the target image into the model, the output image can be generated to include a rotated face, such that the face in the target image has the same facial pose as the face in the previously stored image.

[0102] Figure 12C The diagram illustrates the process of inputting a previously stored image (Gallery IMG) into a model, generating an output image 1220 including faces with various facial poses, and performing user verification based on the generated output image 1220 and the target image (Probe IMG). Here, the rotation information used in the model can be determined based on various preset facial poses. User verification can be performed based on the generated output image 1220 and the target image. (Refer to...) Figure 12C User verification can be performed by selecting an output image from the generated output image 1220 that corresponds to a facial pose similar to that in the target image.

[0103] In one example, images of faces with various poses output from an image generation model or generator can be used as training data for a face recognition model. For instance, if the training data primarily consists of images containing frontal faces, by using a generator, the training data can be augmented by outputting images of faces from various angles while preserving the identity of faces from images containing frontal faces. By augmenting the training data with facial images containing faces from various angles, the performance of the face recognition model can be improved, enabling it to recognize faces even when images containing faces from various angles are input. In one example, the face image generation method can extract features for face recognition based on the output image and recognize faces based on the extracted features.

[0104] Additionally, auxiliary feature vectors and / or final feature vectors from one or more embodiments can augment the observed features in the feature space, thereby improving the accuracy of the face recognition model. For example, the augmented features can be used for meta-variance transfer (MVT), which transfers variance observed in some classes to another class.

[0105] In situations where the training data used to train a facial recognition model contains various facial angles or where there is significant disparity between facial angles in the training data, a generator of one or more embodiments can be applied to effectively train the facial recognition model. For example, for an image included in the training data, by determining a displacement vector related to the facial angle and fitting the displacement vector to a single trainable or learnable vector using a generator, one or more embodiments can train a trainable facial recognition model to minimize the disparity between facial angles in the training data.

[0106] Figure 13 An example of a facial image generation device is shown.

[0107] Any or all of the facial image generation model (or generator) or facial image generation method described herein may be implemented by facial image generation device 1300. Any or all of the operations of the facial image generation method may be performed by processor 1310 of the device. Facial image generation device 1300 may include output device 1340 and processor 1310, output device 1340 being configured to output an image including a face having a facial pose different from that in the input image, and processor 1310 being configured to perform user verification using the output image. As a non-limiting example, facial image generation device 1300 may be a smartphone, wearable device, tablet computer, netbook, laptop computer, or desktop computer, or may include a smartphone, wearable device, tablet computer, netbook, laptop computer, or desktop computer.

[0108] Reference Figure 13 The facial image generation device 1300 may include a processor 1310 (e.g., one or more processors), a storage device 1320 (e.g., one or more memories), a sensor 1330, and an output device 1340. The processor 1310, storage device 1320, sensor 1330, and output device 1340 may communicate with each other via a communication bus 1350.

[0109] Sensor 1330 may include at least one sensor. Sensor 1330 may be an image sensor (e.g., a camera), or may include an image sensor (e.g., a camera). Sensor 1330 may capture an input image.

[0110] The processor 1310 can execute the functions and instructions in the facial image generation device 1300. For example, the processor 1310 can process instructions stored in the storage device 1320. The processor 1310 can execute the above-mentioned... Figures 1 to 12C The processor 1310 may include one or more of the operations described. The processor 1310 may include an encoding module, a decoding module, and a symmetry transformation module. The encoding module is configured to encode the input image captured by the sensor 1330 and extract feature vectors. The decoding module is configured to decode the feature vectors and generate an output image. The symmetry transformation module is configured to transform the feature vectors output from the encoding module into feature vectors that will be input to the decoding module. The processor 1310 may use the output image to perform user verification. The processor 1310 may unlock the functionality of the face image generation device 1300 based on the result of the user verification.

[0111] Output device 1340 can provide the output of facial image generation device 1300 to a user via a visual channel, an auditory channel, or a tactile channel. Output device 1340 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, a touchscreen, a speaker, a vibration generator, and other devices that can provide the output to the user. In one example, output device 1340 can provide an unlocked screen as a result of user authentication. In another example, output device 1340 can provide an output image.

[0112] Storage device 1320 may store information and data required for execution by processor 1310. Storage device 1320 may store instructions to be executed by processor 1310. Storage device 1320 may include computer-readable storage media (e.g., random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), magnetic hard disk, optical disk, flash memory, erasable programmable read-only memory (EPROM), and other types of computer-readable storage media known in the art to which this disclosure pertains).

[0113] According to the example embodiments described herein, methods and apparatus of one or more embodiments can generate facial images that look similar to actual faces by using facial symmetry to supplement insufficient information of facial features extracted from an input image.

[0114] Regarding Figures 1 to 13The described facial image generation device includes an encoding module, a symmetry transformation module, a decoding module, a flipping module, an auxiliary FE module, an identity preservation module, a generator, a discriminator, a storage device, a sensor, an output device, and encoding module 210, symmetry transformation module 220, decoding module 230, flipping module 240, auxiliary FE module 250, decoding module 310, encoding module 320, encoding module 410, flipping module 420, auxiliary FE module 530, decoding module 540, and encoding module 810. 811, flip module 820, auxiliary FE module 830, decoding module 840, model 910, encoding module 920, auxiliary FE module 930, decoding module 940, encoding module 1010, encoding module 1011, identity retention module 1020, generator 1110, discriminator 1120, facial image generation device 1300, storage device 1320, sensor 1330, output device 1340, and other devices, apparatuses, units, modules, and components are implemented or represented by hardware components. Examples of hardware components that can be used to perform the operations described in this application suitably include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented using one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms “processor” or “computer” may be used in the description of the examples described herein, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented using a single processor, or two or more processors, or a processor and a controller.One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0115] Figures 1 to 13 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented as described above to execute instructions or software to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0116] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers, such as machines or special-purpose computers, to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding description used herein, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0117] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, etc. Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions or software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0118] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

Claims

1. A method for generating a facial image, comprising: By encoding the input image, including the face, a first feature vector related to pose and a second feature vector related to identity are determined. The first eigenvector is determined by flipping the first eigenvector relative to the axis in the corresponding space; Based on the first flipped feature vector and the rotation information corresponding to the input image, an auxiliary feature vector is determined. The final feature vector is determined based on the first feature vector and the auxiliary feature vector; and By decoding the final feature vector and the second feature vector based on rotation information, an output image of the rotated face is generated. The step of determining the first flipped feature vector includes: relative to the vertical axis in the corresponding space, determining the first flipped feature vector by moving the elements in each channel of the first feature vector to the elements in each channel of the flipped first feature vector.

2. The face image generation method according to claim 1, wherein The steps for determining auxiliary feature vectors include: Auxiliary feature vectors are determined by applying the first flipped feature vector, the positional information of the main facial features in the input image corresponding to the size of the first flipped feature vector, and the rotation information corresponding to the size of the first flipped feature vector to the convolutional neural network.

3. The facial image generation method according to claim 1, wherein, The steps to determine the first eigenvector and the second eigenvector include: Determine the location information of the main facial features in the input image; and By encoding the location information and the input image, a first feature vector and a second feature vector are determined, and The steps for determining auxiliary feature vectors include: The position information is transformed by flipping the position information relative to the axis in the corresponding space; and Auxiliary feature vectors are determined based on the first flipped feature vector, the rotation information corresponding to the input image, and the transformed position information.

4. The face image generation method according to claim 3, wherein The steps for transforming location information include: Adjust the size of the position information to correspond to the size of the first flipped feature vector.

5. The face image generation method according to claim 3, wherein Location information includes a heatmap of key points corresponding to the face in the input image.

6. The facial image generation method according to claim 3, wherein, Location information includes: for each pixel in each input image, a value between 0 and 1 representing the probability of the main facial features in the input image.

7. The facial image generation method according to claim 1, wherein, Rotation information includes information indicating the rotation direction used to generate the output image from the input image, and The magnitude of the rotation information corresponds to the magnitude of the first eigenvector after the flip.

8. The facial image generation method according to claim 7, wherein, The steps for determining auxiliary feature vectors based on rotation information include: Transform rotation information to indicate the opposite direction of rotation; and The auxiliary feature vector is determined based on the transformed rotation information.

9. The facial image generation method according to claim 1, wherein, The rotation information also includes information indicating the degree of rotation used to generate the output image from the input image.

10. The facial image generation method according to claim 9, wherein, Rotation information is determined by comparing the positional information of the main facial features in the input image with the preset facial pose in the output image.

11. The facial image generation method according to claim 1, wherein, The steps for generating the output image also include: The output image is passed as the input image for subsequent iterations; and Based on the rotation degree included in the rotation information, subsequent iterations are performed, including determining a first feature vector and a second feature vector, determining a flipped first feature vector, determining an auxiliary feature vector, determining a final feature vector, and generating an output image.

12. The facial image generation method according to claim 1, wherein, The steps for encoding the input image include: encoding the input image using an encoder, and The encoder's neural network includes: The input layer corresponds to the input image; and The output layer corresponds to the first and second feature vectors.

13. The facial image generation method according to claim 1, wherein, The steps for decoding the final eigenvector and the second eigenvector include: using a decoder to decode the final eigenvector and the second eigenvector based on rotation information, and The decoder's neural network includes: The input layer corresponds to the final feature vector, the second feature vector, and rotation information; and The output layer corresponds to the output image.

14. The facial image generation method according to claim 1, wherein, The steps to determine the first flipped eigenvector include: The flipped input image is determined by flipping the input image relative to the axis in the corresponding space; and By encoding the flipped input image, the first feature vector of the flip, which is related to the pose, is determined.

15. The facial image generation method according to any one of claims 1 to 14, further comprising: Features for facial recognition are extracted from the output image; and Face recognition is based on extracted features.

16. The facial image generation method according to any one of claims 1 to 14, further comprising: By changing the degree of rotation included in the rotation information, multiple output images corresponding to the input image are generated. and Face recognition is based on multiple generated output images.

17. The facial image generation method according to any one of claims 1 to 14, further comprising: Based on the output image and the target image corresponding to the input image, a neural network for the encoder is trained, a neural network for the decoder is trained, and a neural network for determining auxiliary feature vectors is trained.

18. A method for generating a facial image, comprising: By applying an input image including the face to the encoder, a first feature vector related to pose and a second feature vector related to identity are determined; The first eigenvector is determined by flipping the first eigenvector relative to the axis in the corresponding space; Auxiliary feature vectors are determined by applying the flipped first feature vector and rotation information corresponding to the input image to the first neural network. The final feature vector is determined based on the first feature vector and the auxiliary feature vector; By applying the final feature vector, the second feature vector, and rotation information to the decoder, an output image of the face, including rotation, is generated; and Based on the output image and the target image corresponding to the input image, train the encoder's neural network, the decoder's neural network, and the first neural network. The step of determining the first flipped feature vector includes: relative to the vertical axis in the corresponding space, determining the first flipped feature vector by moving the elements in each channel of the first feature vector to the elements in each channel of the flipped first feature vector.

19. The facial image generation method according to claim 18, wherein, The training steps include: Training is performed based on a loss, which includes any one or any combination of the following losses: The loss is generated by comparing features extracted from the output image with features extracted from the target image; The loss is generated by comparing the output image and the target image pixel by pixel; The loss is generated by comparing the identity of the output image with the identity of the target image; Loss based on the difference in pixel values ​​between pixels in the output image; and Generative adversarial network loss based on output and target images.

20. A facial image generation device, comprising: One or more processors are configured as follows: By encoding the input image, including the face, a first feature vector related to pose and a second feature vector related to identity are determined. The first eigenvector is determined by flipping the first eigenvector relative to the axis in the corresponding space; Based on the first flipped feature vector and the rotation information corresponding to the input image, an auxiliary feature vector is determined. The final feature vector is determined based on the first feature vector and the auxiliary feature vector; and By decoding the final feature vector and the second feature vector based on rotation information, an output image of the rotated face is generated. The step of determining the first flipped feature vector includes: relative to the vertical axis in the corresponding space, determining the first flipped feature vector by moving the elements in each channel of the first feature vector to the elements in each channel of the flipped first feature vector.

21. The facial image generation device according to claim 20, wherein, To determine the auxiliary feature vector, the one or more processors are configured to: Auxiliary feature vectors are determined by applying the first flipped feature vector, the positional information of the main facial features in the input image corresponding to the size of the first flipped feature vector, and the rotation information corresponding to the size of the first flipped feature vector to the convolutional neural network.

22. The facial image generation device according to claim 20, wherein, The one or more processors are configured to: To determine the first and second feature vectors, the location information of the main facial features in the input image is determined from the input image, and the first and second feature vectors are determined by encoding the location information and the input image; and To determine the auxiliary feature vector, the position information is transformed by flipping the position information relative to the axis in the corresponding space and adjusting the size of the position information to correspond to the size of the flipped first feature vector. The auxiliary feature vector is then determined based on the flipped first feature vector, the rotation information corresponding to the input image, and the transformed position information.

23. The facial image generation device according to claim 20, wherein, Rotation information includes information indicating the rotation direction and degree of rotation used to generate the output image from the input image, and The magnitude of the rotation information corresponds to the magnitude of the first eigenvector after the flip.

24. The facial image generation device according to claim 20, wherein, In order to generate the output image, the one or more processors are configured to: Transmit the output image as the input image for subsequent iterations; and Based on the rotation degree included in the rotation information, subsequent iterations are performed to determine the first and second feature vectors, determine the flipped first feature vector, determine the auxiliary feature vector, determine the final feature vector, and generate the output image.

25. The facial image generation device according to claim 20, wherein, In order to encode the input image, the one or more processors are configured to: encode the input image using an encoder, and The encoder's neural network includes: The input layer corresponds to the input image; and The output layer corresponds to the first and second feature vectors.

26. The facial image generation device according to claim 20, wherein, To decode the final feature vector and the second feature vector, the one or more processors are configured to: decode the final feature vector and the second feature vector using a decoder based on rotation information, and The decoder's neural network includes: The input layer corresponds to the final feature vector, the second feature vector, and rotation information; and The output layer corresponds to the output image.

27. The facial image generation device according to claim 20, wherein, To determine the first flipped feature vector, the one or more processors are configured to: The flipped input image is determined by flipping the input image relative to the axis in the corresponding space; and By encoding the flipped input image, the first feature vector of the flip, which is related to the pose, is determined.

28. The facial image generation apparatus according to any one of claims 20 to 27, wherein, The one or more processors are configured to: Features for facial recognition are extracted from the output image; and Face recognition is based on extracted features.

29. A method for generating a facial image, comprising: Based on the input image including the face, a first feature vector related to pose is determined; The symmetric transformation of the first eigenvector is determined as the flipped eigenvector; Auxiliary feature vectors are determined based on the flipped feature vectors and the rotation information corresponding to the input image. The final feature vector is determined based on the first feature vector and the auxiliary feature vector; and Based on the final feature vector and rotation information, an output image of the face, including its rotation, is generated. The step of determining the symmetric transformation of the first feature vector as the flipped feature vector includes: relative to the vertical axis in the corresponding space, by moving the elements in each channel of the first feature vector to the elements in each channel of the flipped first feature vector to determine the symmetric transformation of the first feature vector as the flipped feature vector.

30. A non-transitory computer-readable storage medium storing instructions, which, when executed by one or more processors, configure the one or more processors to perform the facial image generation method according to any one of claims 1 to 19 and 29.

Images