Face image generation method and device, electronic equipment and storage medium

By mixing the face recognition model and feature fusion model, extracting and replacing face features, the problem of low similarity in AI face swaps is solved, and the accuracy and robustness of face image generation is improved.

CN119068529APending Publication Date: 2024-12-03GUANGZHOU HUYA INFORMATION TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411152037.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, face recognition models in AI face change applications have low similarity in face change images due to the lack of detailed features, and cannot effectively improve the face change effect.

Method used

The face recognition model is trained using discrete face diagrams and continuous video frames with multiple face posture angles. The face recognition model is used to extract target features and input feature fusion model, and face replacement is performed on face replacement images to generate target faces. faces generate images.

Benefits of technology

It significantly improves the robustness and accuracy of face image generation, reduces the distortion of face generated images, and improves the face replacement effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068529A_ABST
    Figure CN119068529A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a face image generation method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. According to the method, the face recognition model is trained in a mixed mode through discrete face images of various face attitude angles and continuous video frames, and the accuracy of feature extraction of the face recognition model is effectively improved. And inputting the target features extracted by using the face recognition model into the feature fusion model, and performing face replacement on the image to be subjected to face replacement to obtain the face generation image containing the target face, thereby reducing the distortion of the face generation image, facilitating the improvement of the face replacement effect of the face image generation model, and improving the face replacement efficiency. And therefore, the robustness and the accuracy of face image generation can be obviously improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for generating a face image. Background Art

[0002] In the Internet live broadcast industry, with the popularity of live short videos and the continuous advancement of artificial intelligence (AI) technology, using AI technology to optimize live content and increase the diversity of live content has become a hot trend. Among them, AI face-changing technology is widely used in the live broadcast industry, social media and other fields.

[0003] In the prior art, face recognition models usually focus on extracting key features from face images so as to identify and classify based on the key features. When face recognition models are directly applied to AI face-changing, there is a problem of low similarity of face-changing images due to the lack of detailed features. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a facial image generation method, device, electronic device and storage medium, which can effectively improve the face-changing effect.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, the present invention provides a method for generating a face image, the method comprising:

[0007] Extracting a target face from a target face image;

[0008] Inputting the face-swapped image and the target face into a pre-trained face image generation model; the face image generation model includes a face recognition model and a feature fusion model; the face recognition model is obtained by mixed training using discrete face images and videos corresponding to a variety of face posture angles;

[0009] Extracting features of the target face using the face recognition model, and inputting the extracted target features into the feature fusion model;

[0010] The feature fusion model is used to perform face replacement on the image to be replaced based on the target features to obtain a generated face image.

[0011] In an optional implementation, the face recognition model is obtained by:

[0012] Obtain a discrete face image set, a video set and a recognition model to be trained; the discrete face image set includes multiple discrete face images of different faces; the video set includes at least one video of different faces; the sampling coefficient of the discrete face image is determined according to the face posture angle of the discrete face image;

[0013] For each human face, a discrete face image whose sampling coefficient exceeds a preset sampling threshold is determined as a target image;

[0014] Determining a corresponding first scaling factor according to the face posture angle of the target image;

[0015] Inputting the target image into the recognition model to be trained to obtain a first feature vector;

[0016] Obtaining first loss information according to the first feature vector, the face label, and the first scaling factor corresponding to the target image;

[0017] Iteratively updating the parameters of the recognition model to be trained according to the first loss information to obtain an initial recognition model;

[0018] The initial recognition model is iteratively trained using each of the videos to obtain a face recognition model.

[0019] In an optional implementation, the face posture angle includes a pitch angle and a yaw angle; and determining the corresponding first scaling factor according to the face posture angle of the target image includes:

[0020] A first scaling factor corresponding to the target image is obtained according to a preset angle threshold, an initial scaling value, and a pitch angle and a yaw angle corresponding to the target image.

[0021] In an optional implementation, obtaining first loss information according to the first feature vector, the face label, and the first scaling factor corresponding to the target image includes:

[0022] determining an angular boundary according to the first scaling factor;

[0023] The first loss information is determined according to the first feature vector, the face label and the angle boundary.

[0024] In an optional implementation, the iterative training of the initial recognition model using each of the videos to obtain a face recognition model includes:

[0025] Determine the face video frame whose sampling coefficient exceeds the preset sampling threshold in each video as the target video frame; the sampling coefficient of the face video frame is determined according to the face posture angle of the face video frame;

[0026] Determining a corresponding second scaling factor according to the face posture angle of the target video frame;

[0027] Inputting the target video frame into the initial recognition model to obtain a second feature vector;

[0028] Obtaining second loss information according to the second feature vector, the face label, and the second scaling factor corresponding to the target video frame;

[0029] Iteratively update the parameters of the initial recognition model according to the second loss information to obtain a face recognition model.

[0030] In an optional implementation, the face posture angle includes a pitch angle and a yaw angle; before obtaining the discrete face image set, the video set and the recognition model to be trained, the method further includes:

[0031] Extract frames from each of the videos to obtain face video frames;

[0032] Sequentially taking each of the discrete face images and each of the face video frames as images to be processed;

[0033] Inputting each of the images to be processed into a preset facial key point model to obtain the pitch angle and yaw angle of the face in the image to be processed;

[0034] Determining an original sampling coefficient of the image to be processed according to the pitch angle, the yaw angle, the preset angle threshold and the preset sampling value corresponding to each image to be processed;

[0035] Based on the same human face, normalizing the original sampling coefficients of the discrete face images to obtain the sampling coefficients corresponding to the discrete face images;

[0036] Based on the same video, the original sampling coefficients of the face video frames are normalized to obtain the sampling coefficients corresponding to the face video frames.

[0037] In an optional implementation, the feature fusion model is trained in the following manner:

[0038] Acquire a training image set; the training image set includes multiple groups of template face images and sample face images;

[0039] Extract a template face from a template face image, and input the template face into a pre-trained face recognition model to obtain template face features;

[0040] Inputting the sample face image and the template face features into the feature fusion model to be trained to obtain a sample face-changing image;

[0041] Determine third loss information according to the facial features in the sample face-swapped image and the facial features of the template;

[0042] Iteratively update the parameters of the feature fusion model to be trained according to the third loss information to obtain a feature fusion model.

[0043] In a second aspect, the present invention provides a facial image generating device, the device comprising:

[0044] A preprocessing module, used for extracting a target face from a target face image;

[0045] The replacement module is used to input the image to be replaced and the target face into a pre-trained face image generation model; the face image generation model includes a face recognition model and a feature fusion model; the face recognition model is obtained by mixed training using discrete face images and videos corresponding to a variety of face posture angles; the face recognition model is used to extract features of the target face, and the extracted target features are input into the feature fusion model; the feature fusion model is used to perform face replacement on the image to be replaced based on the target features to obtain a face generation image.

[0046] In a third aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the facial image generation method described in any of the aforementioned embodiments.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for generating a facial image as described in any one of the aforementioned embodiments is implemented.

[0048] Compared with the prior art, the face image generation method, device, electronic device and storage medium provided by the embodiments of the present invention utilize discrete face images of multiple face posture angles and continuous video frames to mix and train the face recognition model, thereby effectively improving the accuracy of feature extraction of the face recognition model. The target features extracted by the face recognition model are input into the feature fusion model, and the face replacement is performed on the face-changing image to obtain a face generation image containing the target face, thereby reducing the distortion of the face generation image, helping to improve the face replacement effect of the face image generation model, and thus significantly improving the robustness and accuracy of face image generation.

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic flow chart of a method for generating a facial image provided by an embodiment of the present invention is shown.

[0052] Figure 2 Another schematic flow chart of the method for generating a facial image provided by an embodiment of the present invention is shown.

[0053] Figure 3 Shows Figure 2 A flowchart of the sub-steps of step S260.

[0054] Figure 4 Another schematic flow chart of the method for generating a facial image provided by an embodiment of the present invention is shown.

[0055] Figure 5 Another schematic flow chart of the method for generating a facial image provided by an embodiment of the present invention is shown.

[0056] Figure 6 A block diagram of a facial image generating device provided by an embodiment of the present invention is shown.

[0057] Figure 7 A block diagram of an electronic device provided by an embodiment of the present invention is shown.

[0058] Icon: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication module; 500 - face image generating device; 501 - preprocessing module; 502 - replacement module. DETAILED DESCRIPTION

[0059] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0061] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0062] With the rapid development of the Internet and artificial intelligence technology, AI technology has been widely used to optimize live broadcast content to improve the diversity of live broadcast content. In this field, AI technology is applied in a variety of ways, including special effects support based on facial key points and segmentation, image stylization based on Generative Adversarial Networks (GAN), etc.

[0063] Among them, AI face-changing technology is one of these applications. In order to make AI face-changing technology improve the similarity of the face after changing, it is usually constrained by open source face recognition models. However, the data used by open source face recognition models usually include a lot of open source data. Usually, open source data will include photos at different time stages and with different makeup, and the degree of difference in these photos makes it difficult to recognize.

[0064] The inventors have found that face recognition technology is a technology that authenticates identity based on a person's facial feature information. The main research directions currently cover traditional machine learning and deep learning methods. In deep learning, Convolutional Neural Networks (CNN) is one of the commonly used technologies. It extracts high-dimensional facial features in a hierarchical manner to achieve high-precision recognition. Research in this field aims to construct a metric space so that images of the same person are closer and images of different people are farther apart. Initially, contrastive learning and contrastive loss methods were mainly used to improve the metric space. Contrast loss helps to narrow the distance between different images of the same person and expand the distance between different people. The core is to select appropriate positive and negative sample pairs and adjust the distance metric (such as Euclidean or cosine distance).

[0065] In ArcFace technology, the concept of angle margin is introduced into the softmax loss to achieve the compactness of samples of the same type in the feature space and the dispersion of samples of different types. This strengthens the distinction between categories by limiting the angle, thereby improving recognition accuracy, and on this basis, a variety of loss functions are developed to optimize the learning space.

[0066] In the field of face recognition, the main purpose is to identify whether an image belongs to the target ID. Its main purpose is to improve the recognition accuracy and efficiency in complex environments, that is, to improve the recognition accuracy under different lighting, postures, and occlusion conditions. This is different from the face-changing algorithm. The features of face recognition are mainly designed for recognition tasks.

[0067] Face-changing technology mainly uses deep learning methods, especially two key technologies: GAN and autoencoder (AE). In the GAN model, there are two parts: the generator and the discriminator: the generator is used to create realistic images, and the discriminator is responsible for judging whether the image is generated by the generator. Through training, the naturalness and realism of the image are continuously improved to confuse the discriminator. The autoencoder compresses the image to low-dimensional features through the encoder, and reconstructs the image through the decoder to complete the transformation of facial features.

[0068] In addition, there are methods based on deep neural networks (DNNs), which use massive data training to directly edit faces at the pixel level to achieve more accurate and natural results. With the development of technology, hybrid methods that combine image segmentation, 3D modeling and other technologies are also emerging, aiming to further enhance the practicality and realism of face-changing technology. The controllable generation of identity IDs based on Stable Diffusion usually introduces identity ID features as feature injection to influence generation, and adds ID adapters to improve the similarity of generated images to identity IDs.

[0069] In generation tasks such as AI face-changing, it is usually expected that the replaced face is more similar to the target face. This is different from the tendency in the recognition field, where there is no special design and consideration for the controllable generation of identity IDs. In addition, how to constrain the similarity at large angles and improve the similarity of side faces is also an issue that needs to be optimized.

[0070] Based on this, the face image generation method and device provided by the embodiment of the present invention utilizes discrete face images of multiple face posture angles and continuous video frames to mix and train the face recognition model, effectively improving the accuracy of feature extraction of the face recognition model. The target features extracted by the face recognition model are input into the feature fusion model, and the face replacement is performed on the face-changing image to obtain a face generation image containing the target face, thereby reducing the distortion of the face generation image, helping to improve the face replacement effect of the face image generation model, and thus significantly improving the robustness and accuracy of face image generation.

[0071] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0072] Please refer to Figure 1 , Figure 1 A schematic flow chart of a method for generating a face image provided by an embodiment of the present invention is shown. The method should include the following steps:

[0073] Step S100, extracting a target face from a target face image.

[0074] In an embodiment of the present invention, a preset face segmentation model and a pre-trained face image generation model are recorded on the electronic device. The target face image is input into the preset face segmentation model to obtain the target face. The target face is the face image in the target face image.

[0075] Step S110, input the face-swapped image and the target face into a pre-trained face image generation model; the face image generation model includes a face recognition model and a feature fusion model; the face recognition model is obtained by mixed training using discrete face images and videos corresponding to a variety of face posture angles.

[0076] In the embodiment of the present invention, the face image generation model includes a pre-trained feature fusion model and a face recognition model. The face recognition model is used to extract features from the image; the feature fusion model is used to replace the face features in the face-changing image with the target face.

[0077] Step S120, using the face recognition model to extract features of the target face, and inputting the extracted target features into the feature fusion model.

[0078] Step S130, using the feature fusion model to perform face replacement on the image to be replaced based on the target features to obtain a generated face image.

[0079] In the embodiment of the present invention, the target face is input into the face recognition model to extract features to obtain the target features. The target features and the image to be replaced are input into the feature fusion model, and the facial features in the image to be replaced are replaced with the target features to obtain the generated face image.

[0080] In summary, the face image generation method provided by the embodiment of the present invention utilizes a discrete face image of multiple face posture angles and a continuous video frame to mix and train a face recognition model, thereby effectively improving the accuracy of feature extraction of the face recognition model. The target features extracted by the face recognition model are input into the feature fusion model, and the face replacement is performed on the face-changing image to obtain a face generation image containing the target face, thereby reducing the distortion of the face generation image, helping to improve the face replacement effect of the face image generation model, and further significantly improving the robustness and accuracy of face image generation.

[0081] Optionally, before using the face recognition model for face swapping, the recognition model to be trained needs to be trained to obtain the face recognition model. The following is an explanation of the training process of the face recognition model. Figure 2 , Figure 2 Another flow chart of the method for generating a face image provided by an embodiment of the present invention is shown. The face recognition model is obtained by:

[0082] Step S200, obtaining a discrete face image set, a video set and a recognition model to be trained; the discrete face image set includes multiple discrete face images of different faces; the video set includes at least one video of different faces; the sampling coefficient of the discrete face image is determined according to the face posture angle of the discrete face image.

[0083] In an embodiment of the present invention, the recognition model to be trained may be a residual network (e.g., ResNet101, ResNet50), a MobileNet or other deep neural network. An identity ID may be used to identify a face, and each face corresponds to an identity ID. Multiple discrete face images of different faces are collected, and the multiple discrete face images of each face are saved in a separate directory according to the identity ID. One or more videos of different faces are collected.

[0084] When training a face recognition model, you can select open source face images of the face recognition model, which can include discrete face images and videos of different faces in different stages and different works. You can also use face images that have obtained the right to use the portrait. This is not limited in the embodiments of the present invention.

[0085] Step S210: for each human face, a discrete face image whose sampling coefficient exceeds a preset sampling threshold is determined as a target image.

[0086] In the embodiment of the present invention, each discrete face image has a corresponding sampling coefficient, which is calculated based on the face posture angle in the discrete face image, and is used to characterize the importance of the discrete face image for training the face recognition model. The preset sampling threshold is initialized based on the actual application scenario and experience, and the discrete face image with a sampling coefficient greater than the preset sampling threshold is used as the target image, and the face recognition model is trained using the target image.

[0087] Step S220, determining a corresponding first scaling factor according to the face posture angle of the target image.

[0088] Optionally, as a possible implementation, the face posture angle includes a pitch angle and a yaw angle. According to a preset angle threshold, an initial zoom value, and the pitch angle and yaw angle corresponding to the target image, a first zoom factor corresponding to the target image is obtained.

[0089] In the embodiment of the present invention, the pitch angle and yaw angle of the target image, as well as the preset angle threshold and the initial zoom value are obtained, and the zoom factor corresponding to the target image, that is, the first zoom factor, is calculated according to the pitch angle and yaw angle of the target image, as well as the preset angle threshold and the initial zoom value. The calculation formula of the first zoom factor is as follows:

[0090]

[0091] Among them, P is the first scaling factor; patch is the pitch angle; yaw is the yaw angle; threshold is the preset angle threshold; γ is the initial scaling value; |.| represents the absolute value; Max is the maximum value function; Min is the minimum value function.

[0092] Step S230: input the target image into the recognition model to be trained to obtain a first feature vector.

[0093] Step S240, obtaining first loss information according to the first feature vector, the face label and the first scaling factor corresponding to the target image.

[0094] Step S250, iteratively updating the parameters of the recognition model to be trained according to the first loss information to obtain an initial recognition model.

[0095] In an embodiment of the present invention, the recognition model to be trained extracts features from the target image to obtain a first feature vector, calculates first loss information using the first feature vector corresponding to the target image, a face label and a first scaling factor, and iteratively updates the parameters of the recognition model to be trained using the first loss information until the first loss information meets the convergence condition to obtain an initial recognition model.

[0096] It should be noted that the face recognition model improves the accuracy of feature extraction through continuous training, that is, the first feature vector belonging to the same face label is mapped to data close to the center of the category, and the first feature vector not belonging to the same face label is mapped to data far from the center of the category.

[0097] Step S260, iteratively train the initial recognition model using each video to obtain a face recognition model.

[0098] In an embodiment of the present invention, videos of different periods and different makeup can be obtained to perform detailed training on the initial recognition model. The initial recognition model is trained by continuous video frames of makeup and period in the video, further improving the accuracy of feature extraction of the face recognition model.

[0099] It can be seen that the embodiment of the present invention uses a discrete face image containing side face posture angles to perform initial training on the face recognition model, which is beneficial for the face recognition model to learn to extract upper-level features of the side face image, thereby improving the accuracy of the face recognition model in extracting side face features. The face recognition model is then finely trained in combination with the video to further improve the accuracy of the face recognition model's feature extraction.

[0100] Optionally, in practical applications, the pitch angle and yaw angle of the face in the image may be used to dynamically adjust the angle boundary. Figure 2 The sub-steps of step S240 may include:

[0101] An angle boundary is determined according to a first scaling factor; and first loss information is determined according to a first eigenvector, a face label and an angle boundary.

[0102] In an embodiment of the present invention, the angle boundary margin is used to control the parameters of the interval between categories. During the training stage, if features with inconsistent recognition are forcibly pulled to the center of the category with the same angle boundary, part of the information in the original feature space will be lost. In order to reduce feature loss, when the pitch angle and yaw angle do not exceed the preset angle threshold, the larger the pitch angle or yaw angle, the smaller the first scaling factor, and the smaller the angle boundary margin. In other words, the larger the pitch angle or yaw angle, the angle boundary margin can release the scaling. The feature vector of the side face can be equivalent to the feature vector of the front face, and the feature vector is farther away from the center of the category.

[0103] It should be noted that when the pitch angle and the yaw angle do not exceed the preset angle threshold, the first scaling factor is proportional to the pitch angle or the yaw angle, and the first scaling factor is proportional to the angle boundary. The preset angle boundary value of the recognition model to be trained is obtained, the product of the first scaling factor and the preset angle boundary value is determined as the angle boundary, and the first loss information is determined using the first eigenvector, the face label and the angle boundary.

[0104] It can be seen that the embodiment of the present invention dynamically adjusts the angle boundary according to the pitch angle and yaw angle of the face in the training image, and calculates the first loss information based on the adjusted angle boundary, so as to speed up the training speed of the face recognition model and improve the accuracy of the face recognition model.

[0105] Optionally, in practical applications, the fine-grained feature vectors in the video can be used to fine-tune the face recognition model to obtain a face recognition model with higher feature extraction accuracy. Figure 3 , Figure 2 The sub-steps of step S260 may include:

[0106] Step S261, determining the face video frame whose sampling coefficient exceeds the preset sampling threshold in each video as the target video frame; the sampling coefficient of the face video frame is determined according to the face posture angle of the face video frame.

[0107] In the embodiment of the present invention, the video is subjected to frame sampling processing, and video frames containing the same face are determined as face video frames, and each face video frame corresponds to a sampling coefficient. According to the high-low order of the sampling coefficients of the face video frames, the face video frames whose sampling coefficients are higher than a preset sampling threshold are determined as target video frames. The target video frames are used to perform fine training on the initial recognition model.

[0108] Step S262, determining a corresponding second scaling factor according to the face posture angle of the target video frame.

[0109] Step S263: input the target video frame into the initial recognition model to obtain a second feature vector.

[0110] Step S264, obtaining second loss information according to the second feature vector, the face label and the second scaling factor corresponding to the target video frame.

[0111] Step S265, iteratively update the parameters of the initial recognition model according to the second loss information to obtain a face recognition model.

[0112] In an embodiment of the present invention, the method for determining the second scaling coefficient, the second eigenvector, and the second loss information is the same as the method for determining the first scaling coefficient, the first eigenvector, and the first loss information. The method for determining the first scaling coefficient, the first eigenvector, and the first loss information may be referred to and will not be repeated here.

[0113] It should be noted that the classification method of the face recognition model before training for the same face is somewhat different from the expected classification of face swapping. The classification of the face recognition model before training focuses on dividing the face into target categories. For example, if a person has a special feature such as a teardrop mole on his face, the face recognition model can classify the face based on this feature without paying more attention to other features on the face. As a result, the face recognition model does not learn other features of the face well and the extracted features are inaccurate. If the face recognition model before training is directly applied to controllable ID generation (such as face swapping), the similarity between the face after face swapping and the target face is low.

[0114] In the task of controllable ID generation (such as face swapping), it is usually expected that the generated image retains more features of the target face, that is, it is more like the reference image provided by the user rather than the bone structure of the identity ID. The embodiment of the present invention utilizes the diversity of discrete face images to improve the accuracy of extracting bone structure features of the face recognition model, such as facial features proportions, especially tear moles, etc.

[0115] However, discrete face images may be images of the same face in different stages and works, with large differences, especially photos of the same person spanning multiple age stages from childhood to adulthood. The fine-grained features in discrete face images vary greatly, and it is difficult for the face recognition model to ensure the accuracy of extracting fine-grained features. Therefore, the embodiment of the present invention introduces a video training face recognition model, and through a video frame processing link, the face video frames of the same identity ID (i.e., face) in the video are saved in the same directory, and the face video frames of the same person in different videos (such as different makeup) are saved in different directories. The face video frames in each directory are used to train the face recognition model, thereby ensuring that the face recognition model is finely trained through the video.

[0116] It can be seen that the embodiment of the present invention uses discrete face images to improve the face recognition model's ability to recognize upper-level facial features (such as bone features such as the proportion of facial features), and then uses videos to improve the face recognition model's ability to recognize fine-grained facial features (such as detailed features such as makeup), thereby ensuring that the accuracy of facial feature extraction of the trained face recognition model is effectively improved.

[0117] Optionally, in practical applications, the face posture angles include pitch angle and yaw angle, and the sampling coefficient of the training image is dynamically adjusted according to the pitch angle and yaw angle to ensure that the training images with sampling coefficients that meet the requirements are used as much as possible for training. Figure 4 , before training the face recognition model, the face image generation method also includes the following steps:

[0118] Step S300, extract frames from each video to obtain face video frames.

[0119] In the embodiment of the present invention, each video is decoded, the video is parsed into images frame by frame, and the image including the face is determined as the face video frame.

[0120] Step S310: taking each discrete face image and each face video frame as an image to be processed in sequence.

[0121] In the embodiment of the present invention, the calculation process of the original sampling coefficients of the discrete face image is the same as the calculation process of the original sampling coefficients of the face video frame. Therefore, the discrete face image and the face video frame are uniformly described as the images to be processed.

[0122] Step S320: Input each image to be processed into a preset facial key point model to obtain the pitch angle and yaw angle of the face in the image to be processed.

[0123] In the embodiment of the present invention, the preset facial key point model can locate facial feature points of the face in the image to be processed, such as eyes, nose and mouth, etc. The facial posture angles, including pitch angle and yaw angle, can be estimated according to the facial feature points.

[0124] Step S330, determining the original sampling coefficient of the image to be processed according to the pitch angle, yaw angle, preset angle threshold and preset sampling value corresponding to each image to be processed.

[0125] As a possible implementation, the calculation formula of the original sampling coefficient is as follows:

[0126]

[0127] Among them, P1 is the original sampling coefficient; patch is the pitch angle; yaw is the yaw angle; threshold is the preset angle threshold; β is the preset sampling value; Max is the maximum value function; Min is the minimum value function.

[0128] Step S340, based on the same human face, normalize the original sampling coefficients of the discrete face images to obtain sampling coefficients corresponding to each discrete face image.

[0129] In the embodiment of the present invention, for each human face, that is, for each discrete face image of an identity ID, the original sampling coefficients of the discrete face images of the same human face are normalized to obtain the sampling coefficients corresponding to the discrete face image.

[0130] Step S350, based on the same video, normalize the original sampling coefficients of the face video frames to obtain the sampling coefficients corresponding to each face video frame.

[0131] In the embodiment of the present invention, for each video, the original sampling coefficients of each face video frame in the same video are normalized to obtain the sampling coefficient corresponding to the face video frame. The sampling coefficient can be a decimal or fraction between 0 and 1.

[0132] It is worth mentioning that there is a certain difference in the recognition degree of the front face and the side face of the same person. Usually, the side face will have a certain lack of recognizability because it hides some features of the whole face. Therefore, the embodiment of the present invention uses side face images as much as possible to train the face recognition model to improve the face recognition model's recognition ability for the front face and the side face.

[0133] It can be seen that the embodiment of the present invention utilizes the pitch angle and yaw angle of the face to dynamically adjust the sampling coefficients of the discrete face image and the face video frame. The larger the pitch angle or yaw angle, the larger the sampling coefficient. The sampling probability of the side face image is increased by the pitch angle and yaw angle. However, considering that when the pitch angle or yaw angle is greater than a certain value, the similarity loss will be caused due to oversampling, a preset angle threshold is set according to the actual application scenario or experience, and the sampling coefficient is no longer increased after it is greater than the threshold. This ensures that the learning probability of the face recognition model for the side face image is improved, thereby improving the accuracy of the face recognition model in extracting features for the side face image.

[0134] Optionally, before using the feature fusion model for face swapping, the feature fusion model to be trained needs to be trained to obtain a trained feature fusion model. The following is an explanation of the training process of the feature fusion model. Figure 5 , Figure 5 Another schematic diagram of a process of generating a face image according to an embodiment of the present invention is shown. The feature fusion model is obtained by:

[0135] Step S400, obtaining a training image set; the training image set includes multiple groups of template face images and sample face images.

[0136] In the embodiment of the present invention, each sample face image corresponds to a template face image, and different sample face images may correspond to the same template face image or to different template face images. For example, the faces of Xiao A and Xiao B are replaced with the face of Xiao C, and the face of Xiao D is replaced with the face of Xiao E. At this time, the template face images corresponding to the sample face image of Xiao A and the sample face image of Xiao B are both the template face images of Xiao C. The sample face image of Xiao D corresponds to the template face image of Xiao E.

[0137] Step S410, extracting a template face from the template face image, and inputting the template face into a pre-trained face recognition model to obtain template face features.

[0138] In the embodiment of the present invention, before training the feature fusion model, it is necessary to pre-train the face recognition model, and perform feature extraction on the template face based on the pre-trained face recognition model to obtain the template face features.

[0139] Step S420, inputting the sample face image and the template face features into the feature fusion model to be trained to obtain a sample face-changing image.

[0140] Step S430, determining third loss information according to the facial features in the sample face-swapped image and the facial features of the template.

[0141] Step S440, iteratively update the parameters of the feature fusion model to be trained according to the third loss information to obtain the feature fusion model.

[0142] In an embodiment of the present invention, a face-swapping process is performed based on a sample face image and a template face feature using a feature fusion model to be trained to obtain a sample face-swapping image. The sample face-swapping image is input into a pre-trained face recognition model to obtain a sample face-swapping feature, and the third loss information is determined according to the similarity between the sample face-swapping feature and the template face feature. The similarity can be determined by a similarity algorithm such as Euclidean distance and cosine similarity.

[0143] It can be seen that the embodiment of the present invention iteratively trains the feature fusion model based on the pre-trained face recognition model, and uses the pre-trained face recognition model to extract the fine-grained features of the face to improve the accuracy of the feature vector corresponding to the face. Then, the face in the sample face-changing image is changed based on the template face features with high accuracy, and finally, the parameters of the feature fusion model to be trained are iteratively updated based on the difference between the face features after the face change and the template face features to obtain the trained feature fusion model. The trained feature fusion model has a good face-changing effect, can ensure that the similarity between the face-changing image and the template face is close, and effectively improves the face-changing effect of the side face image.

[0144] Based on the same inventive concept, the embodiment of the present invention also provides a facial image generation device. Its basic principle and technical effects are the same as those of the above embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents of the above embodiment.

[0145] Please refer to Figure 6 , Figure 6 FIG. 5 is a block diagram of a facial image generation device 500 provided by an embodiment of the present invention. The facial image generation device 500 includes a preprocessing module 501 and a replacement module 502 .

[0146] A preprocessing module 501 is used to extract a target face from a target face image;

[0147] The replacement module 502 is used to input the image to be replaced and the target face into a pre-trained face image generation model; the face image generation model includes a face recognition model and a feature fusion model; the face recognition model is obtained by mixed training of discrete face images and videos corresponding to a variety of face posture angles; the face recognition model is used to extract features of the target face, and the extracted target features are input into the feature fusion model; the feature fusion model is used to replace the face of the image to be replaced based on the target features to obtain a face generation image.

[0148] In summary, the face image generation device provided by the embodiment of the present invention uses a discrete face image of multiple face posture angles and a continuous video frame to mix and train a face recognition model, thereby effectively improving the accuracy of feature extraction of the face recognition model. The target features extracted by the face recognition model are input into the feature fusion model, and the face replacement is performed on the face-changing image to obtain a face generation image containing the target face, thereby reducing the distortion of the face generation image, helping to improve the face replacement effect of the face image generation model, and further significantly improving the robustness and accuracy of face image generation.

[0149] Optionally, the preprocessing module 501 is also used to obtain a discrete face image set, a video set and a recognition model to be trained; the discrete face image set includes multiple discrete face images of different faces; the video set includes at least one video of different faces; the sampling coefficient of the discrete face image is determined according to the facial posture angle of the discrete face image; for each face, the discrete face image whose sampling coefficient exceeds a preset sampling threshold is determined as the target image; the corresponding first scaling coefficient is determined according to the facial posture angle of the target image; the target image is input into the recognition model to be trained to obtain a first feature vector; according to the first feature vector corresponding to the target image, the face label and the first scaling coefficient, the first loss information is obtained; according to the first loss information, the parameters of the recognition model to be trained are iteratively updated to obtain an initial recognition model; the initial recognition model is iteratively trained using each video to obtain a face recognition model.

[0150] Optionally, the facial posture angle includes a pitch angle and a yaw angle, and the preprocessing module 501 is specifically used to obtain a first scaling factor corresponding to the target image according to a preset angle threshold, an initial scaling value, and a pitch angle and a yaw angle corresponding to the target image.

[0151] Optionally, the preprocessing module 501 is specifically configured to determine an angle boundary according to a first scaling factor; and determine first loss information according to the first eigenvector, the face label and the angle boundary.

[0152] Optionally, the preprocessing module 501 is specifically used to determine the face video frames in each video whose sampling coefficient exceeds a preset sampling threshold as target video frames; the sampling coefficient of the face video frames is determined according to the face posture angle of the face video frames; the corresponding second scaling coefficient is determined according to the face posture angle of the target video frames; the target video frames are input into the initial recognition model to obtain a second feature vector; according to the second feature vector, face label and second scaling coefficient corresponding to the target video frames, second loss information is obtained; and the parameters of the initial recognition model are iteratively updated according to the second loss information to obtain a face recognition model.

[0153] Optionally, the facial posture angle includes a pitch angle and a yaw angle, and the preprocessing module 501 is also used to extract frames from each video to obtain a facial video frame; each discrete face image and each facial video frame is used as an image to be processed in turn; each image to be processed is input into a preset facial key point model to obtain the pitch angle and yaw angle of the face in the image to be processed; the original sampling coefficient of the image to be processed is determined according to the pitch angle, yaw angle, preset angle threshold and preset sampling value corresponding to each image to be processed; based on the same face, the original sampling coefficients of the discrete face image are normalized to obtain the sampling coefficients corresponding to each discrete face image; based on the same video, the original sampling coefficients of the facial video frame are normalized to obtain the sampling coefficients corresponding to each facial video frame.

[0154] Optionally, the preprocessing module 501 is also used to obtain a training image set; the training image set includes multiple groups of template face images and sample face images; extract the template face from the template face image, input the template face into a pre-trained face recognition model to obtain the template face features; input the sample face image and the template face features into the feature fusion model to be trained to obtain a sample face-changing image; determine the third loss information based on the face features in the sample face-changing image and the template face features; iteratively update the parameters of the feature fusion model to be trained based on the third loss information to obtain the feature fusion model.

[0155] Please refer to Figure 7 , Figure 7 A block diagram of an electronic device 100 provided in an embodiment of the present invention. The electronic device 100 may be any device having an image processing function, such as a personal computer, a laptop computer, a server, etc. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, the processor 120, and the communication module 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0156] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0157] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, when the computer program stored in the memory 110 is executed by the processor 120, the facial image generation method disclosed in the above embodiments can be implemented.

[0158] The communication module 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.

[0159] It should be understood that Figure 7 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown. Figure 7 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0160] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 120, the facial image generation method disclosed in the above embodiments is implemented.

[0161] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0163] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a face image, It is characterized in that The method comprises: Extracting a target face from a target face image; Inputting the face-swapped image and the target face into a pre-trained face image generation model; the face image generation model includes a face recognition model and a feature fusion model; the face recognition model is obtained by mixed training using discrete face images and videos corresponding to a variety of face posture angles; Extracting features of the target face using the face recognition model, and inputting the extracted target features into the feature fusion model; The feature fusion model is used to perform face replacement on the image to be replaced based on the target features to obtain a generated face image.

2. The method for generating a facial image according to claim 1, It is characterized in that The face recognition model is obtained by: Obtain a discrete face image set, a video set and a recognition model to be trained; the discrete face image set includes multiple discrete face images of different faces; the video set includes at least one video of different faces; the sampling coefficient of the discrete face image is determined according to the face posture angle of the discrete face image; For each human face, a discrete face image whose sampling coefficient exceeds a preset sampling threshold is determined as a target image; Determining a corresponding first scaling factor according to the face posture angle of the target image; Inputting the target image into the recognition model to be trained to obtain a first feature vector; Obtaining first loss information according to the first feature vector, the face label, and the first scaling factor corresponding to the target image; Iteratively updating the parameters of the recognition model to be trained according to the first loss information to obtain an initial recognition model; The initial recognition model is iteratively trained using each of the videos to obtain a face recognition model.

3. The method for generating a facial image according to claim 2, It is characterized in that The face posture angle includes a pitch angle and a yaw angle; and determining the corresponding first scaling factor according to the face posture angle of the target image includes: A first scaling factor corresponding to the target image is obtained according to a preset angle threshold, an initial scaling value, and a pitch angle and a yaw angle corresponding to the target image.

4. The method for generating a facial image according to claim 2, It is characterized in that The obtaining first loss information according to the first feature vector, the face label and the first scaling factor corresponding to the target image includes: determining an angular boundary according to the first scaling factor; The first loss information is determined according to the first feature vector, the face label and the angle boundary.

5. The method for generating a facial image according to claim 2, It is characterized in that The iterative training of the initial recognition model using each of the videos to obtain a face recognition model includes: Determine the face video frame whose sampling coefficient exceeds the preset sampling threshold in each video as the target video frame; the sampling coefficient of the face video frame is determined according to the face posture angle of the face video frame; Determining a corresponding second scaling factor according to the face posture angle of the target video frame; Inputting the target video frame into the initial recognition model to obtain a second feature vector; Obtaining second loss information according to the second feature vector, the face label, and the second scaling factor corresponding to the target video frame; Iteratively update the parameters of the initial recognition model according to the second loss information to obtain a face recognition model.

6. The method for generating a facial image according to claim 2, It is characterized in that The face posture angles include pitch angle and yaw angle; Before obtaining the discrete face image set, the video set and the recognition model to be trained, the method further includes: Extract frames from each of the videos to obtain face video frames; Sequentially taking each of the discrete face images and each of the face video frames as images to be processed; Inputting each of the images to be processed into a preset facial key point model to obtain the pitch angle and yaw angle of the face in the image to be processed; Determining an original sampling coefficient of the image to be processed according to the pitch angle, the yaw angle, the preset angle threshold and the preset sampling value corresponding to each image to be processed; Based on the same human face, normalizing the original sampling coefficients of the discrete face images to obtain the sampling coefficients corresponding to the discrete face images; Based on the same video, the original sampling coefficients of the face video frames are normalized to obtain the sampling coefficients corresponding to the face video frames.

7. The method for generating a facial image according to any one of claims 1 to 6, It is characterized in that The feature fusion model is trained in the following way: Acquire a training image set; the training image set includes multiple groups of template face images and sample face images; Extract a template face from a template face image, and input the template face into a pre-trained face recognition model to obtain template face features; Inputting the sample face image and the template face features into the feature fusion model to be trained to obtain a sample face-changing image; Determine third loss information according to the facial features in the sample face-swapped image and the facial features of the template; Iteratively update the parameters of the feature fusion model to be trained according to the third loss information to obtain a feature fusion model.

8. A facial image generating device, It is characterized in that The device comprises: A preprocessing module, used for extracting a target face from a target face image; The replacement module is used to input the image to be replaced and the target face into a pre-trained face image generation model; the face image generation model includes a face recognition model and a feature fusion model; the face recognition model is obtained by mixed training using discrete face images and videos corresponding to a variety of face posture angles; the face recognition model is used to extract features of the target face, and the extracted target features are input into the feature fusion model; the feature fusion model is used to perform face replacement on the image to be replaced based on the target features to obtain a face generation image.

9. An electronic device, It is characterized in that It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the facial image generation method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for generating a facial image according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Image feature updating method and apparatus, and electronic device

    WO2026145330A1