A face fusion method, system, electronic device and storage medium

Abstract

The present application discloses a face fusion method, system, electronic device and storage medium, the method comprising: performing face detection and alignment processing on a source face image and a target face image respectively, calculating the source face affine transformation matrix and the target face affine transformation matrix, cropping the source face image and the target face image to obtain the source face region image and the target face region image; performing feature extraction processing on the source face region image to obtain the jewelry features and the source face features; inputting the source face features and the target face region image jointly into a face fusion model for face fusion processing to obtain the initial face fusion result and the jewelry features into a jewelry generation model for reconstruction processing to obtain the jewelry reconstruction result; covering the jewelry reconstruction result to the target face region image according to the inverse matrix of the target face affine transformation matrix to obtain the target face fusion result. The present application can improve the authenticity of face fusion and can be widely used in the field of computer vision technology.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a face fusion method, system, electronic device and storage medium. Background Art

[0002] In the related art, there is a face fusion method based on image processing technology and feature point alignment algorithm, that is, a new synthetic image is generated by simply fusing the features of the source image with the appearance of the target image. However, this method is often limited in effect when dealing with complex facial expressions, postures and lighting changes, and is prone to produce unnatural or distorted synthesis results. In addition, when the face in the image is wearing jewelry, the face of the synthetic result can only have the facial features of the source face, and the accurate restoration of the jewelry features cannot be achieved.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a face fusion method, system, electronic device and storage medium, which can improve the accuracy and naturalness of the fusion results.

[0005] To achieve the above purpose, an embodiment of the present application provides a face fusion method, which includes:

[0006] Obtain a source face image and a target face image;

[0007] Performing face detection and alignment processing on the source face image and the target face image respectively, and calculating the source face affine transformation matrix and the target face affine transformation matrix;

[0008] Performing cropping processing on the source face image and the target face image according to the source face affine transformation matrix and the target face affine transformation matrix respectively to obtain a source face region image and a target face region image;

[0009] Performing feature extraction processing on the source face region image to obtain jewelry features and source face features;

[0010] Inputting the source face features and the target face region image into a face fusion model for face fusion processing to obtain an initial face fusion result;

[0011] Inputting the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain a jewelry reconstruction result;

[0012] The jewelry reconstruction result is overlaid onto the target face area image according to the inverse matrix of the target face affine transformation matrix to obtain the target face fusion result.

[0013] In some embodiments, performing face detection and alignment processing on the source face image and the target face image respectively to calculate the source face affine transformation matrix and the target face affine transformation matrix includes:

[0014] Get a standard face template image;

[0015] Inputting the source face image and the target face image into a face detection model to perform face detection processing, respectively, to obtain source face key point information and target face key point information;

[0016] According to the standard face template image, face alignment processing is performed on the source face key point information and the target face key point information respectively, and the source face affine transformation matrix and the target face affine transformation matrix are calculated.

[0017] In some embodiments, the performing feature extraction processing on the source face region image to obtain jewelry features and source face features includes:

[0018] Inputting the source face region image into the jewelry classification model to perform jewelry feature extraction processing to obtain jewelry features;

[0019] Performing bilinear interpolation processing on the source face region image to obtain a scaled image;

[0020] The scaled image is input into a facial feature extraction model to perform facial feature extraction processing to obtain source facial features.

[0021] In some embodiments, before the source face features and the target face region image are jointly input into a face fusion model for face fusion processing to obtain an initial face fusion result, the method further includes pre-training the face fusion model, specifically including:

[0022] Obtain high-resolution portrait training data and low-resolution portrait training data;

[0023] Performing face detection and alignment processing on the high-resolution portrait training data, and cropping to obtain a training target face image;

[0024] Performing facial feature extraction processing on the high-resolution portrait training data and the low-resolution portrait training data respectively to obtain training facial features;

[0025] Inputting the training face features and the training target face image into the face fusion model to obtain a training fusion result;

[0026] Performing loss value calculation processing on the training fusion result according to the face fusion loss function to obtain the face fusion loss value;

[0027] The parameters of the face fusion model are updated according to the face fusion loss value.

[0028] In some embodiments, the performing loss value calculation processing on the training fusion result according to the face fusion loss function to obtain the face fusion loss value includes:

[0029] According to the training fusion results, the cosine similarity loss, pixel loss, perceptual loss and adversarial loss of facial features are calculated;

[0030] According to the face fusion loss function, the face feature cosine similarity loss, the pixel loss, the perceptual loss and the adversarial loss are weightedly summed to obtain a face fusion loss value.

[0031] In some embodiments, before inputting the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain the jewelry reconstruction result, the method further includes pre-training the jewelry generation model, specifically including:

[0032] The training images are constructed by computer synthesis method;

[0033] Performing jewelry feature extraction processing on the training image to obtain training jewelry features;

[0034] Inputting the training jewelry features and the training images into the jewelry generation model to obtain a training generation result;

[0035] Performing loss value calculation processing on the training generation result according to the jewelry generation loss function to obtain the jewelry generation loss value;

[0036] The parameters of the accessory generation model are updated according to the accessory generation loss value.

[0037] In some embodiments, the step of constructing a training image by computer synthesis includes:

[0038] The simulated digital human and accessories models are constructed through three-dimensional modeling technology;

[0039] Performing random condition modification processing on the simulated digital human to obtain a random model;

[0040] The jewelry model and the random model are synthesized to obtain a training image.

[0041] To achieve the above objective, another aspect of the embodiment of the present application provides a face fusion system, the system comprising:

[0042] The first module is used to obtain a source face image and a target face image;

[0043] The second module is used to perform face detection and alignment processing on the source face image and the target face image respectively, and calculate the source face affine transformation matrix and the target face affine transformation matrix;

[0044] The third module is used to perform cropping processing on the source face image and the target face image according to the source face affine transformation matrix and the target face affine transformation matrix, respectively, to obtain a source face area image and a target face area image;

[0045] The fourth module is used to perform feature extraction processing on the source face area image to obtain jewelry features and source face features;

[0046] The fifth module is used to input the source face features and the target face region image into a face fusion model to perform face fusion processing to obtain an initial face fusion result;

[0047] The sixth module is used to input the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain the jewelry reconstruction result;

[0048] The seventh module is used to overlay the jewelry reconstruction result onto the target face area image according to the inverse matrix of the target face affine transformation matrix to obtain the target face fusion result.

[0049] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0050] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0051] The embodiments of the present application include at least the following beneficial effects: The present application provides a face fusion method, system, electronic device and storage medium, which calculates the source face affine transformation matrix and the target face affine transformation matrix by performing face detection and alignment processing on the source face image and the target face image respectively; the source face image and the target face image are cropped according to the source face affine transformation matrix and the target face affine transformation matrix respectively to obtain the source face area image and the target face area image; feature extraction processing is performed on the source face area image to obtain jewelry features and source face features; the jewelry features and the source face features can be accurately extracted, thereby improving the realism of the fusion effect. The scheme also performs face fusion processing by jointly inputting source facial features and target facial area images into a facial fusion model to obtain an initial facial fusion result; the initial facial fusion result and jewelry features are input into a jewelry generation model for reconstruction processing to obtain a jewelry reconstruction result; the jewelry features can be accurately restored by reconstructing the initial facial fusion result and jewelry features; and this scheme overlays the jewelry reconstruction result onto the target facial area image according to the inverse matrix of the target facial affine transformation matrix to obtain the target facial fusion result, which can improve the accuracy and realism of the fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of a face fusion method provided in an embodiment of the present application;

[0053] Figure 2 is a schematic diagram of an image preprocessing provided in an embodiment of the present application;

[0054] Figure 3 is a schematic diagram of a source face feature extraction process provided by an embodiment of the present application;

[0055] Figure 4 is a schematic diagram of a target face extraction process provided by an embodiment of the present application;

[0056] Figure 5 is a schematic diagram of a facial feature fusion process provided by an embodiment of the present application;

[0057] Figure 6 is a schematic diagram of a jewelry reconstruction process provided by an embodiment of the present application;

[0058] Figure 7 is a schematic diagram of an affine transformation overlay processing provided in an embodiment of the present application;

[0059] Figure 8 It is a structural schematic diagram of a face fusion system provided in an embodiment of the present application;

[0060] Fig. 9 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0061] Fig.10 It is a schematic diagram of the structure of the storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0063] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0064] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0067] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0068] Face fusion technology is a technology that synthesizes two or more face images to generate a new image with the characteristics of the source image and the appearance of the target image. Face fusion technology has made significant progress and can be widely used in virtual reality, entertainment, advertising and other fields.

[0069] In the related art, there is a face fusion method based on image processing technology and feature point alignment algorithm, that is, a new synthetic image is generated by simply fusing the features of the source image with the appearance of the target image. However, this method is often limited in effect when dealing with complex facial expressions, postures and lighting changes, and is prone to produce unnatural or distorted synthesis results. There are also methods for face fusion through convolutional neural networks, but when these deal with source faces wearing accessories, after the features of the source face are fused to the target face to generate a synthetic result, the face of the synthetic result can only have the facial features of the source face, and cannot achieve accurate restoration of the features of the accessories.

[0070] In view of this, a face fusion method, system, electronic device and storage medium are provided in the embodiments of the present application. The scheme calculates the source face affine transformation matrix and the target face affine transformation matrix by performing face detection and alignment processing on the source face image and the target face image respectively; the source face image and the target face image are cropped according to the source face affine transformation matrix and the target face affine transformation matrix respectively to obtain the source face area image and the target face area image; feature extraction processing is performed on the source face area image to obtain jewelry features and source face features; the jewelry features and the source face features can be accurately extracted, thereby improving the realism of the fusion effect. The scheme also performs face fusion processing by jointly inputting source facial features and target facial area images into a facial fusion model to obtain an initial facial fusion result; the initial facial fusion result and jewelry features are input into a jewelry generation model for reconstruction processing to obtain a jewelry reconstruction result; the jewelry features can be accurately restored by reconstructing the initial facial fusion result and jewelry features; and this scheme overlays the jewelry reconstruction result onto the target facial area image according to the inverse matrix of the target facial affine transformation matrix to obtain the target facial fusion result, which can improve the accuracy and realism of the fusion.

[0071] A face fusion method provided in an embodiment of the present application relates to the field of computer vision technology. The face fusion method provided in an embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the face fusion method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] Figure 1 is an optional flowchart of a face fusion method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0075] Step S101, obtaining a source face image and a target face image;

[0076] Step S102, performing face detection and alignment processing on the source face image and the target face image respectively, and calculating the source face affine transformation matrix and the target face affine transformation matrix;

[0077] Step S103, performing cropping processing on the source face image and the target face image according to the source face affine transformation matrix and the target face affine transformation matrix, respectively, to obtain a source face region image and a target face region image;

[0078] Step S104, performing feature extraction processing on the source face region image to obtain jewelry features and source face features;

[0079] Step S105, inputting the source face features and the target face region image into a face fusion model for face fusion processing to obtain an initial face fusion result;

[0080] Step S106, inputting the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain a jewelry reconstruction result;

[0081] Step S107, overlaying the jewelry reconstruction result onto the target face area image according to the inverse matrix of the target face affine transformation matrix to obtain a target face fusion result.

[0082] Steps S101 to S107 shown in the embodiment of the present application are obtained by obtaining a source face image and a target face image input by a user, wherein the source face image can be a selfie image uploaded by the user or an image with a face selected in an album, and can be a cartoon image or a hand-drawn image, etc. The target face image is an image that needs to extract relevant features from the source face image for fusion processing, which can be understood as a material face image or a template face image, such as an ID image, a cartoon image, an ancient costume image, etc. Face detection and alignment processing are performed on the source face image and the target face image respectively, and the source face affine transformation matrix and the target face affine transformation matrix are calculated; then the source face image and the target face image are cropped according to the source face affine transformation matrix and the target face affine transformation matrix, respectively, to obtain a source face area image and a target face area image. The source face region image and the target face region image are images obtained by cropping the face regions of the source face image and the target face image according to the preset image size, respectively, by using the affine transformation matrix calculated after the alignment process after the source face image and the target face image are input into the face detection model for detection and alignment. Then, the source face region image is subjected to feature extraction process, and feature extraction is performed through the jewelry feature detection model and the face feature feature detection model to obtain jewelry features and source face features, and the jewelry may include glasses, masks, earrings, etc. Finally, the source face features and the target face region image are jointly input into the face fusion model for face fusion process to obtain the initial face fusion result; the initial face fusion result and the jewelry features are input into the jewelry generation model for reconstruction process to obtain the jewelry reconstruction result; the jewelry reconstruction result is overlaid on the target face region image according to the inverse matrix of the target face affine transformation matrix to obtain the target face fusion result. By calculating the inverse matrix of the target face affine transformation matrix, the aligned face image can be restored to the posture and position of the original image, so that the jewelry reconstruction result can be covered back to the face area cropped out of the target face. The embodiment of the present application obtains jewelry features and extracts source face feature information through a feature extraction model, obtains an initial face fusion result by inputting the target face image and the source face into a face fusion model, and outputs the initial face fusion result and the jewelry features through a jewelry generation model to solve the problem that the existing face fusion method is prone to produce unnatural or distorted synthesis results when dealing with complex facial expressions, postures and lighting changes, and cannot achieve eyeglass feature reconstruction, thereby improving the realism and fidelity of the fusion effect and achieving a more realistic and lifelike face fusion effect.

[0083] In step S101 of some embodiments, the source face image and the target face image may be obtained through user upload. The source face image and the target face image may also be obtained through other methods such as photo albums, material libraries, etc., but are not limited thereto. Figure 2, when the source face image and the target face image are obtained, the embodiment of the present application preprocesses the obtained images. Specifically, the source face image and the target face image are decoded, and the image data is recorded in the form of a numpy array. The pixel values ​​of the image data are normalized, and the value range of the pixel values ​​is normalized from 0 to 255 to the interval of -1 to 1, so as to obtain the preprocessed source face image and the target face image.

[0084] In step S102 of some embodiments, performing face detection and alignment processing on the source face image and the target face image respectively to calculate the source face affine transformation matrix and the target face affine transformation matrix includes:

[0085] Get a standard face template image;

[0086] Inputting the source face image and the target face image into a face detection model to perform face detection processing, respectively, to obtain source face key point information and target face key point information;

[0087] According to the standard face template image, face alignment and cropping processing are performed on the source face key point information and the target face key point information respectively to obtain a source face area image and a target face area image.

[0088] In the embodiment of the present application, a standard face template image is first obtained. The standard face template image is an image with a standard face proportion, which can be obtained from a face image database. Figure 3 , input the preprocessed source face image into the face detection model for face detection to obtain key point information, which includes face position information, and then perform face alignment with the standard face template image based on the key point information. The mainstream alignment methods such as FFHQ dataset alignment can be used. The source face affine transformation matrix can be calculated based on the alignment result, and then the source face image is cropped according to the source face affine transformation matrix to obtain a source face area image with a size of 256*256. The face detection model can be selected from the YOLO model, etc. Reference Figure 4 , similarly, face detection and alignment processing is performed on the preprocessed target face image, and the target face affine transformation matrix is ​​calculated, so that the target face image is cropped according to the target face affine transformation matrix to obtain a target face region image with a size of 256*256. The embodiment of the present application can accurately locate the face region by performing face detection and alignment processing on the source face image and the target face image, which facilitates the subsequent face fusion and improves the accuracy of the fusion result.

[0089] In step S103 of some embodiments, the source face image and the target face image are cropped according to the source face affine transformation matrix and the target face affine transformation matrix to obtain a source face region image and a target face region image;

[0090] In the embodiment of the present application, by performing face detection and alignment processing on the source face image and the target face image, the source face affine transformation matrix and the target face affine transformation matrix are calculated. Face alignment may include affine transformation processing such as translation, rotation, scaling, and shearing, and the affine transformation matrix is ​​calculated by establishing a homogeneous coordinate system. Figure 4 After inputting the preprocessed target face image into the face detection model to obtain the key point information in the target face image, the affine transformation matrix is ​​calculated after face alignment with the standard face template image according to the key point information, and the target face area image with a size of 256*256 is cropped from the preprocessed target face image according to the calculated affine transformation matrix. By obtaining the affine transformation matrix, the embodiment of the present application can better find the corresponding relationship between the fusion result and the target face image, thereby better improving the fusion effect.

[0091] In step S104 of some embodiments, the feature extraction process is performed on the source face region image to obtain the jewelry feature and the source face feature, including:

[0092] Inputting the source face region image into the jewelry classification model to perform jewelry feature extraction processing to obtain jewelry features;

[0093] Performing bilinear interpolation processing on the source face region image to obtain a scaled image;

[0094] The scaled image is input into a facial feature extraction model to perform facial feature extraction processing to obtain source facial features.

[0095] In the embodiment of the present application, the cropped source face region image is input into the jewelry classification model for jewelry feature extraction processing to obtain jewelry features. Figure 3, the source face image may be a face image of a person wearing glasses, so the cropped source face area image is input into the glasses classification model for glasses feature extraction processing to obtain glasses classification features. The cropped source face area image of size 256*256 is then scaled to a size of 112*112 using a bilinear interpolation method to obtain a scaled image. Finally, the scaled image is input into the facial feature extraction model for facial feature extraction processing to obtain source facial features, and the facial feature extraction model may use an Arcface model, etc. The embodiment of the present application can better fuse the features with the target face image by extracting jewelry features and facial features, thereby improving the authenticity of image fusion.

[0096] In step S105 of some embodiments, before jointly inputting the source face features and the target face region image into a face fusion model for face fusion processing to obtain an initial face fusion result, the method further includes pre-training the face fusion model, specifically including:

[0097] Obtain high-resolution portrait training data and low-resolution portrait training data;

[0098] Performing face detection and alignment processing on the high-resolution portrait training data, and cropping to obtain a training target face image;

[0099] Performing facial feature extraction processing on the high-resolution portrait training data and the low-resolution portrait training data respectively to obtain training facial features;

[0100] Inputting the training face features and the training target face image into the face fusion model to obtain a training fusion result;

[0101] Performing loss value calculation processing on the training fusion result according to the face fusion loss function to obtain the face fusion loss value;

[0102] The parameters of the face fusion model are updated according to the face fusion loss value.

[0103] In the embodiments of this application, refer to Figure 5, the source face features and the target face region image are jointly input into the face fusion model for face fusion processing to obtain the initial face fusion result. Among them, the face fusion model adopts the Encoder-Decoder network architecture, and the model input is the target face region image with a size of 256*256*3 and the source face features with a size of 1*512. When training the face fusion model, first obtain high-resolution portrait training data and low-resolution portrait training data as training data, wherein the high-resolution portrait training data is a face image with a vertical resolution greater than or equal to 720 and a face region size greater than 256*256, and the low-resolution portrait training data is a face image with a vertical resolution less than 720 and a face region less than 256*256 and greater than 112*112. By performing face detection and alignment on the high-resolution portrait data, the cropped 256*256*3 face region image is scaled to a size of 112*112*3 using a bilinear difference method, and the face features are obtained through the face extraction model. Perform face detection and alignment on low-resolution portrait data, crop an image of 112*112*3 size, and obtain face features through a face extraction model. Use a 256*256*3 face region image cropped from a high-resolution data set as a training target face image. Use the face features extracted from the high-resolution data set and the face features extracted from the low-resolution data set as training face features. Input the training face features and the training target face image into the face fusion model to obtain a training fusion result, thereby calculating the loss value of the training fusion result according to the face fusion loss function to obtain a face fusion loss value. Calculate the loss value of the training fusion result and the training target face image through the loss function of the face fusion model, optimize the loss function of the face fusion model according to the loss value, back-propagate the model loss of the loss function, and continuously adjust the model parameters until the loss value is greater than or equal to the loss value threshold, stop optimizing the face fusion model, and obtain a face fusion model that meets the requirements. The embodiment of the present application uses high-resolution portrait training data and low-resolution portrait training data to train the face fusion model, which can better cope with scenes with complex facial expressions, postures and lighting changes in the image, thereby improving the realism of image fusion.

[0104] In some embodiments, the performing loss value calculation processing on the training fusion result according to the face fusion loss function to obtain the face fusion loss value includes:

[0105] According to the training fusion results, the cosine similarity loss, pixel loss, perceptual loss and adversarial loss of facial features are calculated;

[0106] According to the face fusion loss function, the face feature cosine similarity loss, the pixel loss, the perceptual loss and the adversarial loss are weightedly summed to obtain a face fusion loss value.

[0107] In the embodiment of the present application, the expression of the face fusion loss function is as follows:

[0108] L total =αL id +βL pixel +γL percent +δL GAN ;

[0109] Where, L total Represents the face fusion loss value, L id is the cosine similarity loss between the face features extracted by the face feature extractor and the training face features in the training fusion result, L pixel is the pixel loss, L percent is the perceptual loss, L GAN For adversarial loss, α, β, γ, and δ are weights of each loss function, and a suitable value is 5, 1, 0.5, and 0.2. The embodiment of the present application obtains the face fusion loss value by weighted summing the cosine similarity loss, pixel loss, perceptual loss, and adversarial loss of facial features, which can better train the face fusion model and improve the effect of face fusion.

[0110] In step S106 of some embodiments, before inputting the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain the jewelry reconstruction result, the method further includes pre-training the jewelry generation model, specifically including:

[0111] The training images are constructed by computer synthesis method;

[0112] Performing jewelry feature extraction processing on the training image to obtain training jewelry features;

[0113] Inputting the training jewelry features and the training images into the jewelry generation model to obtain a training generation result;

[0114] Performing loss value calculation processing on the training generation result according to the jewelry generation loss function to obtain the jewelry generation loss value;

[0115] The parameters of the accessory generation model are updated according to the accessory generation loss value.

[0116] In the embodiments of this application, refer to Figure 6, by inputting the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing, the jewelry reconstruction result is obtained. The embodiment of the present application is specifically described using glasses as jewelry, and the glasses reconstruction result is obtained by inputting the face ID fusion result, that is, the face feature fusion result, and the source face glasses classification feature into the glasses generation model. The glasses generation model uses an Encoder-Decoder network structure and is trained using a generative adversarial network training method. During training, the images without glasses and the glasses classification feature vectors in the training data are jointly input into the glasses generation model to obtain the training generation result. Thereby, the training generation result is subjected to loss value calculation processing according to the jewelry generation loss function to obtain the jewelry generation loss value; the parameters of the jewelry generation model are updated according to the jewelry generation loss value. Among them, the expression of the jewelry generation loss function is as follows:

[0117] L total =αL pixel +βL percent +γL GAN

[0118] Where, L total Generates loss value for the accessory, L pixel is the pixel-level loss, L percent is the perceptual loss, L GAN is the adversarial loss. α, β, and γ are the weights of each loss function, and a suitable value is 10, 1, and 0.2. The embodiment of the present application can better train the jewelry generation model by jointly inputting the image of the jewelry worn and the jewelry features into the model for training processing, realize the reconstruction of the jewelry features, and improve the realism of image fusion.

[0119] In some embodiments, the step of constructing a training image by computer synthesis includes:

[0120] The simulated digital human and accessories models are constructed through three-dimensional modeling technology;

[0121] Performing random condition modification processing on the simulated digital human to obtain a random model;

[0122] The jewelry model and the random model are synthesized to obtain a training image.

[0123] In the embodiment of the present application, the accessories are glasses for illustration, wherein the training image is constructed by a computer synthesis method, using a pre-constructed 3D modeled high-precision simulation digital human, and designing 10 common glasses 3D models, randomly modifying the character image, character accessories, character skin color, character angle, character lighting and other conditions, and synthesizing an image without glasses and 10 images wearing different glasses to obtain a training image. The embodiment of the present application can quickly obtain training images through computer synthesis technology, thereby improving the training speed of the accessories generation model.

[0124] In step S107 of some embodiments, the jewelry reconstruction result is overlaid on the target face area image according to the inverse matrix of the target face affine transformation matrix to obtain a target face fusion result.

[0125] In the application embodiment, glasses are used as accessories for illustration. Figure 7 , the glasses reconstruction result image is overlaid back to the target face area image cropped out of the target face through affine transformation. And after inverse normalization, the value range is changed from 0 to 1 to 0 to 255, and saved as a picture as the final output result. This application can restore the aligned face image to the posture and position of the original image by calculating the inverse matrix of the target face affine transformation matrix, so as to overlay the glasses reconstruction result back to the face area cropped out of the target face, and obtain the final target face fusion result.

[0126] Below, in combination with specific application scenarios, the solution of the embodiment of the present application is described in detail and explained:

[0127] The face fusion method proposed in the present application can be applied to various photo shooting scenes or image processing scenes, and can be applied to advertising marketing, image applications, various social APPs (Applications), video processing APPs, etc., to provide face fusion functions for various APPs, and can be used to make fun videos and video ringtones, etc. For example, in an image processing APP, a material library can be provided, and the material library is used to store various types of material face images, such as animated images, ancient costume images, group photo images, etc., and each type of material face image can have multiple. When the user needs to perform face fusion, for example, by clicking the "fusion" function button, and then selecting a material face image as the target face image from the pop-up interface, such as a material face image in the animated image, the user can then use the camera to take a portrait selfie or select a portrait photo from the local album as the source face image, and perform face fusion on the source face image and the target face image selected by the user, so as to obtain a fused image having both the user's appearance features and the image features of the target face image. Among them, when performing face fusion, by adopting the face fusion method of the embodiment of the present invention, the facial features and jewelry features of the source face image and the target face image can be made more obvious, so that the fused image can realize jewelry feature reconstruction, and can reduce the distortion of the facial part when processing complex facial expressions, postures and lighting changes, making the entire fused image more natural and realistic.

[0128] See also Figure 8 The embodiment of the present application also provides a face fusion system, which can implement the above face fusion method, and the system includes:

[0129] The first module 801 is used to obtain a source face image and a target face image;

[0130] The second module 802 is used to perform face detection and alignment processing on the source face image and the target face image respectively, and calculate the source face affine transformation matrix and the target face affine transformation matrix;

[0131] The third module 803 is used to perform cropping processing on the source face image and the target face image according to the source face affine transformation matrix and the target face affine transformation matrix, respectively, to obtain a source face region image and a target face region image;

[0132] The fourth module 804 is used to perform feature extraction processing on the source face region image to obtain jewelry features and source face features;

[0133] The fifth module 805 is used to input the source face features and the target face region image into a face fusion model to perform face fusion processing to obtain an initial face fusion result;

[0134] The sixth module 806 is used to input the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain a jewelry reconstruction result;

[0135] The seventh module 807 is used to overlay the jewelry reconstruction result onto the target face area image according to the inverse matrix of the target face affine transformation matrix to obtain the target face fusion result.

[0136] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0137] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned face fusion method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0138] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0139] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0140] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0141] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the face fusion method of the embodiment of this application;

[0142] Input / output interface 903, used to implement information input and output;

[0143] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0144] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0145] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0146] Reference Fig.10 The embodiment of the present application further provides a computer-readable storage medium 1001, which stores a computer program 1002. When the computer program 1002 is executed by a processor, the above-mentioned face fusion method is implemented.

[0147] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The embodiments of the present application provide a face fusion method, system, electronic device and storage medium. The solution calculates the source face affine transformation matrix and the target face affine transformation matrix by performing face detection and alignment processing on the source face image and the target face image respectively; the source face image and the target face image are cropped according to the source face affine transformation matrix and the target face affine transformation matrix respectively to obtain the source face area image and the target face area image; feature extraction processing is performed on the source face area image to obtain jewelry features and source face features; the jewelry features and the source face features can be accurately extracted, thereby improving the realism of the fusion effect. The scheme also performs face fusion processing by jointly inputting source facial features and target facial area images into a facial fusion model to obtain an initial facial fusion result; the initial facial fusion result and jewelry features are input into a jewelry generation model for reconstruction processing to obtain a jewelry reconstruction result; the jewelry features can be accurately restored by reconstructing the initial facial fusion result and jewelry features; and this scheme overlays the jewelry reconstruction result onto the target facial area image according to the inverse matrix of the target facial affine transformation matrix to obtain the target facial fusion result, which can improve the accuracy and realism of the fusion.

[0150] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0151] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0152] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0153] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0154] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0155] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0156] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0157] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0160] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A face fusion method, characterized in that: The method comprises: Obtain a source face image and a target face image; Performing face detection and alignment processing on the source face image and the target face image respectively, and calculating a source face affine transformation matrix and a target face affine transformation matrix; Performing cropping processing on the source face image and the target face image according to the source face affine transformation matrix and the target face affine transformation matrix respectively to obtain a source face region image and a target face region image; Performing feature extraction processing on the source face region image to obtain jewelry features and source face features; Inputting the source face features and the target face region image into a face fusion model for face fusion processing to obtain an initial face fusion result; Inputting the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain a jewelry reconstruction result; Overlaying the jewelry reconstruction result onto the target face region image according to the inverse matrix of the target face affine transformation matrix to obtain a target face fusion result; Before inputting the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain the jewelry reconstruction result, the method also includes pre-training the jewelry generation model, specifically including: The training images are constructed by computer synthesis method; Performing jewelry feature extraction processing on the training image to obtain training jewelry features; Inputting the training jewelry features and the training images into the jewelry generation model to obtain a training generation result; Performing loss value calculation processing on the training generation result according to the jewelry generation loss function to obtain the jewelry generation loss value; The parameters of the accessory generation model are updated according to the accessory generation loss value.

2. The method according to claim 1, characterized in that The face detection and alignment processing are performed on the source face image and the target face image respectively, and the source face affine transformation matrix and the target face affine transformation matrix are calculated, including: Get a standard face template image; Inputting the source face image and the target face image into a face detection model to perform face detection processing, respectively, to obtain source face key point information and target face key point information; According to the standard face template image, face alignment processing is performed on the source face key point information and the target face key point information respectively, and a source face affine transformation matrix and a target face affine transformation matrix are calculated.

3. The method according to claim 1, characterized in that: The step of performing feature extraction processing on the source face region image to obtain jewelry features and source face features includes: Inputting the source face region image into the jewelry classification model to perform jewelry feature extraction processing to obtain jewelry features; Performing bilinear interpolation processing on the source face region image to obtain a scaled image; The scaled image is input into a facial feature extraction model to perform facial feature extraction processing to obtain source facial features.

4. The method according to claim 1, characterized in that: Before the source face feature and the target face region image are jointly input into a face fusion model for face fusion processing to obtain an initial face fusion result, the method further includes pre-training the face fusion model, specifically including: Obtain high-resolution portrait training data and low-resolution portrait training data; Performing face detection and alignment processing on the high-resolution portrait training data, and cropping to obtain a training target face image; Performing facial feature extraction processing on the high-resolution portrait training data and the low-resolution portrait training data respectively to obtain training facial features; Inputting the training face features and the training target face image into the face fusion model to obtain a training fusion result; Performing loss value calculation processing on the training fusion result according to the face fusion loss function to obtain the face fusion loss value; The parameters of the face fusion model are updated according to the face fusion loss value.

5. The method according to claim 4, characterized in that The step of performing loss value calculation processing on the training fusion result according to the face fusion loss function to obtain the face fusion loss value includes: According to the training fusion results, the cosine similarity loss, pixel loss, perceptual loss and adversarial loss of facial features are calculated; According to the face fusion loss function, the face feature cosine similarity loss, the pixel loss, the perceptual loss and the adversarial loss are weightedly summed to obtain a face fusion loss value.

6. The method according to claim 1, characterized in that The training image is constructed by computer synthesis method, comprising: The simulated digital human and accessories models are constructed through three-dimensional modeling technology; Performing random condition modification processing on the simulated digital human to obtain a random model; The jewelry model and the random model are synthesized to obtain a training image.

7. A face fusion system, characterized in that: The system comprises: The first module is used to obtain a source face image and a target face image; The second module is used to perform face detection and alignment processing on the source face image and the target face image respectively, and calculate the source face affine transformation matrix and the target face affine transformation matrix; The third module is used to perform cropping processing on the source face image and the target face image according to the source face affine transformation matrix and the target face affine transformation matrix, respectively, to obtain a source face area image and a target face area image; The fourth module is used to perform feature extraction processing on the source face area image to obtain jewelry features and source face features; The fifth module is used to input the source face features and the target face region image into a face fusion model to perform face fusion processing to obtain an initial face fusion result; The sixth module is used to input the initial face fusion result and the jewelry feature into the jewelry generation model for reconstruction processing to obtain the jewelry reconstruction result; The seventh module is used to cover the jewelry reconstruction result to the target face area image according to the inverse matrix of the target face affine transformation matrix to obtain the target face fusion result; In the sixth module, the initial face fusion result and the jewelry feature are input into the jewelry generation model for reconstruction processing. Before obtaining the jewelry reconstruction result, the sixth module also includes pre-training the jewelry generation model, specifically including: The training images are constructed by computer synthesis method; Performing jewelry feature extraction processing on the training image to obtain training jewelry features; Inputting the training jewelry features and the training images into the jewelry generation model to obtain a training generation result; Performing loss value calculation processing on the training generation result according to the jewelry generation loss function to obtain the jewelry generation loss value; The parameters of the accessory generation model are updated according to the accessory generation loss value.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images