Face synthesis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a face synthesis method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for financial science and technology and medical science and technology scenes. The method comprises the following steps: acquiring initial face sample data, and performing facial feature extraction on the initial face sample data to obtain sample face features; performing abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data; performing model training on the original face synthesis model based on the target face sample data and the sample face features to obtain a target face synthesis model; performing image remodeling on the reference face image through the target face synthesis model to obtain a target remodeling feature; performing convolution processing on the target remodeling feature through a target face synthesis model to obtain a target convolution feature; and performing face synthesis on the target convolution features through a target face synthesis model to obtain a target face image. According to the embodiment of the invention, the accuracy of face synthesis can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to fintech scenarios and medical technology scenarios. In particular, it relates to a face synthesis method, an apparatus, an electronic device, and a storage medium. Background Art

[0002] Face synthesis is an artificial intelligence technology aimed at integrating the face of a reference person into a candidate picture to achieve face migration. Face synthesis can be applied to a variety of application scenarios. For example, in the fintech scenario, it can generate a character image for business personnel; and in the medical technology scenario, it can generate a character image for medical staff for image display in scenarios such as personal profiles and online registration systems.

[0003] Currently, diffusion models are mainly used to achieve face synthesis. However, due to the strong randomness of the character images synthesized by diffusion models, the synthesized face images may have a large difference from the actual face of the reference person, thus affecting the accuracy of face synthesis.

[0004] Therefore, how to improve the accuracy of face synthesis has become a technical problem to be urgently solved. Summary of the Invention

[0005] The main purpose of the embodiments of this application is to propose a face synthesis method, an apparatus, an electronic device, and a storage medium, aiming to improve the accuracy of face synthesis.

[0006] To achieve the above object, in the first aspect of the embodiments of this application, a face synthesis method is proposed. The method includes:

[0007] Obtain initial face sample data;

[0008] Extract facial features from the initial face sample data to obtain sample face features;

[0009] Based on the sample face features, perform abnormal image cleaning on the initial face sample data to obtain target face sample data;

[0010] Based on the target face sample data and the sample face features, train a preset original face synthesis model to obtain a target face synthesis model;

[0011] Through the target face synthesis model, perform image reshaping on a pre-obtained reference face image to obtain target reshaping features;

[0012] Through the target face synthesis model, perform convolution processing on the target reshaping features to obtain target convolution features;

[0013] Performing face synthesis on the target convolutional features through the target face synthesis model to obtain a target face image.

[0014] In some embodiments, the target face synthesis model includes a target feature extraction sub-model and a target feature reshaping sub-model; the performing image reshaping on a pre-acquired reference face image through the target face synthesis model to obtain target reshaped features includes:

[0015] Performing feature extraction on the reference face image through the target feature extraction sub-model to obtain target image features;

[0016] Performing feature reshaping on the target image features through the target feature reshaping sub-model to obtain the target reshaped features.

[0017] In some embodiments, the training a preset original face synthesis model based on the target face sample data and the sample face features to obtain a target face synthesis model includes:

[0018] Performing face synthesis on the target face sample data through the original face synthesis model to obtain a predicted sample face image;

[0019] Calculating a loss between the predicted sample face image and the sample face features to obtain face synthesis loss data;

[0020] Optimizing the original face synthesis model based on the face synthesis loss data to obtain the target face synthesis model.

[0021] In some embodiments, the calculating a loss between the predicted sample face image and the sample face features to obtain face synthesis loss data includes:

[0022] Performing face detection on the predicted sample face image to obtain predicted face image data;

[0023] Performing feature extraction on the predicted face image data to obtain initial predicted features;

[0024] Performing feature dimension transformation on the initial predicted features to obtain predicted face features;

[0025] Calculating a similarity loss between the predicted face features and the sample face features to obtain the face synthesis loss data.

[0026] In some embodiments, the obtaining initial face sample data includes:

[0027] Obtain the original face sample data; wherein, the original face sample data includes an original sample image and a sample image label, and the sample image label is used to characterize whether the original sample image is a valid image or an invalid image;

[0028] Perform image screening on the original sample image based on the sample image label to obtain a training sample image; wherein, the training sample image is the valid image;

[0029] Perform resolution screening on the training sample image to obtain the initial face sample data.

[0030] In some embodiments, the initial face sample data includes multiple initial sample images, and performing facial feature extraction on the initial face sample data to obtain sample face features includes:

[0031] Perform face detection on the initial sample image to obtain a sample face region;

[0032] Perform feature extraction on the sample face region to obtain initial sample features;

[0033] Perform feature dimension transformation on the initial sample features to obtain the sample face features.

[0034] In some embodiments, performing abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data includes:

[0035] Calculate the similarity of the sample face features of any two initial sample images to obtain the initial feature similarity data between the two initial sample images, and calculate the similarity of the sample face regions of any two initial sample images to obtain the sample face region similarity data between the two initial sample images;

[0036] Calculate the difference between the initial feature similarity data and the sample face region similarity data between the two initial sample images to obtain the sample face similarity difference between the two initial sample images;

[0037] If the sample face similarity difference is less than a preset similarity difference threshold, then remove one initial sample image from the two initial sample images to obtain the target face sample data.

[0038] To achieve the above object, a second aspect of the embodiments of the present application proposes a face synthesis device, and the device includes:

[0039] The device includes:

[0040] A sample data acquisition module, configured to acquire initial face sample data;

[0041] A sample feature acquisition module, configured to perform facial feature extraction on the initial face sample data to obtain sample face features;

[0042] An image cleaning module, configured to perform abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data;

[0043] A raw model training module, configured to perform model training on a preset raw face synthesis model based on the target face sample data and the sample face features to obtain a target face synthesis model;

[0044] A target image reshaping module, configured to perform image reshaping on a pre-acquired reference face image through the target face synthesis model to obtain target reshaping features;

[0045] A target image convolution module, configured to perform convolution processing on the target reshaping features through the target face synthesis model to obtain target convolution features;

[0046] A target face synthesis module, configured to perform face synthesis on the target convolution features through the target face synthesis model to obtain a target face image.

[0047] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, where the electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect above is implemented.

[0048] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0049] The face synthesis method, device, electronic device, and storage medium provided by the present application obtain initial face sample data, perform facial feature extraction on the initial face sample data to obtain sample face features, and then perform abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data, improving the quality of the sample data; further, training a raw face synthesis model according to the target face sample data to obtain a target face synthesis model, which can accurately learn face features and perform face synthesis; finally, performing image reshaping and convolution processing on a reference face image through the target face synthesis model to synthesize a high-quality target face image, and the target face image has the face features of the reference face image, improving the accuracy of face synthesis. Description of the Drawings

[0050] Figure 1 is the flowchart of the face synthesis method provided by the embodiments of the present application;

[0051] Figure 2 is Figure 1 the flowchart of step S101 in

[0052] Figure 3 is Figure 1 the flowchart of step S102 in

[0053] Figure 4 is Figure 1 the flowchart of step S103 in

[0054] Figure 5 is Figure 1 the flowchart of step S104 in

[0055] Figure 6 is Figure 5 the flowchart of step S502 in

[0056] Figure 7 is Figure 1 the flowchart of step S105 in

[0057] Figure 8 is the structural schematic diagram of the face synthesis device provided by the embodiments of the present application;

[0058] Figure 9 is the hardware structural schematic diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0059] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different module division in the device or a different order in the flowchart. Terms such as "first" and "second" in the description, claims and the above drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some terms involved in this application are parsed:

[0063] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence; artificial intelligence is a branch of computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. The research in this field includes robots, speech recognition, image recognition, natural language processing, and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. It also uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use knowledge to obtain the best results in terms of theories, methods, technologies, and application systems.

[0064] Face synthesis: Face synthesis is a technology that uses artificial intelligence technology to synthesize face images. Based on deep learning algorithms, face synthesis can create realistic and new face images with specific attributes by analyzing and synthesizing the characteristics of reference face images. Face synthesis technology has broad application prospects in fields such as advertising and virtual image creation. For example, in the fintech scenario, it can generate character images for business personnel; in the medical technology scenario, it can generate character images for medical staff, which can be used for image display in scenarios such as personal profiles and online registration systems.

[0065] Diffusion Models: Diffusion Models is a generative model based on probability theory. It simulates the gradual diffusion and reverse diffusion processes in the data generation process and can generate high-quality and diverse data. The core idea of this model is to gradually add noise to the data (forward diffusion), and then learn how to reverse this process to generate new data samples from the noise (reverse diffusion). Diffusion Models can be used for face synthesis. By learning the characteristics and distributions of face images, it can generate realistic face images from noise. During the face synthesis process, Diffusion Models can capture the subtle features and expression changes of faces, thus generating face images with a high degree of realism and rich details.

[0066] The ControlNet model is a neural network technology for enhancing image generation, especially suitable for the Stable Diffusion Model. The ControlNet model achieves more refined control over the content of the generated images by adding an independent control module to the original diffusion model. These control signals can be information such as the edges, sketches, depth maps, or other forms of the image, thus allowing users to guide the generation process according to specific requirements. Through end-to-end training, the ControlNet model can significantly improve the controllability and quality of image generation while maintaining the stability of the original model, bringing revolutionary progress to the field of image generation.

[0067] Currently, diffusion models are mainly used to achieve face synthesis. However, due to the strong randomness of the human face images synthesized by diffusion models, the synthesized face images may have a large difference from the actual face of the reference person, thus affecting the accuracy of face synthesis.

[0068] To overcome the problem of strong randomness in diffusion models, a face fusion model can be bridged. However, the face fusion model not only increases the consumption of computing resources, but also has general stability of the fusion results and still cannot improve the accuracy of face synthesis.

[0069] Based on this, the embodiments of this application provide a face synthesis method, device, electronic device, and storage medium, aiming to improve the accuracy of face synthesis.

[0070] The face synthesis method, device, electronic device, and storage medium provided by the embodiments of this application are specifically described through the following embodiments. First, the face synthesis method in the embodiments of this application is described.

[0071] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Among them, Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0072] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0073] The face synthesis method provided by the embodiments of the present application relates to the field of artificial intelligence technology. The face synthesis method provided by the embodiments of the present application can be applied to a terminal, or to a server side, or can be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the face synthesis method, etc., but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] It should be noted that in each specific embodiment of the present application, when it comes to performing relevant processing based on data related to the user's identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or redirecting to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0076] Figure 1 is an optional flowchart of the face synthesis method provided by the embodiments of the present application. Figure 1 The method in may include but is not limited to steps S101 to S107.

[0077] Step S101, obtain initial face sample data;

[0078] Step S102, perform face feature extraction on the initial face sample data to obtain sample face features;

[0079] Step S103, perform abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data;

[0080] Step S104, perform model training on a preset original face synthesis model based on the target face sample data and the sample face features to obtain a target face synthesis model;

[0081] Step S105, perform image reshaping on a pre-obtained reference face image through the target face synthesis model to obtain target reshaping features;

[0082] Step S106, perform convolution processing on the target reshaping features through the target face synthesis model to obtain target convolution features;

[0083] Step S107, perform face synthesis on the target convolution features through the target face synthesis model to obtain a target face image.

[0084] Steps S101 to S107 illustrated in the embodiments of the present application, by obtaining initial face sample data and performing face feature extraction on the initial face sample data to obtain sample face features, and then performing abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data, improve the quality of the sample data; further, training the original face synthesis model according to the target face sample data to obtain a target face synthesis model, which can accurately learn face features and perform face synthesis; finally, performing image reshaping and convolution processing on the reference face image through the target face synthesis model to synthesize a high-quality target face image, and the target face image has the face features of the reference face image, improving the accuracy of face synthesis.

[0085] Please refer to Figure 2 , in some embodiments, step S101 may include but is not limited to steps S201 to S203:

[0086] Step S201, obtain original face sample data; wherein, the original face sample data includes an original sample image and a sample image label, and the sample image label is used to characterize whether the original sample image is a valid image or an invalid image;

[0087] Step S202, perform image screening on the original sample image based on the sample image label to obtain a training sample image; wherein, the training sample image is a valid image;

[0088] Step S203: Screen the resolution of the training sample images to obtain the initial face sample data.

[0089] Steps S201 to S203 illustrated in the embodiments of the present application obtain the original face sample data, where the original face sample data includes the original sample images and the sample image labels used to characterize whether the original sample images are valid images or invalid images. Then, based on the sample image labels, the valid images are screened out from the original sample images to obtain the training sample images. Finally, the resolution of the training sample images is screened to obtain the initial face sample data, which can remove invalid images, reduce noise interference, ensure the clarity of the images, help the model learn more accurate face features, and thus improve the training efficiency and accuracy of the face synthesis model.

[0090] In step S201 of some embodiments, the original face sample data is a picture data set pre-obtained from the network or the actual business application scenario, such as the Laion-5B data set, the IMDB-WIKI face data set, the images of authorized business personnel, etc.; among them, the original face sample data includes the original sample images and the sample image labels, and the sample image labels are used to characterize whether the original sample images are valid images or invalid images.

[0091] It should be noted that valid images and invalid images are defined for specific application scenarios. For example, in the fintech scenario, generating a character image for business personnel belongs to a commercial scenario. Some images in the data set do not meet the requirements of the fintech scenario (such as anime characters, AI pictures, etc.), or are required not to be used commercially. These sample images need to be excluded. Another example is in the medical technology scenario, generating a character image for medical staff, which can be used for image display in scenarios such as personal profiles and online registration systems, and is also considered a commercial scenario. Therefore, some images that do not conform to the image of medical staff, or are required not to be used commercially, also need to be excluded.

[0092] Therefore, screening images based on the sample image labels of each original sample image and screening out the valid images can improve the quality of the images, avoid potential copyright disputes and privacy leakage risks, and ensure the legality and compliance of data use.

[0093] Furthermore, in step S203 of some embodiments, screening the resolution of the training sample images based on a preset resolution threshold can ensure the clarity of the input data during model training and improve the model's ability to recognize face features and the quality of the generated images.

[0094] Specifically, the preset resolution threshold needs to be set according to the actual application scenario. For example, the resolution threshold can be set to 1024*1024, 512*512, etc., which is not limited thereto.

[0095] In some embodiments, the initial face sample data includes multiple initial sample images.

[0096] Please refer to Figure 3 , in some embodiments, step S102 may include but is not limited to steps S301 to S303:

[0097] Step S301: Perform face detection on the initial sample images to obtain the sample face regions.

[0098] Step S302: Extract features from the sample face regions to obtain the initial sample features.

[0099] Step S303: Perform feature dimension transformation on the initial sample features to obtain the sample face features.

[0100] Steps S301 to S303 illustrated in the embodiments of the present application can accurately locate and extract the key information of the face region by performing face detection, feature extraction, and feature dimension transformation on the initial sample images. Furthermore, the high-dimensional initial features can be converted into low-dimensional and more representative sample face features, which can simplify the data complexity processed by the model and enhance the model's recognition and learning ability of face features.

[0101] In some embodiments, step S301 may include but is not limited to the following steps:

[0102] Perform face detection on the initial sample images through a preset face detection model to obtain the number of faces in the initial sample images.

[0103] Screen the initial sample images based on the number of faces to obtain candidate sample images; among them, there is exactly one face image in the candidate sample images.

[0104] Perform region detection on the candidate sample images through a preset face recognition model to obtain the sample face regions.

[0105] By screening out the images containing a single face and extracting the sample face regions, the accuracy and efficiency of face processing are improved.

[0106] In some embodiments, the face detection model can be a pre-trained model such as MTCNN, FaceNet, etc.; the face recognition model can be a pre-trained model such as DeepFace, ArcFace, etc.; or the face detection model and the face recognition model can be the same model, and it is not limited thereto.

[0107] Furthermore, in step S302 of some embodiments, face extraction is performed on the candidate sample images based on the sample face regions through a face recognition model to obtain the initial sample features.

[0108] In some embodiments, the dimension of the initial sample features may be a relatively high dimension, which includes some noise information or unimportant features. Therefore, it is necessary to perform a dimension transformation on the initial sample features to reduce the consumption of computing resources.

[0109] In step S303 of some embodiments, the dimension of the initial sample features can be transformed to a preset feature dimension by means of dimensionality reduction or reduction, etc., to obtain the sample face features; among them, the preset feature dimension needs to be set according to the design of the model structure, computing resources, and application scenarios. For example, the preset feature dimension is 2048 dimensions, 1024 dimensions, etc., which is not limited thereto.

[0110] After step S303 of some embodiments, the face synthesis method further includes, but is not limited to: storing the sample face image and the corresponding sample face features, which can be used for subsequent abnormal image cleaning and model training processes.

[0111] Please refer to Figure 4 , in some embodiments, step S103 may include, but is not limited to, steps S401 to S403:

[0112] Step S401, calculate the similarity of the sample face features of any two initial sample images to obtain the initial feature similarity data between the two initial sample images, and calculate the similarity of the sample face regions of any two initial sample images to obtain the sample face region similarity data between the two initial sample images;

[0113] Step S402, calculate the difference between the initial feature similarity data and the sample face region similarity data between the two initial sample images to obtain the sample face similarity difference between the two initial sample images;

[0114] Step S403, if the sample face similarity difference is less than the preset similarity difference threshold, then remove one of the two initial sample images to obtain the target face sample data.

[0115] Steps S401 to S403 illustrated in the embodiments of the present application, by calculating the similarity of the sample face features and the sample face regions between any two initial sample images, obtain the initial feature similarity data and the sample face region similarity data, and then calculate the difference between the initial feature similarity data and the sample face region similarity data between the two initial sample images to obtain the sample face similarity difference between the two initial sample images; and then perform deduplication according to the similarity difference threshold and the sample face similarity difference, and remove the redundant samples with high similarity, so as to streamline the face sample data and improve the efficiency and accuracy of the subsequent face synthesis training task.

[0116] In some embodiments, for any two initial sample images, by calculating the similarity of the sample face features and the sample face regions, initial feature similarity data and sample face region similarity data are obtained. In addition to considering the similarity of the face features, the similarity of the face regions is also considered. Compared with the method of only judging the similarity of the face features, image deduplication can be performed more accurately.

[0117] It can be understood that the higher the similarity between two initial sample images, the larger the initial feature similarity data and the larger the sample face region similarity data. Furthermore, after subtracting the initial feature similarity data from the sample face region similarity data, the sample face similarity difference is smaller. Therefore, if the sample face similarity difference is less than a preset similarity difference threshold, it indicates that the two initial sample images are the same image, and deduplication is required. One of the two initial sample images is retained, and when one of the initial sample images is removed, the corresponding sample face features are synchronously removed.

[0118] In some embodiments, after step S402, the face synthesis method further includes, but is not limited to:

[0119] If the initial feature similarity data is greater than a preset face feature similarity threshold and the sample face similarity difference is greater than a preset abnormal style threshold, one of the two initial sample images is removed to obtain target face sample data.

[0120] It can be understood that if the initial feature similarity data is greater than a preset face feature similarity threshold, it is considered to be the same person, but the sample face similarity difference is greater than a preset abnormal style threshold, which means the same person is in two different photos. Then, one of the two initial sample images needs to be removed to obtain target face sample data, and when one of the initial sample images is removed, the corresponding sample face features are synchronously removed. This can avoid interference from abnormal styles and help improve the robustness of the face synthesis model.

[0121] In some embodiments, after step S403, the face synthesis method further includes:

[0122] Obtain the sample quantity of the target face sample data;

[0123] If the sample quantity is less than a preset training sample quantity threshold, steps S101 to S03 are repeatedly executed.

[0124] The specific implementation manners are basically the same as those of steps S201 to S203, steps S301 to S303, and steps S401 to S403 above, and will not be elaborated here.

[0125] In some embodiments, after step S103 and before step S104, the face synthesis method further includes constructing an original face synthesis model. The original face synthesis model is based on the ControlNet model, and the structure of each zero convolution layer in the ControlNet model is adjusted. Specifically, it is changed to: an original feature extraction sub-model (MLP layer), an original feature reshaping sub-model (reshape layer), an original two-dimensional convolution sub-model (3 CONV2D layers), and an original zero convolution sub-model (zero convolution layer). The finally output structure is input into the original diffusion sub-model (stable diffusion model) in the ControlNet model, thereby realizing face synthesis and improving the accuracy of face synthesis.

[0126] At the same time, it can overcome the limitation that the ControlNet model only supports controlling the generation result according to information such as pose and shape, but cannot generate a human face according to expression information.

[0127] Please refer to Figure 5 , in some embodiments, step S104 includes but is not limited to steps S501 to S503:

[0128] Step S501, perform face synthesis on the target face sample data through the original face synthesis model to obtain a predicted sample face image;

[0129] Step S502, calculate the loss between the predicted sample face image and the sample face features to obtain face synthesis loss data;

[0130] Step S503, optimize the original face synthesis model based on the face synthesis loss data to obtain a target face synthesis model.

[0131] Steps S501 to S503 illustrated in the embodiments of the present application perform face synthesis on the target face sample data through the original face synthesis model to obtain a predicted sample face image; and calculate the loss data between the predicted sample face image and the sample face features, so as to finely adjust the model parameters of the original face synthesis model, thereby improving the face learning ability and synthesis ability of the face synthesis model. Finally, the optimized target face synthesis model can achieve more efficient and accurate face image generation.

[0132] In some embodiments, step S501 includes but is not limited to the following steps:

[0133] Extract features from the target face sample data through the original feature extraction sub-model to obtain sample image features;

[0134] The sample image features are reshaped by the original feature reshaping sub-model to obtain the sample reshaped features;

[0135] The sample reshaped features are subjected to two-dimensional convolution processing by the original two-dimensional convolution sub-model to obtain the sample two-dimensional convolution features;

[0136] The sample two-dimensional convolution features are subjected to zero convolution processing by the original zero convolution sub-model to obtain the sample zero convolution features;

[0137] The sample zero convolution features are subjected to diffusion calculation by the original diffusion sub-model to obtain the predicted sample face image.

[0138] Specifically, the original feature extraction sub-model can learn the feature representation from the target face sample data through multi-layer non-linear transformation, that is, the sample image features, and the sample image features have stronger expression ability and robustness.

[0139] The original feature reshaping sub-model rearranges the sample image features according to the specified new shape to obtain the sample reshaped features with different dimensional structures, which is convenient for further processing by subsequent network layers and improves the effectiveness and utilization rate of the features.

[0140] Since the original two-dimensional convolution sub-model includes multiple two-dimensional convolution layers, the sample reshaped features are gradually subjected to feature extraction through the multiple two-dimensional convolution layers to obtain the sample two-dimensional convolution features, which can enhance the feature representation and improve the model's ability to capture image details.

[0141] The sample two-dimensional convolution features are subjected to zero convolution processing by the original zero convolution sub-model to obtain the sample zero convolution features, which can facilitate subsequent extended calculations and reduce the computational overhead.

[0142] The original diffusion sub-model gradually adds and removes noise to the sample zero convolution features in the diffusion framework, and finally generates the predicted sample face image, which has a certain similarity with the face data in the target face sample data.

[0143] It should be noted that step S501 can gradually refine and transform the feature information of the target face sample data through the above steps, including feature extraction, feature reshaping, two-dimensional convolution, zero convolution, and diffusion calculation, and finally generate a high-quality predicted sample face image while retaining the key features, which helps to improve the detail richness and authenticity of face image generation and enhance the adaptability and generalization ability of the model to different face feature changes.

[0144] Please refer to Figure 6 In some embodiments, step S502 includes but is not limited to steps S601 to S604:

[0145] Step S601: Perform face detection on the predicted sample face image to obtain predicted face image data;

[0146] Step S602: Extract features from the predicted face image data to obtain initial predicted features;

[0147] Step S603: Perform feature dimension transformation on the initial predicted features to obtain predicted face features;

[0148] Step S604: Calculate the similarity loss between the predicted face features and the sample face features to obtain face synthesis loss data.

[0149] Steps S601 to S604 illustrated in the embodiments of the present application can accurately extract the key information of the face from the predicted sample face image through face detection, feature extraction, and feature dimension transformation. Moreover, the dimension of the predicted face features is the same as that of the sample face features, which is conducive to calculating the similarity loss and provides an optimized reverse for model training.

[0150] It should be noted that the specific implementation manners of the above steps S601 to S602 are basically the same as those of the above steps S301 to S303, and will not be elaborated here.

[0151] It should be noted that the feature dimensions of the predicted face features and the sample face features are the same.

[0152] In step S604 of some embodiments, by calculating the similarity between the predicted face features and the sample face features, predicted similarity data is obtained, and 1 minus the predicted similarity data is used to obtain face synthesis loss data. The goal of model training is to minimize the face synthesis loss data, that is, to improve the similarity between the predicted face features and the sample face features.

[0153] In step S503 of some embodiments, methods such as backpropagation, gradient descent, and momentum update can be used to adjust the parameters of the original face synthesis model based on the face synthesis loss data to obtain the target face synthesis model. Specifically, the parameters of the original feature extraction sub-model, the original feature reshaping sub-model, the original two-dimensional convolutional sub-model, the original zero convolutional sub-model, and the original diffusion sub-model are adjusted to obtain the target feature extraction sub-model, the target feature reshaping sub-model, the target two-dimensional convolutional sub-model, the target zero convolutional sub-model, and the target diffusion sub-model. That is, the target face synthesis model includes the target feature extraction sub-model, the target feature reshaping sub-model, the target two-dimensional convolutional sub-model, the target zero convolutional sub-model, and the target diffusion sub-model.

[0154] Please refer to Figure 7, in some embodiments, step S105 may include but is not limited to steps S701 to S702:

[0155] Step S701, extracting features from the reference face image through the target feature extraction sub-model to obtain target image features;

[0156] Step S702, reshaping the target image features through the target feature reshaping sub-model to obtain target reshaped features.

[0157] Steps S701 to S702 illustrated in the embodiments of the present application can accurately extract and optimize the key information in the face image by first using the target feature extraction sub-model to extract key features from the reference face image and then using the target feature reshaping sub-model for feature reshaping, thereby enhancing the expression ability and robustness of the features and providing more accurate and distinguishable feature data for subsequent face synthesis tasks.

[0158] In some embodiments, before step S701, the face synthesis method further includes performing image preprocessing on the reference face image. Specifically, the image preprocessing includes but is not limited to: size adjustment, brightness adjustment, resolution adjustment, etc., and specific selection needs to be combined with the actual application scenario and is not limited thereto.

[0159] In step S701 of some embodiments, the target feature extraction sub-model can learn feature representations, that is, target image features, from the reference face image through multi-layer non-linear transformation, and the target image features have stronger expression ability and robustness.

[0160] In step S702 of some embodiments, the target feature reshaping sub-model rearranges the target image features according to the specified new shape to obtain target reshaped features with different dimensional structures, which is convenient for further processing by subsequent network layers and improves the effectiveness and utilization rate of the features.

[0161] In some embodiments, step S106 may include but is not limited to the following steps:

[0162] Performing two-dimensional convolution processing on the target reshaped features through the target two-dimensional convolution sub-model in the target face synthesis model to obtain target two-dimensional convolution features;

[0163] Performing zero convolution processing on the target two-dimensional convolution features through the target zero convolution sub-model in the target face synthesis model to obtain target convolution features.

[0164] Specifically, since the target two-dimensional convolution sub-model includes multiple two-dimensional convolution layers, gradually extracting features from the target reshaped features through the multiple two-dimensional convolution layers to obtain target two-dimensional convolution features can enhance feature representation and improve the model's ability to capture image details;

[0165] Performing zero convolution on the target two-dimensional convolution feature through the target zero convolution sub-model to obtain the target convolution feature, which can facilitate subsequent extended calculations and reduce the computational overhead at the same time.

[0166] In step S107 of some embodiments, noise is gradually added to and removed from the target convolution feature in the diffusion framework through the target diffusion sub-model in the target face synthesis model to synthesize the target face image, where the face in the target face image is the same as the face in the reference face image, so as to quickly synthesize the desired face.

[0167] In some embodiments, the face synthesis method provided in the embodiments of the present application can be applied to the fintech scenario to quickly synthesize a new picture according to the real face picture of the business personnel, which can meet the publicity requirements of the financial scenario (such as dressing neatly and having a simple and capable hairstyle), and can be used as the personal publicity poster of the business personnel, realizing personalized promotion.

[0168] The specific face synthesis process may include but is not limited to the following steps:

[0169] Step 1: Obtain the real face photo of the business personnel and perform image preprocessing on the real face picture to obtain the reference face image; where the image preprocessing includes but is not limited to: size adjustment, brightness adjustment, resolution adjustment, etc.;

[0170] Step 2: Perform face synthesis on the reference face image through the target face synthesis model to obtain the target face image; where the target face synthesis model includes a target feature extraction sub-model, a target feature reshaping sub-model, a target two-dimensional convolution sub-model, a target zero convolution sub-model, and a target diffusion sub-model;

[0171] Step 2.1: Extract features from the reference face image through the target feature extraction sub-model to obtain the target image features;

[0172] Step 2.2: Reshape the target image features through the target feature reshaping sub-model to obtain the target reshaped features.

[0173] Step 2.3: Perform two-dimensional convolution on the target reshaped features through the target two-dimensional convolution sub-model in the target face synthesis model to obtain the target two-dimensional convolution features;

[0174] Step 2.4: Perform zero convolution on the target two-dimensional convolution features through the target zero convolution sub-model in the target face synthesis model to obtain the target convolution features;

[0175] Step 2.5: Perform face synthesis on the target convolution features through the target face synthesis model to obtain the target face image.

[0176] It can be understood that after the target face synthesis model is trained, it is possible to perform person synthesis based on the reference face images of any person, which can be quickly applied to fintech scenarios, such as the synthesis of virtual character images, the generation of promotional materials related to business personnel, etc., and can achieve personalized promotion and increase customer stickiness. Similarly, it can also be quickly applied to medical technology scenarios, and can generate character images for medical staff for image display in scenarios such as personal profiles and online registration systems.

[0177] Please refer to Figure 8 , this embodiment of the present application also provides a face synthesis device, which can implement the above face synthesis method. The device includes:

[0178] A sample data acquisition module 801, configured to acquire initial face sample data;

[0179] A sample feature acquisition module 802, configured to perform facial feature extraction on the initial face sample data to obtain sample face features;

[0180] An image cleaning module 803, configured to perform abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data;

[0181] A raw model training module 804, configured to perform model training on a preset raw face synthesis model based on the target face sample data and the sample face features to obtain a target face synthesis model;

[0182] A target image reshaping module 805, configured to perform image reshaping on a pre-acquired reference face image through the target face synthesis model to obtain target reshaping features;

[0183] A target image convolution module 806, configured to perform convolution processing on the target reshaping features through the target face synthesis model to obtain target convolution features;

[0184] A target face synthesis module 807, configured to perform face synthesis on the target convolution features through the target face synthesis model to obtain a target face image.

[0185] The specific implementation manner of this face synthesis device is basically the same as the specific embodiment of the above face synthesis method, and will not be elaborated here.

[0186] This embodiment of the present application also provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above face synthesis method. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0187] Please refer to Figure 9 , Figure 9Schematically shows the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0188] A processor 901, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0189] A memory 902, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the face synthesis method in the embodiments of the present application;

[0190] An input / output interface 903, which is used to implement information input and output;

[0191] A communication interface 904, which is used to implement communication and interaction between this device and other devices. It can implement communication through a wired method (such as USB, network cable, etc.) or through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.);

[0192] A bus 905, which transmits information between various components of the device (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0193] Among them, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other inside the device through the bus 905.

[0194] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned face synthesis method is implemented.

[0195] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0196] The face synthesis method, device, electronic device, and storage medium provided by the embodiments of the present application obtain initial face sample data, extract facial features from the initial face sample data to obtain sample face features, and then perform abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data, improving the quality of the sample data; further, training the original face synthesis model according to the target face sample data to obtain a target face synthesis model, which can accurately learn face features and perform face synthesis; finally, performing image reshaping and convolution processing on the reference face image through the target face synthesis model to synthesize a high-quality target face image, and the target face image has the face features of the reference face image, improving the accuracy of face synthesis.

[0197] The embodiments described in the embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0198] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0199] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0201] In the description of this application and the above-mentioned accompanying drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0202] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expressions refer to any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0203] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the above-mentioned division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0204] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] In addition, in each embodiment of the present application, each functional unit may be integrated into a processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0206] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store programs.

[0207] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings. This does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A face synthesis method, characterized in that: The method comprises: Obtain initial face sample data; Extracting facial features from the initial facial sample data to obtain sample facial features; Based on the sample face features, the initial face sample data is cleaned of abnormal images to obtain target face sample data; Based on the target face sample data and the sample face features, a preset original face synthesis model is trained to obtain a target face synthesis model; Reshaping a pre-acquired reference face image using the target face synthesis model to obtain target reshaping features; Performing convolution processing on the target reshaping feature through the target face synthesis model to obtain a target convolution feature; The target face synthesis model is used to perform face synthesis on the target convolution features to obtain a target face image.

2. The method according to claim 1, characterized in that The target face synthesis model includes a target feature extraction sub-model and a target feature reconstruction sub-model; The step of reshaping the pre-acquired reference face image using the target face synthesis model to obtain target reshaping features includes: Performing feature extraction on the reference face image through the target feature extraction sub-model to obtain target image features; The target image feature is reshaped by the target feature reshaping submodel to obtain the target reshaped feature.

3. The method according to claim 1, characterized in that The method of training a preset original face synthesis model based on the target face sample data and the sample face features to obtain a target face synthesis model includes: Performing face synthesis on the target face sample data by using the original face synthesis model to obtain a predicted sample face image; Performing loss calculation on the predicted sample face image and the sample face features to obtain face synthesis loss data; The original face synthesis model is optimized based on the face synthesis loss data to obtain the target face synthesis model.

4. The method according to claim 3, characterized in that The step of performing loss calculation on the predicted sample face image and the sample face features to obtain face synthesis loss data includes: Performing face detection on the predicted sample face image to obtain predicted face image data; Extracting features from the predicted face image data to obtain initial predicted features; Performing feature dimension transformation on the initial predicted features to obtain predicted facial features; A similarity loss calculation is performed on the predicted facial features and the sample facial features to obtain the facial synthesis loss data.

5. The method according to claim 1, characterized in that The obtaining of initial face sample data includes: Acquire original face sample data; wherein the original face sample data includes an original sample image and a sample image label, and the sample image label is used to characterize whether the original sample image is a valid image or an invalid image; Performing image screening on the original sample image based on the sample image label to obtain a training sample image; wherein the training sample image is the valid image; The training sample images are screened for resolution to obtain the initial face sample data.

6. The method according to claim 1, characterized in that The initial face sample data includes a plurality of initial sample images, and the facial feature extraction is performed on the initial face sample data to obtain sample face features, including: Performing face detection on the initial sample image to obtain a sample face area; Extracting features from the sample face region to obtain initial sample features; The initial sample features are transformed in feature dimension to obtain the sample face features.

7. The method according to claim 6, characterized in that The performing abnormal image cleaning on the initial face sample data based on the sample face features to obtain target face sample data includes: Performing similarity calculation on sample face features of any two of the initial sample images to obtain initial feature similarity data between the two initial sample images, and performing similarity calculation on sample face regions of any two of the initial sample images to obtain sample face region similarity data between the two initial sample images; Performing difference calculation on the initial feature similarity data and the sample face region similarity data between the two initial sample images to obtain a sample face similarity difference between the two initial sample images; If the sample face similarity difference is less than a preset similarity difference threshold, one initial sample image is removed from the two initial sample images to obtain the target face sample data.

8. A face synthesis device, characterized in that: The device comprises: A sample data acquisition module is used to acquire initial face sample data; A sample feature acquisition module, used to extract facial features from the initial facial sample data to obtain sample facial features; An image cleaning module, used for cleaning the initial face sample data for abnormal images based on the sample face features to obtain target face sample data; The original model training module is used to perform model training on a preset original face synthesis model based on the target face sample data and the sample face features to obtain a target face synthesis model; A target image reshaping module is used to reshape the pre-acquired reference face image using the target face synthesis model to obtain target reshaping features; A target image convolution module, used for performing convolution processing on the target reshaping features through the target face synthesis model to obtain target convolution features; The target face synthesis module is used to perform face synthesis on the target convolution features through the target face synthesis model to obtain a target face image.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images