Face image generation method, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application provide a face image generation method, an electronic device and a computer readable storage medium, and relate to the technical field of artificial intelligence. The method comprises: acquiring a source face image of a preset object; inputting the source face image into a preset three-dimensional face model for face modeling processing to obtain a three-dimensional face prior feature; performing feature extraction on the source face image to obtain a source face image feature; inputting the three-dimensional face prior feature and the source face image feature into a preset heterogeneous convolution model for face local feature refinement processing to obtain a target face fusion feature vector; inputting the target face fusion feature vector into a preset generator for face reconstruction to obtain a candidate face image; and adjusting the generator through a perceptual loss between the source face image and the candidate face image to obtain a target face image. In the embodiments, the local detail quality of the face image can be improved while ensuring the identity consistency of the face image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a face image generation method, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In related technologies, image generation is performed through a machine learning method. For example, a generative adversarial network using unsupervised learning is used to generate an image using a generative model G (Generative Model), and a discriminative model D (Discriminative Model) is used to determine whether the image is a real image or a generated image. The generative model G and the discriminative model D compete with each other, and finally reach a convergence effect, so that the generative model G can generate a high-resolution face image. However, when a real image is used and a local feature of the real image is modified, identity consistency is inaccurate. For example, when a face image of a real person is used as a source image, and a target image that changes the local detail features (such as pose features and expression features) of the source image is obtained, the identity features of the target image may change. Therefore, how to provide a face image generation method that can improve the local detail quality of a face image while ensuring the identity consistency of the face image has become a technical problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a face image generation method, an electronic device and a computer readable storage medium, which can improve the local detail quality of a face image while ensuring the identity consistency of the face image.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a face image generation method, which comprises:

[0005] obtaining a source face image of a preset object;

[0006] inputting the source face image into a preset three-dimensional face model for face modeling processing to obtain a three-dimensional face prior feature; wherein the three-dimensional face prior feature is used to represent a face identity feature of the preset object;

[0007] performing feature extraction on the source face image to obtain a source face image feature;

[0008] inputting the three-dimensional face prior feature and the source face image feature into a preset heterogeneous convolution model for face local feature refinement processing to obtain a target face fusion feature vector;

[0009] inputting the target face fusion feature vector into a preset generator for face reconstruction to obtain a candidate face image;

[0010] According to the source face image and the candidate face image, a perceptual loss is calculated, and the generator is adjusted according to the perceptual loss to obtain a target face image; the target face image is used to represent a reconstructed face image of the preset object.

[0011] In some embodiments, the heterogeneous convolution model comprises a splitting block, a first processing block, a second processing block, and a fusion block, and the inputting of the three-dimensional face prior feature and the source face image feature into the preset heterogeneous convolution model for face local feature refinement processing to obtain a target face fusion feature vector comprises:

[0012] The three-dimensional face prior feature and the source face image feature are fused to obtain an initial face fusion feature vector;

[0013] The initial face fusion feature vector is split by the splitting block to obtain a sub-channel initial feature vector; the sub-channel initial feature vector comprises a first sub-channel initial feature vector and a second sub-channel initial feature vector;

[0014] The first sub-channel initial feature vector is processed by the first processing block to obtain a first sub-channel target feature vector;

[0015] The second sub-channel initial feature vector is processed by the second processing block to obtain a second sub-channel target feature vector;

[0016] The first sub-channel target feature vector and the second sub-channel target feature vector are fused by the fusion block to obtain the target face fusion feature vector.

[0017] In some embodiments, the first processing block comprises a first convolution layer, a first pooling layer, and a first activation function layer, and the processing of the first sub-channel initial feature vector by the first processing block to obtain a first sub-channel target feature vector comprises:

[0018] The first sub-channel initial feature vector is processed by the first convolution layer to obtain a first sub-channel linear feature vector;

[0019] The first sub-channel linear feature vector is processed by the first pooling layer to obtain a first sub-channel pooling feature vector;

[0020] The first sub-channel pooling feature vector is processed by the first activation function layer to obtain the first sub-channel target feature vector.

[0021] In some embodiments, the second processing block includes a weight calculation layer, a second convolutional layer, a second pooling layer, and a second activation function layer, the feature processing of the second sub-channel initial feature vector through the second processing block includes:

[0022] The weight calculation layer is used to calculate the weight parameters of the second sub-channel initial feature vector, and the second convolutional layer is adjusted according to the weight parameters to obtain a dynamic convolutional layer;

[0023] The dynamic convolutional layer is used to perform convolutional processing on the second sub-channel initial feature vector to obtain a second sub-channel linear feature vector;

[0024] The second pooling layer is used to perform pooling processing on the second sub-channel linear feature vector to obtain a second sub-channel pooling feature vector;

[0025] The second activation function layer is used to perform activation processing on the second sub-channel pooling feature vector to obtain the second sub-channel target feature vector.

[0026] In some embodiments, the weight calculation layer includes a global average pooling unit, a first full connection unit, a first activation function unit, a second full connection unit, and a second activation function unit connected in sequence, and the weight calculation of the second sub-channel initial feature vector through the weight calculation layer includes:

[0027] The global average pooling unit, the first full connection unit, the first activation function unit, the second full connection unit, and the second activation function unit are sequentially used to calculate the weight parameters of the second sub-channel initial feature vector.

[0028] In some embodiments, the heterogeneous convolutional model further includes a third processing block, and the sub-channel initial feature vector further includes a third sub-channel initial feature vector and a fourth sub-channel initial feature vector, the third processing block includes a first Transformer layer, a third pooling layer, a third activation function layer, a third convolutional layer, a fourth pooling layer, a fourth activation function layer, a fourth convolutional layer, and a fifth activation function layer, and the method further includes: performing feature processing on the third sub-channel initial feature vector and the fourth sub-channel initial feature vector through the third processing block to obtain a third sub-channel target feature vector, specifically including:

[0029] The first Transformer layer, the third pooling layer, and the third activation function layer are sequentially used to extract the first sub-channel latent feature vector from the third sub-channel initial feature vector;

[0030] The fourth sub-channel initial feature vector is sequentially subjected to feature extraction by the third convolutional layer, the fourth pooling layer and the fourth activation function layer to obtain a second sub-channel latent feature vector;

[0031] The first sub-channel latent feature vector and the second sub-channel latent feature vector are spliced, and the spliced feature vector is sequentially subjected to feature extraction by the fourth convolutional layer and the fifth activation function layer to obtain the third sub-channel target feature vector;

[0032] Correspondingly, the vector fusion of the first sub-channel target feature vector and the second sub-channel target feature vector by the fusion block to obtain the target face fusion feature vector comprises:

[0033] The vector fusion of the first sub-channel target feature vector, the second sub-channel target feature vector and the third sub-channel target feature vector by the fusion block to obtain the target face fusion feature vector.

[0034] In some embodiments, the fusion block comprises a splicing layer, a second Transformer layer, a fusion first convolutional layer, a down-sampling layer, a flattening layer, a fusion second convolutional layer and a sixth activation function layer connected in sequence, and the vector fusion of the first sub-channel target feature vector, the second sub-channel target feature vector and the third sub-channel target feature vector by the fusion block to obtain the target face fusion feature vector comprises:

[0035] The first sub-channel target feature vector, the second sub-channel target feature vector and the third sub-channel target feature vector are subjected to vector splicing by the splicing layer to obtain a first spliced face feature vector;

[0036] The first spliced face feature vector is subjected to feature extraction by the second Transformer layer to obtain a second spliced face feature vector;

[0037] The second spliced face feature vector is subjected to channel number increase by the fusion first convolutional layer to obtain a third spliced face feature vector;

[0038] The third spliced face feature vector is subjected to down-sampling processing by the down-sampling layer to obtain a fourth spliced face feature vector;

[0039] The fourth spliced face feature vector is subjected to feature flattening by the flattening layer to obtain a fifth spliced face feature vector;

[0040] The fifth spliced face feature vector is subjected to convolution processing by the fusion second convolutional layer to obtain a sixth spliced face feature vector;

[0041] activating the sixth spliced face feature vector through the sixth activation function layer to obtain the target face fusion feature vector.

[0042] In some embodiments, before inputting the target face fusion feature vector into a preset generator for face reconstruction to obtain a candidate face image, the method further comprises:

[0043] training the generator, specifically comprising:

[0044] obtaining a sample source face image of a preset sample object;

[0045] inputting the sample source face image into a preset three-dimensional face model for face modeling processing to obtain a sample three-dimensional face prior feature; wherein the sample three-dimensional face prior feature is used to represent a face identity feature of the preset sample object;

[0046] extracting features from the sample source face image to obtain a sample source face image feature;

[0047] inputting the sample three-dimensional face prior feature and the source face image feature into a preset heterogeneous convolution model for face local feature refinement processing to obtain a sample target face fusion feature vector;

[0048] inputting the sample target face fusion feature vector into a preset generator for face reconstruction to obtain a sample target face image; the sample target face image is used to represent a reconstructed face image of the preset sample object;

[0049] extracting intermediate features from the sample source face image through a preset feature extraction model to obtain a sample source face intermediate feature; and extracting intermediate features from the sample target face image through the feature extraction model to obtain a sample target face intermediate feature;

[0050] performing mean square error calculation according to the sample source face intermediate feature and the sample target face intermediate feature to obtain image perception loss data;

[0051] adjusting parameters of the generator according to the image perception loss data.

[0052] To achieve the above object, a second aspect of the embodiment of the present application proposes an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the face image generation method of the first aspect when executing the computer program.

[0053] To achieve the above objectives, a third aspect of this application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the face image generation method described in the first aspect.

[0054] This application proposes a face image generation method, electronic device, and computer-readable medium. The method preserves the identity features of the source face image using a 3D face model to obtain 3D face prior features. Then, a heterogeneous convolutional model is used to refine the local features of the 3D face prior features and the source face image features, resulting in a target face fusion feature vector. This target face fusion feature vector is then input into a preset generator for face reconstruction, yielding a candidate face image. Finally, the generator is adjusted based on perceptual loss to obtain the target face image, which represents the reconstructed face image of a preset object. This target face image retains the same identity features as the source face image while refining the local features of the image. In summary, the embodiments of this application, while ensuring the identity consistency of the face image, can modify the local details of the face image, thereby obtaining a high-quality reconstructed face image. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the system architecture for performing a face image generation method provided in an embodiment of this application;

[0056] Figure 2 This is a schematic diagram of the overall framework of the face image generation method provided in the embodiments of this application;

[0057] Figure 3 This is an optional flowchart of the face image generation method provided in the embodiments of this application;

[0058] Figure 4A -D is a visual image example provided in the embodiments of this application;

[0059] Figure 5 yes Figure 3 The flowchart of step S104 in the process;

[0060] Figure 6 yes Figure 3 The flowchart of step S104 in the process;

[0061] Figure 7 yes Figure 6 The flowchart of step S303 in the process;

[0062] Figure 8 yes Figure 6 The flowchart of step S304 in the process;

[0063] Figure 9is Figure 6 the flow chart of step S305 in

[0064] Figure 10 a structural schematic diagram of a heterogeneous convolution model provided by an embodiment of the present application;

[0065] Figure 11 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0067] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flow chart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0069] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] In recent years, the face image generation method using neural networks and machine learning has developed rapidly. In the application aspect, the face image generation method can change some attributes of a face, such as aging, expression, and light direction, by changing the latent code. In the related art, a generative adversarial network using unsupervised learning is used for face image generation. Specifically, a generative model G is used to generate an image, and a discriminative model D is used to determine whether the image is a real image or a generated image. The generative model G and the discriminative model D compete with each other, and finally reach a convergence effect, so that the generative model G can generate a high-resolution face image. However, when a real face image is used and a local feature of the face image is modified, identity consistency is inaccurate. For example, when a real face image is used as a source face image, and a target face image that changes part of the features (for example, pose features and expression features) of the source face image is obtained, the identity features of the target face image may not be consistent with the identity features of the source face image. Therefore, a high-quality face image reconstruction technology is needed to improve the local detail quality of the face image while ensuring the identity consistency of the face image.

[0071] First, some terms involved in the present application are analyzed:

[0072] Prior knowledge: an end-to-end deep neural network is a black box, although it can automatically learn some distinguishable features, it often fits to some non-important features, resulting in the model being locally collapsed to some bad features (for example, in face image reconstruction, the identity features of the target image are changed, and it is difficult to maintain identity consistency). To solve this problem, adding artificially designed prior knowledge to the model can make the model learn some key features.

[0073] For example, in the embodiments of the present application, the source face image is three-dimensionally modeled by a three-dimensional face model, and the three-dimensional face prior features of the source face image are obtained. The three-dimensional prior face features are used as face geometry prior knowledge, and are added to the heterogeneous convolution model together with the source face image for learning, so as to ensure the face identity consistency.

[0074] Heterogeneous convolution model: Since the three-dimensional face prior feature needs to be input into the heterogeneous convolution model together with the source face image in the present application, the calculation amount and the learning difficulty are increased. Therefore, in order to reduce the learning difficulty of the model, a heterogeneous convolution model is introduced. The heterogeneous convolution model of the present application can be a deep heterogeneous network (DHN) existing at present, and the present application also provides another specific composition structure of the heterogeneous convolution model, which will be described in the following embodiments. The heterogeneous convolution model can solve the problems of high cost, large model parameter quantity and low network running efficiency in the training of the traditional method.

[0075] Generator: The generator of the present application specifically refers to a face style generator, such as StyleGAN2 and the like. The "Style" in StyleGAN2 refers to the main attributes of the face in the data set, such as the posture of the person and the like, but not the image style in the style conversion. Here, the Style refers to the style of the face, including the expression on the face shape, the face orientation, the hairstyle and the like, and also including the face skin color, the face illumination and the like on the texture details. StyleGAN2 uses the style to affect the posture and the identity feature of the face, and uses the noise to affect the hair, the wrinkle and the like.

[0076] For example, in the embodiments of the present application, the face reconstruction is performed by the generator, and the target face image will be obtained. The target face image will be different from the source face image in the face style, and since the face prior feature is introduced as the geometric face prior knowledge in advance, the target face image can maintain the identity consistency with the source face image.

[0077] Perception loss: The perception loss proposed in the present application can calculate the intermediate feature loss of the target face image and the source face image by using the pre-trained VGG-19 network, so that the shallow features to the deep features of the target face image are more similar to the shallow features to the deep features of the source face image, and the problem of low quality of the generated image can be solved. In addition, the face image generation method of the present application is suitable for the generation of low-resolution face images and also suitable for the generation of high-resolution face images.

[0078] System architecture and scene description applied in the embodiments of the present application

[0079] Reference Figure 1 , Figure 1Fig. 1 is a schematic diagram of a system architecture for performing the face image generation method provided in the embodiments of the present application; the face image generation method provided in the embodiments of the present application can be performed by the target server 400 alone, or can be performed by the first terminal 100, the second terminal 200 or the third terminal 300 alone, or can be performed by the first terminal 100, the second terminal 200, the third terminal 300 and the target server 400 together. In addition, the face image generation method provided in the embodiments of the present application can also be software running on the target server 400. The target server 400 can be configured as a stand-alone physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN and basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application for implementing the face image generation method, but is not limited to the above forms.

[0080] The face image generation method provided in the embodiments of the present application can be used in many general or specific computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0081] The embodiments of the present application can be applied in various scenarios, including a face age conversion scenario, a face expression conversion scenario, a special shot generation scenario, etc.

[0082] (I) Face age conversion scenario

[0083] At present, artificial intelligence has rapidly penetrated into every corner of people's production and life. People not only expect to perform the tasks needed in production and life through artificial intelligence, but also expect it to provide various value-added services outside the execution of tasks. For example, when a young actor needs to play an old man, aging makeup needs to be done on the young actor, which consumes a lot of time and cost. At this time, the young actor can be taken as a preset object, a source face image is obtained through an intelligent terminal, and a face image generation is performed on the source face image through the face image generation method of the embodiment of the application to obtain a target face image, which is an aging version of the face image of the young actor, so that the cost of makeup and props can be saved. It should be noted that the age conversion referred to in the embodiment of the application can be from small to old, or from old to small.

[0084] (II) Scene of face expression conversion

[0085] For example, a user takes a selfie through a mobile phone to obtain a source face image. However, the expression of the source face image is not satisfactory, and at this time, the expression of the source face image can be converted through the face image generation method of the embodiment of the application to obtain a target face image.

[0086] (III) Scene of special effect shot generation

[0087] For example, in a film and television work, when an actor needs to present a high-risk special effect shot, a high-quality face image containing special effects can be obtained through the face image generation method of the embodiment of the application, which not only ensures the personal safety of the actor, but also avoids the additional cost of using expensive special effects.

[0088] The embodiment of the application provides a face image generation method, an electronic device and a computer readable storage medium, which are specifically described as follows. First, the face image generation method in the embodiment of the application is described.

[0089] It should be noted that in each specific embodiment of the application, when it is necessary to perform relevant processing according to the face image data of the user and other data related to the identity or characteristics of the user, the permission or consent of the user will be obtained first, and the collection, use and processing of these data will comply with the relevant laws, regulations and standards of the country and region.

[0090] In an embodiment, please refer to Figure 2 、 Figure 3 . Figure 2 is a schematic diagram of a whole framework of the face image generation method provided by the embodiment of the application. Figure 3 is an optional flowchart of the face image generation method provided by the embodiment of the application, which can include but is not limited to steps S101 to S106.

[0091] Step S101, obtaining a source face image of a preset object;

[0092] Step S102, inputting the source face image into a preset three-dimensional face model for face modeling processing to obtain a three-dimensional face prior feature; wherein the three-dimensional face prior feature is used to represent a face identity feature of the preset object;

[0093] Step S103, performing feature extraction on the source face image to obtain a source face image feature;

[0094] Step S104, inputting the three-dimensional face prior feature and the source face image feature into a preset heterogeneous convolution model for face local feature refinement processing to obtain a target face fusion feature vector;

[0095] Step S105, inputting the target face fusion feature vector into a preset generator for face reconstruction to obtain a candidate face image;

[0096] Step S106, calculating a perceptual loss according to the source face image and the candidate face image, and adjusting the generator according to the perceptual loss to obtain a target face image; the target face image is used to represent a reconstructed face image of the preset object.

[0097] The steps S101 to S106 shown in the embodiments of the present application are used to solve the problem that the face identity feature and the image detail feature of the generated face are difficult to maintain. The embodiments of the present application retain the identity feature of the source face image through a three-dimensional face model to obtain a three-dimensional face prior feature; then perform local feature refinement processing on the three-dimensional face prior feature and the source face image feature through a heterogeneous convolution model to obtain a target face fusion feature vector; then input the target face fusion feature vector into a preset generator for face reconstruction to obtain a candidate face image; and in order to further improve the image quality, a perceptual loss is introduced to adjust the generator so as to obtain a target face image. The target face image not only retains the identity feature consistent with the source face image, but also refines the local features of the image, such as changing the pose feature, the expression feature, the skin color feature, the face light feature, the skin state feature and the like. In summary, the embodiments of the present application can change the local details of the face image while ensuring the identity consistency of the face image, so as to obtain a high-quality reconstructed face image.

[0098] Please refer to Figure 4A , Figure 4B , Figure 4C and Figure 4D . Figure 4A The source face image provided by the embodiments of the present application is a virtual face synthesized by artificial intelligence technology, not a real face. Figure 4BThe three-dimensional prior knowledge provided in this application embodiment is obtained after face modeling using a three-dimensional face model. Figure 4C The target face image obtained by adding three-dimensional prior knowledge for the embodiments of this application. Figure 4D The reconstructed face image provided in this embodiment of the application is obtained without the addition of 3D prior knowledge. From... Figure 4C and Figure 4D It can be observed that the addition of three-dimensional prior knowledge yields... Figure 4C The target face image can better represent the... Figure 4A The consistency of identity of the source facial image, and Figure 4D Because no three-dimensional prior knowledge is added, identity consistency is difficult to maintain, and Figure 4A The source facial images show significant differences in identity.

[0099] In step S101 of some embodiments, the preset object refers to the object to be reconstructed from the face image, such as a young actor in a scene of face age change, or a user taking a selfie in a scene of face expression change. The source face image refers to the image to be reconstructed from the face.

[0100] Specifically, when the face image generation method is applied to the first terminal 100, the source face image can be acquired through shooting, Bluetooth transmission, wired transmission, or downloading. When the source face image is acquired through shooting, the first terminal 100 is equipped with a camera, and the image is captured using the camera. When the face image generation method is applied to the target server 400, the source face image can be uploaded from the first terminal 100 to the target server 400, or it can be downloaded by the target server 400 from another server.

[0101] In step S102 of some embodiments, the source face image is input into a preset three-dimensional face model for face modeling processing to obtain three-dimensional face prior features; wherein, the three-dimensional face prior features are used to represent: the face identity features of the preset object.

[0102] Specifically, the source face image includes identity features and local detail features, such as pose features and expression features. Facial modeling of the source face image using a 3D face model essentially involves extracting facial identity features from the source face image and using these features as prior geometric knowledge to obtain 3D face prior features. This embodiment introduces 3D face prior features through a 3D face model, solving the problem of preserving the identity features of face images.

[0103] It should be noted that the three-dimensional face model of the embodiments of the present application includes a three-dimensional face statistical model (3D Morphable Models, 3DMM) and a face alignment model (3D Dense Face Alignment, 3DDFA).

[0104] Specifically, the 3DMM divides the source face image into two parts, i.e., a face shape and a face expression, and the linear expression of the source face image is:

[0105]

[0106] wherein, represents an average face shape, s i represents a face shape component of the i-th reference face shape image, e i represents a face expression component of the i-th reference face expression image, a i is a coefficient of the face shape component of the i-th reference face shape image, b i is a coefficient of the face expression component of the i-th reference face expression image, m refers to the number of reference face shape images, and n refers to the number of reference face expression images. The embodiments of the present application take the 3DDFA as the three-dimensional face model, input the source face image into the 3DFFA to predict the shape coefficient and the expression coefficient, i.e., to solve a i and b i . That is, a prior face image containing the shape information of the source face image can be obtained. Therefore, the prior face image can be used to represent the face identity feature of the preset object.

[0107] In step S103 of some embodiments, the source face image can be subjected to feature extraction through a convolutional layer to obtain a source face image feature. The source face image feature is used to represent the identity feature and the local detail feature of the preset object.

[0108] In step S104 of some embodiments, the three-dimensional face prior feature and the source face image feature are input into a preset heterogeneous convolutional model for face local feature refinement processing to obtain a target face fusion feature vector. Specifically, the heterogeneous convolutional model can be a deep heterogeneous network (Deep Heterogeneous Network, DHN). The obtained three-dimensional face prior feature and the source face image feature are jointly used as the input of the deep heterogeneous network, which fully maintains the face identity consistency and greatly reduces the learning difficulty. On the basis of being able to perform face local feature refinement processing, the model processing speed is also improved. This is especially obvious in processing higher image resolution.

[0109] In addition to the DHN, the embodiments of the present application propose a heterogeneous convolution model that can further improve the quality of the reconstructed face image. In some embodiments, the heterogeneous convolution model includes a splitting block, a first processing block, a second processing block, and a fusion block, which are described with reference to Figure 5 Step S104 specifically includes, but is not limited to, steps S201 to S205:

[0110] Step S201: performing feature fusion on the three-dimensional face prior feature and the source face image feature to obtain an initial face fusion feature vector.

[0111] Step S202: performing channel splitting on the initial face fusion feature vector by the splitting block to obtain a sub-channel initial feature vector; the sub-channel initial feature vector includes a first sub-channel initial feature vector and a second sub-channel initial feature vector.

[0112] Step S203: performing feature processing on the first sub-channel initial feature vector by the first processing block to obtain a first sub-channel target feature vector.

[0113] Step S204: performing feature processing on the second sub-channel initial feature vector by the second processing block to obtain a second sub-channel target feature vector.

[0114] Step S205: performing vector fusion on the first sub-channel target feature vector and the second sub-channel target feature vector by the fusion block to obtain a target face fusion feature vector.

[0115] The steps S201 to S205 shown in the embodiments of the present application perform channel splitting on the initial face fusion feature vector to obtain a sub-channel initial feature vector, and the number of channels of the sub-channel initial feature vector is less than the number of channels of the initial face fusion feature vector. In this embodiment, the calculation amount can be reduced and the efficiency of face reconstruction can be improved without loss of face features.

[0116] It should be noted that the number of channels of the sub-channel initial feature vector is determined according to the number of channel splitting. For example, in this embodiment, the number of channel splitting is 2, and the number of channels of each sub-channel initial feature vector is 1 / 2*n (n is the number of channels of the initial face fusion feature vector), so the number of channels of the first sub-channel initial feature vector is 1 / 2*n, and the number of channels of the second sub-channel initial feature vector is 1 / 2*n.

[0117] It should be noted that the heterogeneous convolution model in FIG. 4 includes a splitting block, a first processing block, a second processing block, and a fusion block, but in another embodiment, the heterogeneous convolution model further includes a third processing block, which is described with reference to Figure 6 Step S104 specifically includes, but is not limited to, steps S301 to S305:

[0118] Step S301, fusing the three-dimensional face prior feature and the source face image feature to obtain an initial face fusion feature vector;

[0119] Step S302, performing channel segmentation on the initial face fusion feature vector by a segmentation block to obtain a sub-channel initial feature vector; the sub-channel initial feature vector includes a first sub-channel initial feature vector, a second sub-channel initial feature vector, a third sub-channel initial feature vector, and a fourth sub-channel initial feature vector.

[0120] Step S303, performing feature processing on the first sub-channel initial feature vector by a first processing block to obtain a first sub-channel target feature vector.

[0121] Step S304, performing feature processing on the second sub-channel initial feature vector by a second processing block to obtain a second sub-channel target feature vector.

[0122] Step S305, performing feature processing on the third sub-channel initial feature vector and the fourth sub-channel initial feature vector by a third processing block to obtain a third sub-channel target feature vector.

[0123] Step S306, performing vector fusion on the first sub-channel target feature vector, the second sub-channel target feature vector, and the third sub-channel target feature vector by a fusion block to obtain a target face fusion feature vector.

[0124] The steps S301 to S306 shown in the embodiments of the present application can be understood as follows: the number of channel segmentations in the present embodiment is 4, and the number of channels of each sub-channel initial feature vector is 1 / 4*n, so the number of channels of the first sub-channel initial feature vector is 1 / 4*n, the number of channels of the second sub-channel initial feature vector is 1 / 4*n, the number of channels of the third sub-channel initial feature vector is 1 / 4*n, and the number of channels of the fourth sub-channel initial feature vector is 1 / 4*n. Figure 5 Compared with the embodiments of the prior art, the number of channels of each sub-channel is further reduced in the present embodiment, so that the calculation amount of the model is further reduced, and the efficiency of face reconstruction is improved.

[0125] In step S301 of some embodiments, the three-dimensional face prior feature and the source face image feature can be spliced to obtain an initial face fusion feature vector.

[0126] In step S302 of some embodiments, the split block includes at least two split layers. Specifically, if the split block includes a first split layer and a second split layer, the initial face fusion feature vector is channel-split by the first split layer to obtain a first sub-channel initial feature vector; and the initial face fusion feature vector is channel-split by the second split layer to obtain a second sub-channel initial feature vector. If the split block includes a first split layer, a second split layer, a third split layer and a fourth split layer, the initial face fusion feature vector is channel-split by the first split layer to obtain a first sub-channel initial feature vector; the initial face fusion feature vector is channel-split by the second split layer to obtain a second sub-channel initial feature vector; the initial face fusion feature vector is channel-split by the third split layer to obtain a third sub-channel initial feature vector; and the initial face fusion feature vector is channel-split by the fourth split layer to obtain a fourth sub-channel initial feature vector.

[0127] In some embodiments, the first processing block includes a first convolutional layer, a first pooling layer and a first activation function layer, referring to Figure 7 Step S303 specifically includes but is not limited to steps S401 to S403:

[0128] In step S401, the first sub-channel initial feature vector is convoluted by the first convolutional layer to obtain a first sub-channel linear feature vector;

[0129] In step S402, the first sub-channel linear feature vector is pooled by the first pooling layer to obtain a first sub-channel pooled feature vector;

[0130] In step S403, the first sub-channel pooled feature vector is activated by the first activation function layer to obtain a first sub-channel target feature vector.

[0131] The steps S401 to S403 shown in the embodiments of the present application process the first sub-channel initial feature vector by the first convolutional layer (3*3Conv), the first pooling layer (Pooling) and the first activation function layer (ReLU), which effectively reduces the parameters of the model, avoids overfitting, and is conducive to deepening the network layers to increase the nonlinear expression ability of the network.

[0132] In some embodiments, the second processing block includes a weight calculation layer, a second convolutional layer, a second pooling layer and a second activation function layer, referring to Figure 8 Step S304 specifically includes but is not limited to steps S501 to S504:

[0133] In step S501, the second sub-channel initial feature vector is weighted by the weight calculation layer to obtain a weight parameter, and the second convolutional layer is parameter-adjusted according to the weight parameter to obtain a dynamic convolutional layer;

[0134] In step S502, the second sub-channel initial feature vector is processed by a dynamic convolution layer to obtain a second sub-channel linear feature vector.

[0135] In step S503, the second sub-channel linear feature vector is processed by a second pooling layer to obtain a second sub-channel pooling feature vector.

[0136] In step S504, the second sub-channel pooling feature vector is processed by a second activation function layer to obtain a second sub-channel target feature vector.

[0137] The steps S501 to S504 shown in the embodiments of the present application are as follows: the second sub-channel initial feature vector passes through the weight calculation layer to obtain the weight parameters of the dynamic convolution layer with data dependency, the weight parameters are multiplied by the initial parameters of the second convolution layer to obtain target parameters, and the initial parameters are replaced by the target parameters to obtain the dynamic convolution layer. Then the second sub-channel initial feature vector is processed by the dynamic convolution layer, the second pooling layer and the second activation function layer in sequence, which effectively reduces the parameters of the model, avoids overfitting, and is beneficial to deepening the network layers to increase the nonlinear expression ability of the network.

[0138] In some embodiments, the weight calculation layer includes sequentially connected global average pooling unit, first full connection unit, first activation function unit, second full connection unit and second activation function unit. Step S501 specifically includes:

[0139] The second sub-channel initial feature vector is sequentially processed by the global average pooling unit, the first full connection unit, the first activation function unit (GeLU activation function), the second full connection unit and the second activation function unit (SoftMax activation function) to obtain the weight parameters. In an example, after the second sub-channel initial feature vector passes through the weight calculation layer, four different weight parameters are obtained, and the four weight parameters are multiplied by the initial parameters of the four initial convolution layers to obtain four target convolution layers. Further, the four target convolution layers are combined by weighting to obtain an initial dynamic convolution layer with data dependency. Then the normalization layer and the activation function (GeLU) are continued to obtain the dynamic convolution layer (DynamicConvolution). The second processing block is composed of the dynamic convolution layer, the second pooling layer and the second activation function, which solves the problem of limited network expression ability and low model performance, and can increase the expression ability of the model without increasing the depth or width of the network.

[0140] In some embodiments, the third processing block comprises a first Transformer layer, a third pooling layer, a third activation function layer, a third convolutional layer, a fourth pooling layer, a fourth activation function layer, a fourth convolutional layer, and a fifth activation function layer, with reference to Figure 9 Step S305 specifically comprises, but is not limited to, steps S601 to S603:

[0141] In step S601, the third sub-channel initial feature vector is sequentially subjected to feature extraction by the first Transformer layer, the third pooling layer, and the third activation function layer, to obtain a first sub-channel latent feature vector.

[0142] In step S602, the fourth sub-channel initial feature vector is sequentially subjected to feature extraction by the third convolutional layer, the fourth pooling layer, and the fourth activation function layer, to obtain a second sub-channel latent feature vector.

[0143] In step S603, the first sub-channel latent feature vector and the second sub-channel latent feature vector are spliced, and the spliced feature vector is sequentially subjected to feature extraction by the fourth convolutional layer and the fifth activation function layer, to obtain a third sub-channel target feature vector.

[0144] The steps S601 to S603 shown in the embodiments of the present application, the first Transformer layer, the third pooling layer, and the third activation function layer form a branch, without changing the dimension of the input feature. The third convolutional layer, the fourth pooling layer, and the fourth activation function layer form another branch, for reducing the dimension of the input feature. Through the two branches, the latent information of the feature can be obtained, and finally the third sub-channel target feature vector is obtained.

[0145] The first Transformer layer and the second Transformer layer of the embodiments of the present application each comprise a Transformer structure and an activation function layer (for example, GeLU). The overall architecture of the Transformer structure can comprise an encoding module, a decoding module. The encoding module comprises a multi-head self-attention layer, a residual connection layer, and a normalization layer. The decoding module comprises a multi-head self-attention layer to be masked, a multi-head self-attention layer, a residual connection layer, and a normalization layer.

[0146] It can be understood that the first Transformer layer, the third pooling layer, and the third activation function layer form a first branch, for solving the problem that the feature information is easily lost in the transmission of the network, perfecting the shallow feature, paying attention to the region that should be paid attention to, and suppressing irrelevant information. The third convolutional layer, the fourth pooling layer, and the fourth activation function layer form a second branch, which can reduce the dimension to realize efficient calculation, perform linear combination on different channels to integrate information, and is used to supplement information for the first branch.

[0147] In step S306 of some embodiments, the first sub-channel target feature vector, the second sub-channel target feature vector and the third sub-channel target feature vector are vector fused by a fusion block to obtain a target face fusion feature vector. The fusion block includes a concatenation layer, a second Transformer layer, a fusion first convolutional layer, a down-sampling layer, a flattening layer, a fusion second convolutional layer and a sixth activation function layer connected in sequence, and step S306 specifically includes:

[0148] The first sub-channel target feature vector, the second sub-channel target feature vector and the third sub-channel target feature vector are vector spliced by the concatenation layer to obtain a first spliced face feature vector;

[0149] The first spliced face feature vector is feature extracted by the second Transformer layer to obtain a second spliced face feature vector;

[0150] The second spliced face feature vector is increased in channel number by the fusion first convolutional layer to obtain a third spliced face feature vector;

[0151] The third spliced face feature vector is down-sampled by the down-sampling layer to obtain a fourth spliced face feature vector;

[0152] The fourth spliced face feature vector is feature flattened by the flattening layer to obtain a fifth spliced face feature vector;

[0153] The fifth spliced face feature vector is convolutionally processed by the fusion second convolutional layer to obtain a sixth spliced face feature vector;

[0154] The sixth spliced face feature vector is activated by the sixth activation function layer to obtain the target face fusion feature vector.

[0155] Specifically, the concatenation layer uses a Concat operation to splice different features, and the next layer is the second Transformer layer, the fusion first convolutional layer and the down-sampling layer. Among them, the fusion first convolutional layer doubles the channel number, and the down-sampling operation changes the feature map size to 1 / 2 of the source face image. Referring to Figure 10 , the heterogeneous convolutional model can include four layers, and after 4 times of down-sampling, the feature dimension becomes 1 / 16 of the original feature. After another 4 times of down-sampling, a 64x64x256 feature is obtained. The Flatten operation of the flattening layer is used to flatten the feature, and then the fusion second convolutional layer is used for 3x3 convolutional layer operation, and an 18x512-dimensional feature vector is obtained. Taking this feature vector as the input of the generator, the face reconstruction can be completed. The obtained face image has good identity consistency and image detail preservation, and the resolution can be as high as the original Figure 1 .

[0156] Step S104, input the target face fusion feature vector into a preset generator for face reconstruction to obtain a target face image; the target face image is used to represent a reconstructed face image of the preset object.

[0157] Specifically, the generator of the embodiment of the present application can be a face generation model StyleGAN2 (also referred to as StyleGAN2G), and can also be other deformation models based on the StyleGAN model.

[0158] In another embodiment, the generator can be trained in advance by the following process:

[0159] Obtain a sample source face image of a preset sample object;

[0160] Input the sample source face image into a preset three-dimensional face model for face modeling processing to obtain a sample three-dimensional face prior feature; wherein the sample three-dimensional face prior feature is used to represent a face identity feature of the preset sample object;

[0161] Feature extraction is performed on the sample source face image to obtain a sample source face image feature;

[0162] Input the sample three-dimensional face prior feature and the source face image feature into a preset heterogeneous convolution model for face local feature refinement processing to obtain a sample target face fusion feature vector;

[0163] Input the sample target face fusion feature vector into a preset generator for face reconstruction to obtain a sample target face image; the sample target face image is used to represent a reconstructed face image of the preset sample object;

[0164] Intermediate feature extraction is performed on the sample source face image by a preset feature extraction model to obtain a sample source face intermediate feature; and intermediate feature extraction is performed on the sample target face image by the feature extraction model to obtain a sample target face intermediate feature;

[0165] Calculate the mean square error according to the sample source face intermediate feature and the sample target face intermediate feature to obtain image perception loss data;

[0166] Adjust the parameters of the generator according to the image perception loss data.

[0167] Specifically, the input includes a source face image feature x and a three-dimensional face prior feature The generator output is defined as:

[0168]

[0169] DHN(·) is a heterogeneous convolution model, and G(·) is a generator of StyleGAN2.

[0170] Intermediate features are extracted by a pre-trained VGG-19 network as a feature extraction model, to obtain sample source face intermediate features and sample target face intermediate features. VGG-16 can also be used to extract intermediate features. Specifically, VGG-16 contains 16 hidden layers (13 convolutional layers and 3 fully connected layers), and VGG-19 contains 19 hidden layers (16 convolutional layers and 3 fully connected layers).

[0171] For example, the VGG-19 network is used to calculate the intermediate features of layers 4, 9, 18, 27, and 36 from the sample target face image and the sample source face image. Then, the mean square loss between these intermediate features is calculated as the perceptual loss ， Image perceptual loss data is obtained. The image perceptual loss data is shown in equation (3).

[0172]

[0173] It should be noted that the loss function in the model training process includes not only the image perceptual loss data described above, but also image pixel loss data and image similarity loss data. Specifically, the L2 loss of each pixel is used to extract features from the sample source face image and the sample target face image, and the L2 loss between the features, the sample target face image, and the sample source face image is calculated:

[0174]

[0175] A dedicated recognition loss is also introduced to measure the cosine similarity between the sample target face image and the sample source face image:

[0176]

[0177] Where R is a pre-trained face recognition model: ArcFace network.

[0178] The total loss function of the embodiments of the present application is defined as:

[0179] L(x) = λ1L2(x) + λ2L LPIPS (x) + λ3L ID (x) (6)

[0180] Where λ1, λ2, and λ3 are constants that define the loss weight, and in the embodiments of the present application are set to 1.0, 0.8, and 0.1, respectively. The parameters of the generator are adjusted by the total loss function until the loss function converges, completing the training of the generator.

[0181] The embodiment of the application designs a face image generation method, specifically, face image generation is performed through three different stages. The first stage is to obtain three-dimensional face prior features of the corresponding source face image by using a three-dimensional face model. The three-dimensional face prior features and the source face image are taken as the input of the heterogeneous convolution model of the second stage, the feature fusion is performed by taking the three-dimensional face prior features as the geometric prior knowledge, which greatly reduces the learning difficulty in the subsequent heterogeneous convolution model, and the consistency of the face identity can be fully maintained. The second stage is a heterogeneous convolution model, which reduces the channel number by channel splitting of the input features and processes the features by different processing blocks, greatly reducing the parameter quantity of the model, and since the calculation process is parallel, there is no delay, which effectively improves the calculation efficiency of the network. The third stage is to generate a face image by using a pre-trained StyleGAN2 generator. Here, the multi-layer perception loss is used to calculate the intermediate loss of the target face image and the source face image, which effectively improves the image quality.

[0182] The following is a specific application example of the application:

[0183] Please refer to Figure 10 , Figure 10 The structure diagram of the heterogeneous convolution model provided by the embodiment of the application is shown in the figure. The slice block includes a first slice layer (Slice), a second slice layer (Slice), a third slice layer (Slice), and a fourth slice layer (Slice). The first processing block includes a first convolution layer (3x3Conv), a first pooling layer (Pooling), and a first activation function layer (GeLU). The second processing block includes a dynamic convolution layer (Dynamic Convolution), a second pooling layer (Pooling), and a second activation function layer (GeLU). The third processing block includes a first Transformer layer (Transformer), a third pooling layer (Pooling), a third activation function layer (GeLU), a third convolution layer (1x1Conv), a fourth pooling layer (Pooling), a fourth activation function layer (GeLU), a fourth convolution layer (Conv), and a fifth activation function layer (GeLU). The fusion block includes a concatenation layer (Concat) connected in sequence, a second Transformer layer, a fusion first convolution layer (Conv), a downsampling layer (Downsampling), a flattening layer (Flatten), a fusion second convolution layer (Conv), and a sixth activation function layer (GeLU).

[0184] For example, the source face image size of the preset object is 1024x1024. First, the three-dimensional face 3D model is used to extract the three-dimensional face prior feature of the source face image. After the source face image is subjected to 3x3 convolution, the source face image feature is obtained. The three-dimensional face prior feature and the source face image feature are jointly used as the input of the heterogeneous convolution model. The heterogeneous convolution model includes an input block, a segmentation block, a first processing block, a second processing block, a third processing block, a fusion block, and an output block. The input block is used to fuse the three-dimensional face prior feature and the source face image feature to obtain an initial face fusion feature vector. The segmentation block divides the channel of the initial face fusion feature vector (the feature dimension is 1024x1024x16) into four equal parts to obtain a first sub-channel initial feature vector (the feature dimension is 1024x1024x4), a second sub-channel initial feature vector (the feature dimension is 1024x1024x4), a third sub-channel initial feature vector (the feature dimension is 1024x1024x4), and a fourth sub-channel initial feature vector (the feature dimension is 1024x1024x4). The first processing block is used to process the first sub-channel initial feature vector to obtain a first sub-channel target feature vector. The second processing block is used to process the second sub-channel initial feature vector to obtain a second sub-channel target feature vector. The third processing block is used to process the third sub-channel initial feature vector and the fourth sub-channel initial feature vector to obtain a third sub-channel target feature vector. Then, the fusion block is used to fuse the first sub-channel target feature vector, the second sub-channel target feature vector, and the third sub-channel target feature vector to obtain a target face fusion feature vector (the feature dimension is 18x512). The target face fusion feature vector is used as the input of the StyleGAN2 generator, and the output is a reconstructed face image, that is, the target face image of the preset object is obtained.

[0185] It should be further noted that the embodiments of the present application have at least the following advantages:

[0186] 1) The three-dimensional face prior is obtained by rendering the source face image using a three-dimensional face model, which can solve the problem of maintaining face identity information and reduce the difficulty of network learning.

[0187] 2) The proposed heterogeneous convolutional network includes a split block, a first processing block, a second processing block, a third processing block, and a fusion block, which solves the problems of high cost, large model parameter quantity, and low network running efficiency in the training of traditional methods. The first processing block is composed of a first convolutional layer, a first activation function layer (to avoid insufficient linear expression energy), and a first pooling layer, which solves the problem of large model parameter quantity and increases the nonlinear expression capability of a deeper network. The second processing block is composed of a weight calculation layer, a second convolutional layer, a second activation function layer, and a second pooling layer, which solves the problem of limited network expression capability and low model performance and increases the model expression capability without increasing the network depth or width. The third processing block includes a first Transformer layer, a third activation function layer, a third pooling layer, a third convolutional layer, a fourth activation function layer, a fourth pooling layer, a fourth convolutional layer, and a fifth activation function layer, which solves the problem of easy loss of feature information in the transmission of the network, perfects the shallow features, focuses on the regions that should be focused on, and suppresses irrelevant information. It can also reduce the dimension to realize efficient calculation and integrate the information by linear combination on different channels.

[0188] 3) The proposed perceptual loss is to calculate the intermediate feature loss of the target face image and the source face image by using a pre-trained VGG-19 network, so that the shallow features to deep features of the generated face image are more similar to the shallow features to deep features of the real face image, which is more reasonable and effective than the traditional perceptual loss and solves the problem of low quality of the generated image.

[0189] 4) The present application is suitable for the generation of low-resolution face images and high-resolution face images.

[0190] The embodiments of the present application also provide an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing the connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned video generation method based on a generative adversarial network. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0191] Please refer to Figure 11 , Figure 11 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0192] The processor 701 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute related programs to realize the technical solutions provided by the embodiments of the present application.

[0193] The memory 702 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 702 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 702 and are called and executed by the processor 701 to implement the video generation method based on the generative adversarial network according to the embodiments of the present application;

[0194] The input / output interface 703 is configured to realize information input and output.

[0195] The communication interface 704 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0196] The bus 705 is configured to transmit information between various components (for example, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device.

[0197] The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other through the bus 705 to realize the communication connection between the device.

[0198] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and is configured to store computer readable information. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above face image generation method.

[0199] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0200] The face image generation method, the electronic device and the storage medium provided by the embodiments of the present application retain the identity features of the source face image through the three-dimensional face model to obtain three-dimensional face prior features; then the local feature refinement processing is performed on the three-dimensional face prior features and the source face image features through the heterogeneous convolution model to obtain a target face fusion feature vector; and then the target face fusion feature vector is input into a preset generator for face reconstruction to obtain a target face image. The target face image not only retains the identity features consistent with the source face image, but also refines the local features of the image, such as changing the posture features, expression features, skin color features, face light features, skin state features and the like. In summary, the embodiments of the present application can change the local details of the face image while ensuring the identity consistency of the face image, so as to obtain a high-quality reconstructed face image.

[0201] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0202] Those skilled in the art can understand that, Figures 3-9 The technical solutions shown in the above description do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown, or combine certain steps, or different steps.

[0203] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0204] The terms "first", "second", "third", "fourth" and the like used in the description of the present application and the above drawings (if any) are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0205] It should be understood that in the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0206] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment of the present application according to actual needs.

[0207] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0208] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0209] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A face image generation method, characterized by, The method comprises: obtaining a source face image of a preset object; inputting the source face image into a preset three-dimensional face model for face modeling processing to obtain a three-dimensional face prior feature; wherein the three-dimensional face prior feature is used to represent a face identity feature of the preset object; performing feature extraction on the source face image to obtain a source face image feature; inputting the three-dimensional face prior feature and the source face image feature into a preset heterogeneous convolution model for face local feature refinement processing to obtain a target face fusion feature vector, the heterogeneous convolution model comprising a split block, a first processing block, a second processing block and a fusion block, the first processing block comprising a first convolution layer, a first pooling layer and a first activation function layer, the second processing block comprising a weight calculation layer, a second convolution layer, a second pooling layer and a second activation function layer, the weight calculation layer comprising a global average pooling unit, a first full connection unit, a first activation function unit, a second full connection unit and a second activation function unit connected in sequence; inputting the target face fusion feature vector into a preset generator for face reconstruction to obtain a candidate face image; calculating a perceptual loss according to the source face image and the candidate face image, and adjusting the generator according to the perceptual loss to obtain a target face image; the target face image is used to represent a reconstructed face image of the preset object; wherein the inputting the three-dimensional face prior feature and the source face image feature into a preset heterogeneous convolution model for face local feature refinement processing to obtain a target face fusion feature vector comprises: performing feature fusion on the three-dimensional face prior feature and the source face image feature to obtain an initial face fusion feature vector; performing channel splitting on the initial face fusion feature vector through the split block to obtain a sub-channel initial feature vector; the sub-channel initial feature vector comprises a first sub-channel initial feature vector and a second sub-channel initial feature vector; performing convolution processing on the first sub-channel initial feature vector through the first convolution layer to obtain a first sub-channel linear feature vector, performing pooling processing on the first sub-channel linear feature vector through the first pooling layer to obtain a first sub-channel pooling feature vector, and performing activation processing on the first sub-channel pooling feature vector through the first activation function layer to obtain a first sub-channel target feature vector; sequentially performing weight calculation on the second sub-channel initial feature vector through the global average pooling unit, the first full connection unit, the first activation function unit, the second full connection unit and the second activation function unit to obtain a weight parameter; The second convolutional layer is parameter adjusted according to the weight parameter, a dynamic convolutional layer is obtained, the second sub-channel initial feature vector is processed through convolution by the dynamic convolutional layer, a second sub-channel linear feature vector is obtained, the second sub-channel linear feature vector is processed through the second pooling layer, a second sub-channel pooling feature vector is obtained, and the second sub-channel pooling feature vector is processed through the second activation function layer, a second sub-channel target feature vector is obtained. The first sub-channel target feature vector and the second sub-channel target feature vector are fused through the fusion block to obtain the target face fusion feature vector.

2. The method of claim 1, wherein, The heterogeneous convolutional model further comprises a third processing block, the sub-channel initial feature vector further comprises a third sub-channel initial feature vector and a fourth sub-channel initial feature vector, the third processing block comprises a first Transformer layer, a third pooling layer, a third activation function layer, a third convolutional layer, a fourth pooling layer, a fourth activation function layer, a fourth convolutional layer and a fifth activation function layer, and the method further comprises: processing the third sub-channel initial feature vector and the fourth sub-channel initial feature vector through the third processing block to obtain a third sub-channel target feature vector, specifically comprising: processing the third sub-channel initial feature vector through the first Transformer layer, the third pooling layer and the third activation function layer in sequence to obtain a first sub-channel latent feature vector; processing the fourth sub-channel initial feature vector through the third convolutional layer, the fourth pooling layer and the fourth activation function layer in sequence to obtain a second sub-channel latent feature vector; splicing the first sub-channel latent feature vector and the second sub-channel latent feature vector, and processing the spliced feature vector through the fourth convolutional layer and the fifth activation function layer in sequence to obtain the third sub-channel target feature vector; correspondingly, the first sub-channel target feature vector and the second sub-channel target feature vector are fused through the fusion block to obtain the target face fusion feature vector, comprising: the first sub-channel target feature vector, the second sub-channel target feature vector and the third sub-channel target feature vector are fused through the fusion block to obtain the target face fusion feature vector.

3. The method of claim 2, wherein, The fusion block comprises a splicing layer, a second Transformer layer, a fusion first convolutional layer, a down-sampling layer, a flattening layer, a fusion second convolutional layer and a sixth activation function layer connected in sequence, and the first sub-channel target feature vector, the second sub-channel target feature vector and the third sub-channel target feature vector are fused through the fusion block to obtain the target face fusion feature vector, comprising: the first sub-channel target feature vector, the second sub-channel target feature vector and the third sub-channel target feature vector are spliced through the splicing layer to obtain a first spliced face feature vector; The first spliced face feature vector is subjected to feature extraction through the second Transformer layer to obtain a second spliced face feature vector; The second spliced face feature vector is subjected to an increase in the number of channels through the fusion first convolutional layer to obtain a third spliced face feature vector; The third spliced face feature vector is subjected to down-sampling processing through the down-sampling layer to obtain a fourth spliced face feature vector; The fourth spliced face feature vector is subjected to feature flattening through the flattening layer to obtain a fifth spliced face feature vector; The fifth spliced face feature vector is subjected to convolution processing through the fusion second convolutional layer to obtain a sixth spliced face feature vector; The sixth spliced face feature vector is subjected to activation processing through the sixth activation function layer to obtain the target face fusion feature vector.

4. The method according to any one of claims 1 to 3, characterized in that, Before inputting the target face fusion feature vector into a preset generator for face reconstruction to obtain a candidate face image, the method further comprises: training the generator, specifically comprising: obtaining a sample source face image of a preset sample object; inputting the sample source face image into a preset three-dimensional face model for face modeling processing to obtain a sample three-dimensional face prior feature; wherein the sample three-dimensional face prior feature is used to represent a face identity feature of the preset sample object; extracting features from the sample source face image to obtain a sample source face image feature; inputting the sample three-dimensional face prior feature and the source face image feature into a preset heterogeneous convolutional model for face local feature refinement processing to obtain a sample target face fusion feature vector; inputting the sample target face fusion feature vector into a preset generator for face reconstruction to obtain a sample target face image; the sample target face image is used to represent a reconstructed face image of the preset sample object; extracting intermediate features from the sample source face image through a preset feature extraction model to obtain a sample source face intermediate feature; and extracting intermediate features from the sample target face image through the feature extraction model to obtain a sample target face intermediate feature; performing mean square error calculation according to the sample source face intermediate feature and the sample target face intermediate feature to obtain image perception loss data; adjusting parameters of the generator according to the image perception loss data.

5. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the face image generation method of any one of claims 1 to 4 when executing the computer program.

6. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the face image generation method of any one of claims 1 to 4.

Images