Face image generation method, device, electronic device and storage medium

Abstract

The present invention relates to the field of image processing, and provides a method, device, electronic device and storage medium for generating a facial image. An original image including the initial expression of a target person and an expression map including the target expression are obtained; then a mask map, a first image and a second image are obtained according to the original image and the expression map through a preset model, wherein the first image includes the facial features of the target person that match the target expression, and the second image includes the head features of the target person that match the target expression; then the first image and the second image are fused according to the mask map through a preset model to obtain a facial image, and the virtual expression of the target person in the facial image matches the target expression. Thus, facial expression migration is achieved, and based on the facial features and the head features, facial features can be better matched, the generated facial image can be made more realistic, and the generated effect is better.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device, electronic device and storage medium for generating a face image. Background Art

[0002] With the development of face recognition technology, facial expression can be identified from a face image and transferred to another face image. Moreover, with the popularization of video entertainment, this facial expression transfer technology can meet the personalized needs of users.

[0003] In the existing technology, it is usually implemented based on a deep neural network model, but due to the limited training data, the generated face effect is poor. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a facial image generation method, device, electronic device and storage medium.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, the present invention provides a method for generating a face image, which is applied to an electronic device, and the method comprises:

[0007] Acquire an original image and an expression image; the original image includes the initial expression of the target person; the expression image includes the target expression;

[0008] By using a preset model, a mask image, a first image, and a second image are obtained according to the original image and the expression image; the first image includes facial features of the target person that match the target expression; the second image includes head features of the target person that match the target expression;

[0009] The first image and the second image are fused according to the mask image through the preset model to obtain a facial image; the facial image includes a virtual expression of the target person, and the virtual expression matches the target expression.

[0010] In an optional implementation, the step of obtaining the mask image, the first image, and the second image according to the original image and the expression image by using a preset model includes:

[0011] Using the face key point network of the preset model, the face key points of the original image and the expression image are detected to obtain the key points of the original image and the key points of the expression image;

[0012] Through the feature encoding network of the preset model, feature extraction is performed on the original image, the key points of the original image, and the key points of the expression map to obtain the original image features, the original image key point features, and the expression map key point features;

[0013] By using the preset model, the mask image and the first image are obtained respectively according to the original image features, the key point features of the original image and the key point features of the expression image;

[0014] The second image is obtained through the preset model according to the original image, the key point features of the original image and the key point features of the expression image.

[0015] In an optional implementation, the step of obtaining the mask image by using the preset model according to the original image features, the original image key point features and the expression image key point features includes:

[0016] The mask image is obtained through the mask prediction network of the preset model according to the original image features, the key point features of the original image and the key point features of the expression image; the mask image includes a key area and a non-key area; the key area is used to represent the area where the facial features of the target person are located.

[0017] In an optional implementation, the step of obtaining the first image by using the preset model according to the original image features, the original image key point features and the expression image key point features includes:

[0018] The first image is obtained through the generation network of the preset model according to the original image features, the key point features of the original image and the key point features of the expression image; the resolution of the first image is a preset resolution, and the preset resolution is smaller than the resolution of the original image.

[0019] In an optional implementation, the step of obtaining the second image by using the preset model according to the original image, the key point features of the original image, and the key point features of the expression image includes:

[0020] Obtaining optical flow information according to the key point features of the original image and the key point features of the expression map through the optical flow prediction network of the preset model; the optical flow information represents the displacement information of the key point features of the original image and the key point features of the expression map;

[0021] The original image is deformed according to the optical flow information to obtain the second image.

[0022] In an optional implementation manner, the preset model is trained in the following manner:

[0023] Inputting training samples into the basic model to obtain a target image; the training samples are two face video frames preset in the same face video; the training samples include sample labeling information;

[0024] Calculating a loss function according to the target image and the sample label information;

[0025] Optimizing the parameters of the basic model according to the loss function;

[0026] When the preset number of iterations is reached, the preset model is obtained.

[0027] In an optional implementation, the step of obtaining the original image and the expression image includes:

[0028] In response to a user's selection operation on a video template interface, a target video template corresponding to the selection operation is determined, and a picture acquisition interface is displayed; the video template interface includes at least one video template; each video frame of the target video template is the expression picture;

[0029] In response to the user's interactive operation on the image acquisition interface, a target image corresponding to the interactive operation is acquired; the target image is the original image.

[0030] In a second aspect, the present invention provides a facial image generation device, which is applied to an electronic device, and the device comprises:

[0031] An acquisition module, used to acquire an original image and an expression image; the original image includes the initial expression of the target person; the expression image includes the target expression;

[0032] A processing module, configured to obtain a mask image, a first image, and a second image according to the original image and the expression image through a preset model; the first image includes facial features of the target person that match the target expression; the second image includes head features of the target person that match the target expression;

[0033] The first image and the second image are fused according to the mask image through the preset model to obtain a facial image; the facial image includes a virtual expression of the target person, and the virtual expression matches the target expression.

[0034] In a third aspect, the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in any one of the aforementioned embodiments is implemented.

[0035] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any one of the aforementioned embodiments.

[0036] The embodiments of the present invention provide a method, device, electronic device and storage medium for generating a facial image. An original image including the initial expression of a target person and an expression map including the target expression are obtained; then, a mask map, a first image and a second image are obtained according to the original image and the expression map through a preset model, wherein the first image includes the facial features of the target person that match the target expression, and the second image includes the head features of the target person that match the target expression; and then, the first image and the second image are fused according to the mask map through a preset model to obtain a facial image, and the virtual expression of the target person in the facial image matches the target expression. Thus, facial expression migration is achieved, and based on the facial features and head features, facial features can be better matched, the generated facial image can be made more realistic, and the generated effect is better.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A block diagram of an electronic device provided by an embodiment of the present invention is shown;

[0040] Figure 2 A schematic diagram of a process of generating a facial image provided by an embodiment of the present invention is shown;

[0041] Figure 3 An example diagram of a method for generating a face image provided by an embodiment of the present invention is shown;

[0042] Figure 4 Another schematic diagram of a process of generating a facial image according to an embodiment of the present invention is shown;

[0043] Figure 5 Another example diagram of the method for generating a face image provided by an embodiment of the present invention is shown;

[0044] Figure 6 Another example diagram of the method for generating a face image provided by an embodiment of the present invention is shown;

[0045] Figure 7Another schematic diagram of a process of generating a facial image according to an embodiment of the present invention is shown;

[0046] Figure 8 Another example diagram of the method for generating a face image provided by an embodiment of the present invention is shown;

[0047] Fig. 9 A functional module diagram of a facial image generating device provided by an embodiment of the present invention is shown.

[0048] Icon: 100 - electronic device; 110 - bus; 120 - processor; 130 - memory; 150 - I / O module; 170 - communication interface; 300 - face image generating device; 310 - acquisition module; 330 - processing module; 350 - training module. DETAILED DESCRIPTION

[0049] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0051] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0052] Please refer to Figure 1 , is a block diagram of an electronic device 100 provided by an embodiment of the present invention. The electronic device 100 includes a bus 110 , a processor 120 , a memory 130 , an I / O module 150 , and a communication interface 170 .

[0053] The bus 110 may be a circuit that connects the above elements to each other and transfers communications (eg, control messages) between the above elements.

[0054] The processor 120 may receive commands from the other elements described above (eg, the memory 130 , the I / O module 150 , the communication interface 170 , etc.) through the bus 110 , may interpret the received commands, and may perform calculations or data processing according to the interpreted commands.

[0055] The processor 120 may be an integrated circuit chip with signal processing capabilities. The processor 120 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0056] The memory 130 may store commands or data received from the processor 120 or other elements (eg, the I / O module 150 , the communication interface 170 , etc.) or commands or data generated by the processor 120 or other elements.

[0057] The memory 130 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0058] The I / O module 150 may receive commands or data input from a user via input-output means (e.g., a sensor, a keyboard, a touch screen, etc.), and may transmit the received commands or data to the processor 120 or the memory 130 via the bus 110. It is also used to display various information (e.g., multimedia data, text data) received, stored, and processed from the above elements, and may display videos, images, data, etc. to the user.

[0059] The communication interface 170 may be used to communicate signaling or data with other node devices.

[0060] Understandably, Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0061] The electronic device provided in the embodiment of the present invention may be a smart phone, a personal computer, a tablet computer, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present invention does not impose any limitation on this.

[0062] It is understandable that an application capable of generating short videos is installed in the electronic device 100, and during operation of the application, the method for generating facial images provided by the embodiment of the present invention can be implemented. It should be noted that each video frame in the short video of the application can be obtained by the method for generating facial images, and then a short video can be obtained based on multiple generated facial images.

[0063] The following is an introduction to the face image generation method provided by the embodiment of the present invention. Figure 2 , Figure 2 It is a flowchart of a method for generating a facial image provided by an embodiment of the present invention.

[0064] Step S202, obtaining the original image and the expression image;

[0065] The original image includes a face of a person, and this person is the target person. The expression of the face in the original image is the initial expression. The person in the expression image may be different from the person in the original image, or may be the same person, and the expression of the face in the expression image is the target expression.

[0066] It is understandable that different characters have different facial features; the same character has different facial features when in different expression states.

[0067] It should be noted that, in order to better understand the present invention, the embodiments of the present invention are described by taking the original image and the expression image as examples in which the characters are different and the expression states of the characters in the images are also different, which does not limit the implementation of the present invention.

[0068] Optionally, the original image may be a picture obtained according to an interactive operation of the user, and the expression image may be obtained according to a pre-stored video template.

[0069] Step S204, obtaining a mask image, a first image, and a second image according to the original image and the expression image through a preset model;

[0070] It is understandable that the facial expression of a person can be reflected by the facial features and head features of the person, such as crying, laughing, etc., or head features such as lowering the head, turning the head, etc. The initial expression in the original image has corresponding facial features and head features, and the target expression in the expression map also has corresponding facial features and head features.

[0071] Among them, the preset model is a pre-trained model, which will be introduced in the following embodiments.

[0072] Optionally, the original image and the expression image are used as inputs of the preset model, and the mask image, the first image and the second image can be obtained through the preset model.

[0073] The first image includes the facial features of the target person that match the target expression, which can be understood as the first image including the facial features of the target person, and the facial features are the same as the target expression. For example, if the target expression is crying, the facial features in the first image are also crying.

[0074] The second image includes the head features of the target person that match the target expression, which can be understood as the second image including the head features of the target person, which are the same as the target expression. For example, if the target expression is bowing the head, the head features in the second image are also bowing the head.

[0075] Step S206, fusing the first image and the second image according to the mask image using a preset model to obtain a face image;

[0076] The face image includes the face of the target person, and the expression of the face in the face image is the virtual expression, which matches the target expression.

[0077] It can be understood that the mask image obtained in the above steps is used to control the processing area of ​​the image. Based on the mask image, the first image and the second image are fused to control the processing areas in the two images respectively.

[0078] Optionally, based on the mask image, during the fusion process, the area where the facial features in the first image are located and the area where the head features in the second image are located can be obtained, so that the expression of the face in the generated face image matches the target expression.

[0079] It can be seen that based on the above design, by obtaining an original image including the initial expression of the target person, and an expression map including the target expression; then, through a preset model, according to the original image and the expression map, a mask map, a first image and a second image are obtained, wherein the first image includes the facial features of the target person that match the target expression, and the second image includes the head features of the target person that match the target expression; then, through a preset model, the first image and the second image are fused according to the mask map to obtain a face image, and the virtual expression of the target person in the face image matches the target expression. Thus, facial expression migration can be achieved, and based on the facial features and head features, the facial features can be better matched, the generated face image can be made more realistic, and the generated effect is better.

[0080] See also Figure 3 , is an example diagram provided by an embodiment of the present invention, which exemplarily shows the structure of the preset model provided by an embodiment of the present invention. The preset model includes a face key point network, a feature encoding network, a mask prediction network, a generation network, an optical flow prediction network and an image fusion submodule.

[0081] The following will be based on this preset model, combined with Figure 3 and Figure 4 The face image generation method provided by the embodiment of the present invention is introduced. Figure 4 , step S204 may further include the following steps:

[0082] Step S204-1, performing facial key point detection on the original image and the expression image through a facial key point network of a preset model to obtain the key points of the original image and the key points of the expression image;

[0083] Among them, facial key point detection refers to locating the key areas of the face, including eyebrows, eyes, nose, mouth, facial contours and other key parts.

[0084] Optionally, through the face key point network, the position of the key parts of the face in the original image when the target person in the original image is in the initial expression can be determined, and rasterization processing can be performed to obtain the key points of the original image.

[0085] Through the face key point network, the positions of the key parts of the face in the expression map can be determined, and rasterization processing can be performed to obtain the key points of the expression map.

[0086] Rasterization refers to converting a geometric image into a two-dimensional image, that is, converting an image into an image composed of raster.

[0087] Step S204-3, extracting features of the original image, key points of the original image, and key points of the expression map through a feature encoding network of a preset model to obtain features of the original image, features of key points of the original image, and features of key points of the expression map;

[0088] Among them, feature extraction is to determine the features of the face, which include the size and shape of the facial parts;

[0089] Optionally, by extracting features from the original image through a feature encoding network, the shape and size of each part of the face of the target person in the original image when the person has an initial expression can be obtained to obtain the features of the original image.

[0090] By extracting the key points of the original image through the feature encoding network, the shape and size of the key parts of the face of the target person in the original image with the initial expression can be obtained, and the key point features of the original image can be obtained.

[0091] By extracting the key points of the expression graph through the feature encoding network, the shape and size of the key parts of the face in the expression graph can be obtained.

[0092] After obtaining the original image features, the original image key point features and the expression image key point features, the mask image, the first image and the third image can be obtained through different network layers. It can be understood that the order of obtaining these three images can be designed according to actual needs, and the embodiment of the present invention does not limit it.

[0093] See also Figure 5 , Figure 5 (a) in the figure is the original image. Figure 5 (c1), (c2) and (c3) are the mask image, the first image and the second image obtained based on the original image and the expression image, respectively. Figure 5 (d) in the figure is a face image generated based on the original image and the expression image. It should be noted that Figure 5 It is only an exemplary display, which is used to illustrate the principles and technical effects implemented by the present invention and does not involve real information.

[0094] Step S204-5, obtaining a mask image according to the original image features, the original image key point features and the expression image key point features through a preset model;

[0095] Optionally, through the mask preset network of the preset module, the original image features, the original image key point features and the expression image key point features can be compared and calculated to obtain a mask image.

[0096] It is understandable that the pixel value of the mask image obtained by the mask preset network can be 0 or 1. Based on the pixel value, the mask image can be divided into a key area and a non-key area, for example, the area with a pixel value of 1 is the key area, and the area with a pixel value of 0 is the non-key area.

[0097] It is understandable that the expression state of a person's face is mainly reflected by the features of the facial features. The mask image can be used to represent the area of ​​the original image, and the key area in the mask image can be used to represent the area where the facial features are located in the original image, thereby determining the locations of the facial features.

[0098] like Figure 5 In (c1), the mask image includes two areas. The key area represents the area where the facial features are located in the original image, and the non-key area represents other areas in the original image.

[0099] Since the mask image can be used to represent the area information where the features are located in the original image, based on the information of the area where the features are located, other network layers in the model, such as the generative network, can be used to make the generative network focus on the key areas, that is, the areas where the facial features are located, so as to obtain more realistic facial features. The specific introduction will be explained in the model training section below.

[0100] Step S204-7, obtaining a first image through a preset model according to the original image features, the original image key point features and the expression image key point features;

[0101] Optionally, the original image features, the original image key point features and the expression image key point features can be compared and calculated through the generation network of the preset module to obtain the first image. Optionally, the generation network can adopt a VGG network.

[0102] It can be understood that the generation network is used to generate facial features of the target person that match the target expression, and includes facial features that are not in the original image.

[0103] like Figure 5 (c2) in the figure is the first image obtained based on the original image and the expression. The mouth of the target person in the original image is merged and has no features inside the mouth. The first image generated by the generative network based on the target expression in the driving image has an open mouth and includes features inside the mouth.

[0104] Optionally, when designing the generation network, the resolution of the generated first image can be set to be smaller than the resolution of the original image, so that the generation network focuses on the facial features that need to be generated.

[0105] Step S204-9, obtaining a second image according to the original image, key point features of the original image, and key point features of the expression image through a preset model;

[0106] Optionally, optical flow information can be obtained according to the key point features of the original image and the key point features of the expression map through an optical flow prediction module of a preset model, and the optical flow information represents the displacement between the key point features of the original image and the key point features of the expression map.

[0107] Optical flow is used to represent the change information of an image. Different facial expressions will result in different facial features, and optical flow can be used to represent the change information of facial features. Based on the optical flow information, the original image is deformed to obtain a second image.

[0108] like Figure 5 (c3) in the figure is a second image obtained based on the original image and the expression image. The second image includes the background in the original image and the face after deformation, and the face includes the head features of the target object, such as hair and portrait outline.

[0109] It can be understood that the second image is obtained by deforming the original image, and the second image has the same resolution as the original image.

[0110] Step S206, fusing the first image and the second image according to the mask image using a preset model to obtain a face image;

[0111] Optionally, the resolutions of the mask image and the first image are both smaller than the resolution of the original image. The mask image and the first image can be enlarged by bilinear interpolation to obtain an enlarged mask image and an enlarged first image, so that their resolutions are the same as the resolution of the original image and the resolution of the second image.

[0112] It will be appreciated that each image can be represented by an image matrix of that image.

[0113] Optionally, a first fusion matrix may be obtained by performing a dot multiplication of an image matrix of the enlarged mask image and an image matrix of the enlarged first image. The first fusion matrix includes information of facial features in the first image.

[0114] The image matrix of the enlarged mask image is inverted, and then dot-multiplied with the image matrix of the second image to obtain a second fusion matrix. The second fusion matrix includes information other than facial features in the second image, namely, head feature information and background information.

[0115] By adding the first fusion matrix and the second fusion matrix, a facial image can be obtained, which includes a virtual expression of the target person, that is, an expression generated by a preset model, and the expression matches the target expression. Figure 5 (d) in the figure is a face image generated based on the original image and the expression image.

[0116] It is understandable that the facial expression state can be reflected by facial features and head features. And the change range of these two features is different. The change range of facial features is small, while the change range of head features is large. In the prior art, deep neural networks are usually used to generate facial features, but due to the differences in the change range of features, the generated features cannot achieve the same effect at the same time.

[0117] Based on the above design, the embodiment of the present invention obtains a mask image through a mask prediction network, and the mask image includes key areas and non-key areas; applies the generation network to facial features with a small change range to generate a first image; for head features with a large change range, optical flow estimation and image deformation processing are used to obtain a second image; through the key areas and non-key areas in the mask image, the facial features in the first image and the head features in the second image are obtained, and the images are fused to obtain a face image. This can improve the matching degree of facial features, making the generated face image more realistic.

[0118] As mentioned in the above steps, the preset model provided in the embodiment of the present invention is a pre-trained model. Figure 6 and Figure 7 , introducing a possible implementation method for obtaining the preset model.

[0119] The preset model provided in the embodiment of the present invention is obtained by adopting an end-to-end training mode. End-to-end means that during the training process of the model, sample data will be transmitted from the input end to the output end, and a prediction result will be obtained. There is an error between the prediction result and the actual result. This error will be transmitted to each layer in the model, and each layer will be adjusted based on this error until the model achieves the expected effect. Based on the end-to-end training mode, the training steps will be introduced below.

[0120] See also Figure 7 , step S212, inputting the training sample into the basic model to obtain the target image;

[0121] Among them, the basic model is a model that has not been trained;

[0122] Optionally, two video frames may be captured from a preset video of the same face as training samples, and the two video frames both include the face of the same person, but the facial expressions in the two video frames are different.

[0123] It can be understood that in order to distinguish the two video frames, one of the video frames is called the source image and the other video frame is called the driving image. The expression of the face in the source image is the expression before migration, and the expression of the face in the driving image is the expression after migration.

[0124] Optionally, the source image and the driving image are input into the base model, and the output is the target image.

[0125] It can be understood that, for the sake of ease of description, the input of the mask prediction module is called a mask map; the output of the generation network is called a low-definition map; and the image obtained by deforming the source map based on optical flow is called a high-definition map.

[0126] Step S214, calculating the loss function according to the target image and sample label information;

[0127] Wherein, the training sample includes sample labeling information;

[0128] Optionally, after obtaining the target image, the loss function can be calculated in combination with the sample labeling information of the training samples.

[0129] Step S216, optimizing the parameters of the basic model according to the loss function;

[0130] Optionally, according to the loss function, the parameters of each network layer in the base model can be adjusted to optimize the model.

[0131] Step S218: When the preset number of iterations is reached, a preset model is obtained.

[0132] Optionally, the convergence condition of the model training is the number of iterations. When the number of training reaches a preset number of iterations, the current basic model is used as the preset model.

[0133] Since the loss function plays an important role in model training, the loss function of the model will be introduced below.

[0134] 1. Distance loss: L1 = ||XG(z)|| 1 ;

[0135] Among them, L1 represents the distance loss function; X represents the driving map; G(z) represents the target image.

[0136] 2. Perceptual loss:

[0137] Among them, L vgg represents the perceptual loss function; Represents features in the driving graph; Represents the features in the target image; j represents the jth layer of the generated network, which uses the VGG network.

[0138] 3. Identify loss:

[0139]

[0140] Among them, L p represents the discriminant loss function; D represents the discriminator; G represents the generator; X represents the driving map; G(z) represents the target image; D(X) represents the discriminant result of the driving map; D(G(z)) represents the discriminant result of the target image; E(X) represents the expectation of X; E(z) represents the expectation of z; z represents the input of the generator.

[0141] It can be understood that during the training process, the generation network includes a discriminator and a generator.

[0142] 4. Loss of optical flow prediction network: Given the optical flow δ∈R H*W*2 , its regular term, i.e. TVloss function, is:

[0143] The TVloss function is applied to the optical flow to limit the smoothness of the optical flow and avoid mutations in the image.

[0144] 5. Loss of mask prediction network:

[0145] Among them, L mask represents the mask loss function; M represents the mask map; m i,j Represents the value of the pixel in the i-th row and j-th column in the mask image M.

[0146] The mask loss function is used to avoid the pixel values ​​of the generated mask image being all 1, thus avoiding the target image being low-definition.

[0147] 6. Loss of the generated network: L gen =||G(z) i,j *(1-M)||;

[0148] Among them, L gen represents the loss function of the generated network; M represents the mask map; G(z) i,j Indicates a low-resolution image.

[0149] The generative network loss function is used to train the generative network so that the generative network can focus on the image information corresponding to the key areas to improve the generation quality of low-definition images.

[0150] Combining the above loss functions, the total loss function of the model can be obtained as:

[0151] L total =λ1L1+λ vgg L vgg +λ p L p +λ TV L TV +λ mask L mask +λ gen L gen ;

[0152] Among them, λ1, λ vgg , p , TV , mask , genThey respectively represent the adjustable weight parameters of the corresponding loss function.

[0153] In the above embodiments, it is introduced that the face image generation method improved by the embodiments of the present invention can be applied in an application for generating short videos to generate multiple face images to obtain short videos. Furthermore, the embodiments of the present invention provide a possible implementation method. Figure 8 , is an example diagram of an interaction provided by an embodiment of the present invention.

[0154] like Figure 8 (u1) in FIG. 1 is an example diagram of a video template interface provided by an embodiment of the present invention. The interface includes multiple pre-stored video templates, each of which includes the face of the same person. The video presents the process of changes in facial expressions.

[0155] When the user's selection operation is detected on the interface, the target video template corresponding to the selection operation, namely video template 4, is determined. Then display Figure 8 (u2) in the video template interface includes a template preview area, which is used to show the user a video of the changes in the facial expressions of the characters in the target video template, that is, to play the video template 4.

[0156] The video template interface also includes a template confirmation control. When a confirmation operation of the user on the template confirmation control is detected, each video frame in the target template video is obtained, and each video frame is used as an expression image, that is, the expression image is obtained from the video template 4; and the image acquisition interface is displayed, as shown in (u3) of Figure 8.

[0157] The picture acquisition interface includes an upload prompt area and a picture upload area. The upload prompt area is used to display prompt information for uploading pictures to the user.

[0158] When a user's interactive operation in the image upload area is detected, such as a selection operation performed by the user in the local album of the electronic device, or a photo operation performed by the user through the electronic device, the target image corresponding to the interactive operation is obtained, which is the original image.

[0159] The image acquisition interface also includes a video generation control. When a user confirms the operation of the video generation control, the face image generation method described in the above embodiment is executed according to the original image and expression image obtained to obtain multiple face images, and a short video is synthesized according to the multiple face images; and a video preview interface is displayed, such as Figure 8 As shown in (u4) in .

[0160] The video preview interface includes a video playback area and a video interaction area. The video playback area is used to play the short video synthesized based on multiple face images, and the video interaction area is used to interact with users to realize interactive functions such as commenting and sending barrages.

[0161] It can be seen that based on the above design, based on the picture and video template selected by the user, the picture includes the face of the target person, and the video template includes multiple expression pictures, which show the expression change process of the face; through the face image generation method provided by the embodiment of the present invention, multiple face images can be generated, and the face images can be synthesized into a short video, which shows the face of the target person, and the expression changes of the face match the expression changes in the video template. Thus, the video entertainment needs of users can be met.

[0162] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a facial image generation device is given below. Fig. 9 , Fig. 9 This is a functional module diagram of a facial image generation device 300 provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the facial image generation device 300 provided in this embodiment are the same as those of the above embodiments. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above embodiments. The facial image generation device 300 includes: an acquisition module 310 and a processing module 330.

[0163] The acquisition module 310 is used to acquire an original image and an expression image; the original image includes the initial expression of the target person; the expression image includes the target expression;

[0164] The processing module 330 is used to obtain a mask image, a first image, and a second image according to the original image and the expression image through a preset model; the first image includes facial features of a target person that matches a target expression; the second image includes head features of the target person that matches the target expression;

[0165] Through a preset model, the first image and the second image are fused according to the mask image to obtain a face image; the face image includes a virtual expression of a target person, and the virtual expression matches the target expression.

[0166] Optionally, the processing module 330 is further used to: perform facial key point detection on the original image and the expression image through a facial key point network of a preset model to obtain the key points of the original image and the key points of the expression image;

[0167] Through the feature encoding network of the preset model, feature extraction is performed on the original image, key points of the original image, and key points of the expression map to obtain the original image features, the original image key point features, and the expression map key point features;

[0168] Through the preset model, according to the original image features, the key point features of the original image and the key point features of the expression image, the mask image and the first image are obtained respectively;

[0169] Through the preset model, a second image is obtained according to the original image, the key point features of the original image and the key point features of the expression image.

[0170] Optionally, the processing module 330 is also used to: obtain a mask image according to the original image features, the original image key point features and the expression image key point features through a mask prediction network of a preset model; the mask image includes a key area and a non-key area; the key area is used to represent the area where the facial features of the target person are located.

[0171] Optionally, the processing module 330 is also used to: obtain a first image according to the original image features, the original image key point features and the expression image key point features through a generation network of a preset model; the resolution of the first image is a preset resolution, and the preset resolution is smaller than the resolution of the original image.

[0172] Optionally, the processing module 330 is further used to: obtain optical flow information according to the key point features of the original image and the key point features of the expression map through an optical flow prediction network of a preset model; the optical flow information represents the displacement information of the key point features of the original image and the key point features of the expression map;

[0173] According to the optical flow information, the original image is deformed to obtain a second image.

[0174] Optionally, the acquisition module 310 is further used to: respond to the user's selection operation on the video template interface, determine the target video template corresponding to the selection operation, and display the picture acquisition interface; the video template interface includes at least one video template; each video frame of the target video template is an expression picture;

[0175] In response to the user's interactive operation on the image acquisition interface, a target image corresponding to the interactive operation is acquired; the target image is the original image.

[0176] Optionally, the face image generation device further includes a training module 350, which is used to input a training sample into a basic model to obtain a target image; the training sample is two face video frames preset in the same face video; the training sample includes sample labeling information;

[0177] Calculate the loss function based on the target image and sample label information;

[0178] Optimize the parameters of the base model according to the loss function;

[0179] When the preset number of iterations is reached, the preset model is obtained.

[0180] An embodiment of the present invention further provides an electronic device, including a processor 120 and a memory 130. The memory 130 stores a computer program. When the processor executes the computer program, the facial image generation method disclosed in the above embodiment is implemented.

[0181] The embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by the processor 120, the method for generating a facial image disclosed in the embodiment of the present invention is implemented.

[0182] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0183] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0184] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a face image, characterized in that: Applied to electronic equipment, the method comprises: Acquire an original image and an expression image; the original image includes the initial expression of the target person; the expression image includes the target expression; Using a facial key point network of a preset model, facial key point detection is performed on the original image and the expression image to obtain the key points of the original image and the key points of the expression image; Through the feature encoding network of the preset model, feature extraction is performed on the original image, the key points of the original image, and the key points of the expression map to obtain the original image features, the original image key point features, and the expression map key point features; By using the preset model, according to the original image features, the key point features of the original image and the key point features of the expression image, a mask image and a first image are obtained respectively; A second image is obtained by using the preset model according to the original image, the key point features of the original image and the key point features of the expression image; the first image includes the facial features of the target person that match the target expression; the second image includes the head features of the target person that match the target expression; The first image and the second image are fused according to the mask image through the preset model to obtain a facial image; the facial image includes a virtual expression of the target person, and the virtual expression matches the target expression.

2. The method according to claim 1, characterized in that The step of obtaining the mask image by using the preset model according to the original image features, the original image key point features and the expression image key point features comprises: The mask image is obtained through the mask prediction network of the preset model according to the original image features, the key point features of the original image and the key point features of the expression image; the mask image includes a key area and a non-key area; the key area is used to represent the area where the facial features of the target person are located.

3. The method according to claim 1, characterized in that The step of obtaining the first image by using the preset model according to the original image features, the original image key point features and the expression image key point features comprises: The first image is obtained through the generation network of the preset model according to the original image features, the key point features of the original image and the key point features of the expression image; the resolution of the first image is a preset resolution, and the preset resolution is smaller than the resolution of the original image.

4. The method according to claim 1, characterized in that: The step of obtaining the second image by using the preset model according to the original image, the key point features of the original image and the key point features of the expression image comprises: Obtaining optical flow information according to the key point features of the original image and the key point features of the expression map through the optical flow prediction network of the preset model; the optical flow information represents the displacement information of the key point features of the original image and the key point features of the expression map; The original image is deformed according to the optical flow information to obtain the second image.

5. The method according to claim 1, characterized in that The preset model is trained in the following way: Inputting training samples into the basic model to obtain a target image; the training samples are two face video frames preset in the same face video; the training samples include sample labeling information; Calculating a loss function according to the target image and the sample label information; Optimizing the parameters of the basic model according to the loss function; When the preset number of iterations is reached, the preset model is obtained.

6. The method according to claim 1, characterized in that: The step of obtaining the original image and the expression image includes: In response to a user's selection operation on a video template interface, a target video template corresponding to the selection operation is determined, and a picture acquisition interface is displayed; the video template interface includes at least one video template; each video frame of the target video template is the expression picture; In response to the user's interactive operation on the image acquisition interface, a target image corresponding to the interactive operation is acquired; the target image is the original image.

7. A facial image generating device, characterized in that: Applied to electronic equipment, the device comprises: An acquisition module, used to acquire an original image and an expression image; the original image includes the initial expression of the target person; the expression image includes the target expression; A processing module, used to perform facial key point detection on the original image and the expression map through a facial key point network of a preset model to obtain the original image key points and the expression map key points; perform feature extraction on the original image, the original image key points and the expression map key points through a feature encoding network of the preset model to obtain the original image features, the original image key point features and the expression map key point features; obtain a mask image and a first image respectively according to the original image features, the original image key point features and the expression map key point features through the preset model; obtain a second image according to the original image, the original image key point features and the expression map key point features through the preset model; the first image includes the facial features of the target person that match the target expression; the second image includes the head features of the target person that match the target expression; The first image and the second image are fused according to the mask image through the preset model to obtain a facial image; the facial image includes a virtual expression of the target person, and the virtual expression matches the target expression.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images