Face fusion method, system, device, medium and program product

Abstract

The invention provides a face fusion method, system and device, a medium and a program product. The face fusion method comprises the following steps: acquiring actual key points of at least two to-be-fused face images; obtaining a position adjustment coefficient corresponding to the to-be-fused face image based on the preset key point and the actual key point; adjusting the to-be-fused face image based on the position adjustment coefficient to obtain an adjusted to-be-fused face image; and performing face fusion processing on the adjusted to-be-fused face image to obtain a target face fusion image. According to the invention, the position adjustment coefficient is obtained through the preset key point and the actual key point of the to-be-fused face image, the to-be-fused face image is adjusted according to the position adjustment coefficient, and then the face fusion processing is carried out, so that the quality of the face fusion image is improved, the fusion effect is improved, and the difficulty and resource consumption of the face fusion processing are reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a face fusion method, system, device, medium and program product. Background Art

[0002] The study of facial images has always been a very important part in the field of images. In the upsurge of deep learning, facial image recognition is one of the directions that has been put into application earlier. For facial images, in addition to common tasks such as recognition and detection, there are also tasks such as reconstruction and face fusion. Face fusion, as the name suggests, refers to the fusion of faces in two images to generate a natural face. Generally speaking, one of the two images is used as a template image, which will provide relevant information such as the corresponding face position, background, hair, etc.; and the other image is used as a target image, which will provide corresponding facial features information. Based on the above two images, the face fusion model can generate the corresponding face image.

[0003] With the rapid development of AIGC (Artificial Intelligence Generated Content) technology, the generation task of face fusion has received a lot of attention, and a large number of methods using deep networks have been proposed. However, the face fusion images currently obtained using deep networks have problems such as incorrect face angles, and the fusion effect is poor. In addition, more data is needed to train the deep network, which consumes a lot of resources. Summary of the invention

[0004] The technical problem to be solved by the present disclosure is to overcome the defects of the prior art such as poor fusion effect and large consumption of training resources of face fusion images obtained by using deep networks, and to provide a face fusion method, system, device, medium and program product.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] The present disclosure provides a face fusion method, the face fusion method comprising:

[0007] Obtaining actual key points of at least two face images to be fused;

[0008] Based on the preset key points and the actual key points, obtaining the position adjustment coefficient corresponding to the face image to be fused;

[0009] Adjusting the face image to be fused based on the position adjustment coefficient to obtain the adjusted face image to be fused;

[0010] The adjusted face images to be fused are subjected to face fusion processing to obtain a target face fusion image.

[0011] Preferably, the step of performing face fusion processing on the adjusted face image to be fused to obtain a target face fusion image comprises:

[0012] Performing face fusion processing on the adjusted face image to be fused to obtain an initial face fusion image;

[0013] Inputting the initial face fusion image into the high-definition model to obtain a target face fusion image;

[0014] The pixel value of the target face fusion image is greater than the pixel value of the initial face fusion image, and / or the resolution of the target face fusion image is greater than the resolution of the initial face fusion image.

[0015] Preferably, after the step of performing face fusion processing on the adjusted face image to be fused to obtain a target face fusion image, the method further includes:

[0016] Selecting an image from the face images to be fused as a template face image;

[0017] Obtaining a position restoration coefficient based on the position adjustment coefficient corresponding to the template face image;

[0018] Based on the position restoration coefficient, adjusting the target face fusion image to obtain the adjusted target face fusion image;

[0019] Wherein, the positions of the key points in the adjusted target face fusion image are aligned with the positions of the key points in the template face image;

[0020] and / or,

[0021] The step of obtaining actual key points of at least two face images to be fused comprises:

[0022] At least two of the face images to be fused are respectively input into a key point detection model to obtain the actual key points corresponding to each of the face images to be fused.

[0023] Preferably, the step of performing face fusion processing on the adjusted face image to be fused to obtain an initial face fusion image comprises:

[0024] The adjusted face image to be fused is input into the face fusion model for face fusion processing to obtain the initial face fusion image.

[0025] Preferably, the face fusion model is used to perform fusion processing on the corresponding face features in all the adjusted face images to be fused;

[0026] Wherein, the facial features include facial features and background features;

[0027] or,

[0028] The face fusion model is used to obtain partial face features from each adjusted face image to be fused for fusion processing.

[0029] Preferably, the step of obtaining the position adjustment coefficient corresponding to the face image to be fused based on the preset key points and the actual key points comprises:

[0030] Based on the preset key points and the actual key points, the position adjustment coefficient corresponding to the face image to be fused is calculated by adopting singular value decomposition method and least square method.

[0031] Preferably, the calculation formula corresponding to the step of calculating the position adjustment coefficient corresponding to the face image to be fused by using the singular value decomposition method and the least squares method based on the preset key points and the actual key points is as follows:

[0032] ;

[0033] in, represents the coordinates of the i-th preset key point, represents the coordinates of the ith actual key point, j is a constant, represents the position adjustment coefficient, i∈[1, k], and k represents the number of the preset key points or the actual key points.

[0034] Preferably, the calculation formula corresponding to the step of adjusting the face image to be fused based on the position adjustment coefficient to obtain the adjusted face image to be fused is as follows:

[0035] ;

[0036] in, represents the adjusted color value of the pixel (m, n) in the face image to be fused, Represents the pixel point in the face image to be fused The color value, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the various values ​​of the position adjustment coefficients.

[0037] Preferably, the calculation formula corresponding to the step of obtaining the position restoration coefficient based on the position adjustment coefficient corresponding to the template face image is as follows:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] Wherein, A represents the position restoration coefficient, A 11 , A 12 , A 13 , A 21 , A 22 , A 23 Represents the various values ​​of the position restoration coefficient, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the various values ​​of the position adjustment coefficients.

[0046] The present disclosure also provides a face fusion system, the face fusion system comprising:

[0047] A key point acquisition module, used to acquire actual key points of at least two face images to be fused;

[0048] An adjustment coefficient acquisition module, used for obtaining a position adjustment coefficient corresponding to the face image to be fused based on preset key points and the actual key points;

[0049] A face adjustment module, used for adjusting the face image to be fused based on the position adjustment coefficient to obtain the adjusted face image to be fused;

[0050] The face fusion module is used to perform face fusion processing on the adjusted face image to be fused to obtain a target face fusion image.

[0051] Preferably, the face fusion module includes:

[0052] An initial face acquisition unit, used for performing face fusion processing on the adjusted face image to be fused to obtain an initial face fusion image;

[0053] A target face acquisition unit, used for inputting the initial face fusion image into a high-definition model to obtain a target face fusion image;

[0054] The pixel value of the target face fusion image is greater than the pixel value of the initial face fusion image, and / or the resolution of the target face fusion image is greater than the resolution of the initial face fusion image.

[0055] Preferably, the face fusion system further includes:

[0056] A template image acquisition module, used to select an image from the face images to be fused as a template face image;

[0057] A restoration coefficient acquisition module, used to obtain a position restoration coefficient based on the position adjustment coefficient corresponding to the template face image;

[0058] A face restoration module, used for adjusting the target face fusion image based on the position restoration coefficient to obtain the adjusted target face fusion image;

[0059] Wherein, the positions of the key points in the adjusted target face fusion image are aligned with the positions of the key points in the template face image;

[0060] and / or,

[0061] The key point acquisition module is also used to input at least two of the face images to be fused into the key point detection model respectively, so as to obtain the actual key points corresponding to each of the face images to be fused.

[0062] Preferably, the initial face acquisition unit is further used to input the adjusted face image to be fused into a face fusion model to perform face fusion processing to obtain the initial face fusion image.

[0063] Preferably, the face fusion model is used to perform fusion processing on the corresponding face features in all the adjusted face images to be fused;

[0064] Wherein, the facial features include facial features and background features;

[0065] or,

[0066] The face fusion model is used to obtain partial face features from each adjusted face image to be fused for fusion processing.

[0067] Preferably, the adjustment coefficient acquisition module is further used to calculate the position adjustment coefficient corresponding to the face image to be fused based on the preset key points and the actual key points by using a singular value decomposition method and a least squares method.

[0068] Preferably, the calculation formula adopted by the adjustment coefficient acquisition module is as follows:

[0069] ;

[0070] in, represents the coordinates of the i-th preset key point, represents the coordinates of the ith actual key point, j is a constant, represents the position adjustment coefficient, i∈[1, k], and k represents the number of the preset key points or the actual key points.

[0071] Preferably, the calculation formula used by the face adjustment module is as follows:

[0072] ;

[0073] in, represents the adjusted color value of the pixel (m, n) in the face image to be fused, Represents the pixel point in the face image to be fused The color value, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the various values ​​of the position adjustment coefficients.

[0074] Preferably, the calculation formula adopted by the restoration coefficient acquisition module is as follows:

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] ;

[0082] Wherein, A represents the position restoration coefficient, A 11 , A 12 , A 13 , A 21 , A 22 , A 23Represents the various values ​​of the position restoration coefficient, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the various values ​​of the position adjustment coefficients.

[0083] The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and used to run on the processor, and the processor implements the above-mentioned face fusion method when executing the computer program.

[0084] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the above-mentioned face fusion method when executed by a processor.

[0085] The present disclosure also provides a computer program product, including a computer program, which implements the face fusion method as described above when executed by a processor.

[0086] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.

[0087] The positive and progressive effects of this disclosure are:

[0088] The present invention obtains position adjustment coefficients by presetting key points and actual key points of the face image to be fused, adjusts the face image to be fused according to the position adjustment coefficients, and then performs face fusion processing, thereby improving the quality of the face fusion image, enhancing the fusion effect, and reducing the difficulty and resource consumption of face fusion processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 This is a flowchart of the face fusion method of Embodiment 1 of the present disclosure;

[0090] Figure 2 This is a flow chart of the face fusion method of Embodiment 2 of the present disclosure;

[0091] Figure 3 A schematic diagram of a model training process of a face fusion method according to Embodiment 2 of the present disclosure;

[0092] Figure 4 This is a specific example diagram of the face fusion method of Embodiment 2 of the present disclosure;

[0093] Figure 5 A schematic diagram of a model application process of the face fusion method of Embodiment 2 of the present disclosure;

[0094] Figure 6This is a module diagram of a face fusion system according to Embodiment 3 of the present disclosure;

[0095] Figure 7 This is a module diagram of a face fusion system according to Embodiment 4 of the present disclosure;

[0096] Figure 8 This is a schematic diagram of the structure of an electronic device according to Embodiment 5 of the present disclosure. DETAILED DESCRIPTION

[0097] The present disclosure is further described below by way of examples, but the present disclosure is not limited to the scope of the examples.

[0098] Prefixes such as "first" and "second" are used in the embodiments of the present disclosure only to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present disclosure does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes. In addition, in the description of the present embodiment, unless otherwise specified, the meaning of "plurality" is two or more.

[0099] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0100] Example 1

[0101] This embodiment provides a face fusion method, such as Figure 1 As shown, the face fusion method includes:

[0102] S101, obtaining actual key points of at least two face images to be fused;

[0103] S102, obtaining a position adjustment coefficient corresponding to the face image to be fused based on the preset key points and the actual key points;

[0104] The specific form of the position adjustment coefficient may be a matrix;

[0105] S103, adjusting the face image to be fused based on the position adjustment coefficient to obtain an adjusted face image to be fused;

[0106] S104: performing face fusion processing on the adjusted face image to be fused to obtain a target face fusion image.

[0107] Specifically, the actual key points and the preset key points include the coordinate points of the facial features such as eyes, mouth, nose, ears, eyebrows, etc. in the face image. In addition, the key points can also be hundreds of coordinate points obtained by segmenting the entire face image as a whole, and these coordinate points cover key areas such as eyes, mouth, nose, ears, eyebrows, etc. For each face image to be fused, the corresponding actual key points are obtained; based on the preset key points and the actual key points corresponding to each face image to be fused, the corresponding position adjustment coefficients of each face image to be fused are obtained; based on the corresponding position adjustment coefficients of each face image to be fused, each face image to be fused is adjusted to obtain the corresponding adjusted face image to be fused; all the adjusted face images to be fused are subjected to face fusion processing to obtain the target face fusion image.

[0108] In this embodiment, the position adjustment coefficient is obtained by presetting key points and the actual key points of the face image to be fused, and the face image to be fused is adjusted according to the position adjustment coefficient, and then the face fusion processing is performed, which improves the quality of the face fusion image, enhances the fusion effect, and reduces the difficulty and resource consumption of the face fusion processing.

[0109] Example 2

[0110] This embodiment provides a face fusion method, which is a further improvement on Embodiment 1.

[0111] In one feasible solution, Figure 2 As shown, step S104 includes:

[0112] S1041, performing face fusion processing on the adjusted face image to be fused to obtain an initial face fusion image;

[0113] S1042, inputting the initial face fusion image into the high-definition model to obtain a target face fusion image;

[0114] Among them, the pixel value of the target face fusion image is greater than the pixel value of the initial face fusion image, and / or the resolution of the target face fusion image is greater than the resolution of the initial face fusion image. Correspondingly, the high-definition model has three forms to improve the clarity of the initial face fusion image, and the three forms are to improve the pixel value of the image, improve the resolution of the image, and improve the pixel value and resolution of the image.

[0115] The HD model is a neural network based on convolution and deconvolution. It can convert blurry faces into high-definition and natural faces without deforming the faces or changing the characters.

[0116] In this solution, the target face fusion image is input into the high-definition model to obtain a high-definition face fusion image, thereby improving the clarity of the face fusion image and the fusion effect of the face fusion image.

[0117] In one feasible solution, Figure 2 As shown, after step S104, the following steps are also included:

[0118] S105, selecting an image from the face images to be fused as a template face image;

[0119] S106, obtaining a position restoration coefficient based on a position adjustment coefficient corresponding to the template face image;

[0120] S107, adjusting the target face fusion image based on the position restoration coefficient to obtain an adjusted target face fusion image;

[0121] Wherein, the positions of the key points in the adjusted target face fusion image are aligned with the positions of the key points in the template face image;

[0122] Specifically, the target face fusion image is obtained by adjusting the face image to be fused according to the position adjustment coefficient, and its position is adjusted. In fact, face fusion needs to be based on the position of the template face image. Therefore, the target face fusion image needs to be adjusted again, that is, the target face fusion image is reversed relative to the adjustment operation corresponding to the position adjustment coefficient, and the position of the target face fusion image is restored to the position corresponding to the template face image. Therefore, it is necessary to calculate the inverse coefficient of the position adjustment coefficient, that is, the position restoration coefficient. The selection of the template face image can be determined according to user needs.

[0123] In this scheme, the position coefficient restoration coefficient is obtained according to the position adjustment coefficient to adjust the target face fusion image, thereby ensuring that the adjusted target face fusion image is based on the position of the template face image, thereby improving the accuracy and reliability of the target face image.

[0124] In one feasible solution, Figure 2 As shown, step S101 includes:

[0125] S1011. Input at least two facial images to be fused into a key point detection model respectively to obtain actual key points corresponding to each facial image to be fused.

[0126] Specifically, taking two images as an example, key point detection is performed on the two face images to be fused respectively, and k actual key points of each are obtained. The number of actual key points can be determined according to the key point detection model. The key point detection model is trained using a face image with k actual key points, and when the key point detection model is applied to perform key point detection on the face image, k actual key points of the face image can be obtained. Where k is a positive integer.

[0127] In this solution, key point detection is performed on the face image to be fused through a key point detection model to obtain the actual key points corresponding to the face image to be fused, thereby improving the accuracy and reliability of the actual key points.

[0128] In one feasible solution, step S1041 includes:

[0129] The adjusted face image to be fused is input into the face fusion model for face fusion processing to obtain an initial face fusion image.

[0130] In this solution, the adjusted face images to be fused are processed by the face fusion model to obtain an initial face fusion image, thereby ensuring the accuracy and reliability of the initial face fusion image.

[0131] In an implementable solution, the face fusion model is used to fuse the corresponding face features in all adjusted face images to be fused;

[0132] Among them, facial features include facial features and background features.

[0133] Specifically, taking three images as an example, the face fusion model extracts the face features contained in each of the three adjusted face images to be fused, and fuses the face features of the three images to obtain the initial face fusion image. The face features contained in each image include facial features and background features. Facial features include features of the face image such as eyes, nose, ears, mouth, eyebrows, etc., and background features include features of the face image such as face position, image background, hair, etc.

[0134] In this solution, the face features of all the adjusted face images to be fused are fused through a face fusion model to obtain an initial face fusion image, so that the initial face fusion image contains the face features of each face image to be fused.

[0135] In another feasible solution, the face fusion model is used to obtain partial face features from each adjusted face image to be fused for fusion processing.

[0136] Specifically, taking two images as an example, the two images are a target face image and a template face image. The union of some facial features of the target face image and some facial features of the template face image covers the facial features of a single face image. The face fusion model includes two encoders, a fusion module and a generator.

[0137] The two encoders are the face encoder and the attribute encoder. The face encoder can effectively extract the facial features of the target face image, that is, the facial features. In order to train the face encoder, the face recognition architecture can be used to divide the face recognition network into an encoder and a decoder, and then the corresponding face recognition task training is performed using the face recognition task data set. After training, the encoder part of the face recognition network is the corresponding face encoder. For the template face image, another separate encoder is used for feature extraction, that is, the attribute encoder. As the encoder is trained, it will extract the relevant attribute information of the template face image, such as the position, image background, and hair, that is, the background features.

[0138] Different encoders are used to extract features. After extracting facial features and background features, the above two features are input into the fusion module for feature fusion. The fusion module is mainly composed of convolution and attention mechanism, which can effectively fuse the two features. After the two features are fused, they are input into the generator.

[0139] The generator is composed of convolution, deconvolution, etc., and can output a preliminary fused face, which is usually blurry.

[0140] In addition, if Figure 3 FIG. 1 is a schematic diagram of the model training process of the face fusion method of this embodiment. Take two sample face images as an example, namely, a sample target face image and a sample template face image. Figure 3 The modules in the dashed box, i.e., the key point detection model and the face encoder, can use known models, i.e., they are trained in advance and then Figure 3 The modules in the solid box are trained, namely the attribute encoder, fusion module, generator and HD model. Figure 3The position adjustment in is to use the position adjustment coefficients corresponding to the sample target face image and the sample template face image to adjust the sample target face image and the sample template face image respectively. The training labels in the training process are divided into label images and high-definition label images, where the label image is a blurred label image and the high-definition label image is a high-definition label image. Training can be performed by calculating the loss between the initial face fusion image and the label image, and between the target face fusion image and the high-definition label image. Specifically, the four modules of the attribute encoder, the fusion module, the generator and the high-definition model are trained together, and two losses are calculated. A loss is calculated between the initial face fusion image generated by the generator and the label image. The initial face fusion image is input into the high-definition module to obtain the target face fusion image, and a loss is calculated between the target face fusion image and the high-definition label image. The two losses are added together to train the four modules together. The loss calculation between the above two includes at least one of L1 loss, VGG loss (perceptual loss), stylization loss, adversarial network loss and regularization loss.

[0141] In this solution, the face fusion model is used to obtain partial facial features of each face image to be fused after adjustment for fusion processing to obtain an initial face fusion image, so that the initial face fusion image contains partial facial features of each face image to be fused.

[0142] In one feasible solution, Figure 2 As shown, step S102 includes:

[0143] S1021. Based on the preset key points and the actual key points, the singular value decomposition (SVD) method and the least squares method are used to calculate the position adjustment coefficients corresponding to the face image to be fused.

[0144] The calculation formula corresponding to step S1021 is as follows:

[0145] ;

[0146] in, represents the coordinates of the i-th preset key point, represents the coordinates of the ith actual key point, j is a constant, Represents the position adjustment coefficient, i∈[1,k], k represents the number of preset key points or actual key points.

[0147] Specifically, the preset key points are known standard templates, and there are k preset key points in total, which is consistent with the number of actual key points of each face image to be fused. For each preset key point and the corresponding actual key point, the above formula is satisfied, and a total of k above formulas can be obtained. According to the k above formulas, the position adjustment coefficient corresponding to each face image to be fused is calculated by singular value decomposition and least squares method. The position adjustment coefficient can be a matrix, which is a 2x3 matrix here. j can be 1, which is just an example and can be set or adjusted according to actual conditions. A calculation example is given below.

[0148] According to the above formula, if j is 1, we can get the following linear equations:

[0149] ;

[0150] In terms of matrices, the two matrices on the left side of the equal sign of the above linear equations are represented as E and M in order, and the matrix on the right side of the equal sign of the above linear equations is represented as F. Then the above linear equations are represented as EM=F, where E is an overdetermined matrix. In order to solve the equation, according to the idea of ​​least squares method: find an M such that The value of is the smallest. After derivation, it can be known that the solution M satisfies ,Right now . The inverse process can be solved using SVD.

[0151] In this solution, the singular value decomposition method and the least squares method are used to calculate the position adjustment coefficient corresponding to the face image to be fused, which improves the accuracy and reliability of the position adjustment coefficient.

[0152] In one feasible solution, the calculation formula corresponding to step S103 is as follows:

[0153] ;

[0154] in, Represents the color value of the pixel (m, n) in the face image to be fused after adjustment. Represents the pixel points in the face image to be fused The color value, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the individual values ​​of the position adjustment coefficient.

[0155] Specifically, for the fused face image, the predefined direction is It can represent the color value of the adjusted face image to be fused in the mth row and nth column. It can be said that the face image to be fused is Line The color value of the column. For example, m, n, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 If both are taken as 1, we can get , which means that the color value of the first row and first column of the adjusted face image to be fused is equal to the color value of the third row and third column of the face image to be fused.

[0156] In this scheme, the color value of the face image to be fused is adjusted by the position adjustment coefficient corresponding to the face image to be fused to obtain the adjusted face image to be fused, and the face fusion processing is performed according to the adjusted face image to be fused, thereby improving the fusion effect of face fusion.

[0157] In one feasible solution, the calculation formula corresponding to step S106 is as follows:

[0158] ;

[0159] ;

[0160] ;

[0161] ;

[0162] ;

[0163] ;

[0164] ;

[0165] Among them, A represents the position restoration coefficient, A 11 , A 12 , A 13 , A 21 , A 22 , A 23 Represents the values ​​of the position restoration coefficient, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the individual values ​​of the position adjustment coefficient.

[0166] The position adjustment coefficient is the matrix M= , perform an inverse transformation on the matrix and calculate its corresponding inverse matrix, which is the position restoration coefficient .

[0167] After obtaining the position restoration coefficient, the target face fusion image needs to be adjusted, which is similar to the method of adjusting the fusion face image. That is, the adjusted target face fusion image is obtained according to the following formula:

[0168] ;

[0169] in, Represents the color value of the pixel (r, s) in the adjusted target face fusion image, Represents the pixel point in the target face fusion image The color value, A 11 , A 12 , A 13 , A 21 , A 22 , A 23 Indicates the values ​​of the position restoration coefficient.

[0170] In this scheme, the position coefficient restoration coefficient is obtained according to the position adjustment coefficient to adjust the target face fusion image, thereby ensuring that the adjusted target face fusion image is based on the position of the template face image, thereby improving the accuracy and reliability of the target face image.

[0171] The working principle of the face fusion method of this embodiment is explained below with reference to specific examples:

[0172] like Figure 4 and 5 As shown, Figure 4 is a specific example diagram of the face fusion method of this embodiment, Figure 5Schematic diagram of the model application process of the face fusion method of this embodiment. Two face images to be fused are obtained, one is a target face image and the other is a template face image; key point detection models are used to perform key point detection on the two face images to be fused respectively, and the corresponding actual key points are obtained respectively; the corresponding position adjustment coefficients are obtained according to the corresponding actual key points and preset key points; the target face image is adjusted according to the position adjustment coefficient corresponding to the target face image to obtain the adjusted target face image; the template face image is adjusted according to the position adjustment coefficient corresponding to the template face image to obtain the adjusted template face image; different encoders are used to extract features from the adjusted target face image and template face image respectively, wherein the target face image is extracted by a face encoder and the template face image is extracted by an attribute encoder; the extracted features are fused through a fusion module and then input into a generator; the output result of the generator, i.e., the initial face fusion image, is input into a high-definition model to obtain a target face fusion image; the position restoration coefficient is obtained according to the position adjustment coefficient corresponding to the template face image, and the target face fusion image is restored to the position corresponding to the template face image to obtain the adjusted target face fusion image.

[0173] In this embodiment, the position adjustment coefficient is obtained by presetting key points and the actual key points of the face image to be fused, and the face image to be fused is adjusted according to the position adjustment coefficient, and then the face fusion processing is performed, which improves the quality of the face fusion image, enhances the fusion effect, and reduces the difficulty and resource consumption of the face fusion processing.

[0174] Example 3

[0175] This embodiment provides a face fusion system. Figure 6 As shown, the face fusion system includes:

[0176] A key point acquisition module 1 is used to acquire actual key points of at least two face images to be fused;

[0177] The adjustment coefficient acquisition module 2 is used to obtain the position adjustment coefficient corresponding to the face image to be fused based on the preset key points and the actual key points;

[0178] A face adjustment module 3 is used to adjust the face image to be fused based on the position adjustment coefficient to obtain an adjusted face image to be fused;

[0179] The face fusion module 4 is used to perform face fusion processing on the adjusted face image to be fused to obtain a target face fusion image.

[0180] In this embodiment, the position adjustment coefficient is obtained by presetting key points and the actual key points of the face image to be fused, and the face image to be fused is adjusted according to the position adjustment coefficient, and then the face fusion processing is performed, which improves the quality of the face fusion image, enhances the fusion effect, and reduces the difficulty and resource consumption of the face fusion processing.

[0181] Example 4

[0182] This embodiment provides a face fusion system, which is a further improvement on Embodiment 3.

[0183] In a feasible solution, such as Figure 7 As shown, the face fusion module 4 includes:

[0184] The initial face acquisition unit 41 is used to perform face fusion processing on the adjusted face image to be fused to obtain an initial face fusion image;

[0185] The target face acquisition unit 42 is used to input the initial face fusion image into the high-definition model to obtain the target face fusion image;

[0186] The pixel value of the target face fusion image is greater than the pixel value of the initial face fusion image, and / or the resolution of the target face fusion image is greater than the resolution of the initial face fusion image.

[0187] In an implementable solution, the face fusion system further includes:

[0188] The template image acquisition module 5 is used to select an image from the face images to be fused as a template face image;

[0189] A restoration coefficient acquisition module 6 is used to obtain a position restoration coefficient based on the position adjustment coefficient corresponding to the template face image;

[0190] A face restoration module 7, used for adjusting the target face fusion image based on the position restoration coefficient to obtain an adjusted target face fusion image;

[0191] Wherein, the positions of the key points in the adjusted target face fusion image are aligned with the positions of the key points in the template face image;

[0192] and / or,

[0193] The key point acquisition module 1 is also used to input at least two face images to be fused into the key point detection model respectively, so as to obtain the actual key points corresponding to each face image to be fused.

[0194] In an implementable solution, the initial face acquisition unit 41 is further used to input the adjusted face image to be fused into the face fusion model to perform face fusion processing to obtain an initial face fusion image.

[0195] In an implementable solution, the face fusion model is used to fuse the corresponding face features in all adjusted face images to be fused;

[0196] Among them, facial features include facial features and background features;

[0197] or,

[0198] The face fusion model is used to obtain partial facial features from each adjusted face image to be fused for fusion processing.

[0199] In an implementable solution, the adjustment coefficient acquisition module 2 is further used to calculate the position adjustment coefficient corresponding to the face image to be fused based on the preset key points and the actual key points by using the singular value decomposition method and the least square method.

[0200] In an implementable solution, the calculation formula used by the adjustment coefficient acquisition module 2 is as follows:

[0201] ;

[0202] in, represents the coordinates of the i-th preset key point, represents the coordinates of the ith actual key point, j is a constant, Represents the position adjustment coefficient, i∈[1,k], k represents the number of preset key points or actual key points.

[0203] In an implementable solution, the calculation formula used by the face adjustment module 3 is as follows:

[0204] ;

[0205] in, Represents the color value of the pixel (m, n) in the face image to be fused after adjustment. Represents the pixel points in the face image to be fused The color value, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the individual values ​​of the position adjustment coefficient.

[0206] In an implementable solution, the calculation formula used by the restoration coefficient acquisition module 6 is as follows:

[0207] ;

[0208] ;

[0209] ;

[0210] ;

[0211] ;

[0212] ;

[0213] ;

[0214] Among them, A represents the position restoration coefficient, A 11 , A 12 , A 13 , A 21 , A 22 , A 23 Represents the values ​​of the position restoration coefficient, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the individual values ​​of the position adjustment coefficient.

[0215] In this embodiment, the position adjustment coefficient is obtained by presetting key points and the actual key points of the face image to be fused, and the face image to be fused is adjusted according to the position adjustment coefficient, and then the face fusion processing is performed, which improves the quality of the face fusion image, enhances the fusion effect, and reduces the difficulty and resource consumption of the face fusion processing.

[0216] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution.

[0217] Example 5

[0218] Figure 8 This is a structural schematic diagram of an electronic device shown in an example embodiment of the present disclosure, the electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor, and the processor implements the face fusion method described in any of the above embodiments when executing the computer program. Figure 8 The electronic device 90 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0219] like Figure 8 As shown, the electronic device 90 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including the memory 92 and the processor 91).

[0220] The bus 93 includes a data bus, an address bus, and a control bus.

[0221] The memory 92 may include a volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922 , and may further include a read only memory (ROM) 923 .

[0222] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) of program modules 924, such program modules 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0223] The processor 91 executes various functional applications and data processing by running the computer program stored in the memory 92, such as the face fusion method provided in any of the above embodiments.

[0224] The electronic device 90 may also communicate with one or more external devices 94 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 95. Furthermore, the electronic device 90 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 96. As shown, the network adapter 96 communicates with other modules of the electronic device 90 via a bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0225] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0226] Example 6

[0227] The embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the face fusion method provided in any of the above embodiments is implemented.

[0228] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0229] Example 7

[0230] The embodiment of the present disclosure also provides a computer program product, including a computer program, which implements any of the above-mentioned face fusion methods when executed by a processor.

[0231] Among them, the program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0232] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, but these changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A face fusion method, characterized in that: The face fusion method comprises: Obtaining actual key points of at least two face images to be fused; Based on the preset key points and the actual key points, obtaining the position adjustment coefficient corresponding to the face image to be fused; Adjusting the face image to be fused based on the position adjustment coefficient to obtain the adjusted face image to be fused; The adjusted face images to be fused are subjected to face fusion processing to obtain a target face fusion image.

2. The face fusion method according to claim 1, characterized in that: The step of performing face fusion processing on the adjusted face image to be fused to obtain a target face fusion image comprises: Performing face fusion processing on the adjusted face image to be fused to obtain an initial face fusion image; Inputting the initial face fusion image into the high-definition model to obtain a target face fusion image; The pixel value of the target face fusion image is greater than the pixel value of the initial face fusion image, and / or the resolution of the target face fusion image is greater than the resolution of the initial face fusion image.

3. The face fusion method according to claim 1 or 2, characterized in that: After the step of performing face fusion processing on the adjusted face image to be fused to obtain a target face fusion image, the method further includes: Selecting an image from the face images to be fused as a template face image; Obtaining a position restoration coefficient based on the position adjustment coefficient corresponding to the template face image; Based on the position restoration coefficient, adjusting the target face fusion image to obtain the adjusted target face fusion image; Wherein, the positions of the key points in the adjusted target face fusion image are aligned with the positions of the key points in the template face image; and / or, The step of obtaining actual key points of at least two face images to be fused comprises: At least two of the face images to be fused are respectively input into a key point detection model to obtain the actual key points corresponding to each of the face images to be fused.

4. The face fusion method according to claim 2, characterized in that: The step of performing face fusion processing on the adjusted face image to be fused to obtain an initial face fusion image comprises: The adjusted face image to be fused is input into the face fusion model for face fusion processing to obtain the initial face fusion image.

5. The face fusion method according to claim 4, characterized in that: The face fusion model is used to perform fusion processing on the corresponding face features in all the adjusted face images to be fused; Wherein, the facial features include facial features and background features; or, The face fusion model is used to obtain partial face features from each adjusted face image to be fused for fusion processing.

6. The face fusion method according to claim 1 or 2, characterized in that: The step of obtaining the position adjustment coefficient corresponding to the face image to be fused based on the preset key points and the actual key points comprises: Based on the preset key points and the actual key points, the position adjustment coefficient corresponding to the face image to be fused is calculated by using singular value decomposition method and least square method.

7. The face fusion method according to claim 6, characterized in that: The calculation formula corresponding to the step of calculating the position adjustment coefficient corresponding to the face image to be fused by using the singular value decomposition method and the least squares method based on the preset key points and the actual key points is as follows: ； in, represents the coordinates of the i-th preset key point, represents the coordinates of the ith actual key point, j is a constant, represents the position adjustment coefficient, i∈[1, k], and k represents the number of the preset key points or the actual key points.

8. The face fusion method according to claim 7, characterized in that: The calculation formula corresponding to the step of adjusting the face image to be fused based on the position adjustment coefficient to obtain the adjusted face image to be fused is as follows: ； in, represents the adjusted color value of the pixel (m, n) in the face image to be fused, Represents the pixel point in the face image to be fused The color value, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the various values ​​of the position adjustment coefficients.

9. The face fusion method according to claim 3, characterized in that: The calculation formula corresponding to the step of obtaining the position restoration coefficient based on the position adjustment coefficient corresponding to the template face image is as follows: ； ； ； ； ； ； ； Wherein, A represents the position restoration coefficient, A 11 , A 12 , A 13 , A 21 , A 22 , A 23 Represents the various values ​​of the position restoration coefficient, M 11 、M 12 、M 13 、M 21 、M 22 、M 23 Indicates the various values ​​of the position adjustment coefficients.

10. A face fusion system, characterized in that: The face fusion system comprises: A key point acquisition module, used to obtain actual key points of at least two face images to be fused; An adjustment coefficient acquisition module, used for obtaining a position adjustment coefficient corresponding to the face image to be fused based on a preset key point and the actual key point; A face adjustment module, used for adjusting the face image to be fused based on the position adjustment coefficient to obtain the adjusted face image to be fused; The face fusion module is used to perform face fusion processing on the adjusted face image to be fused to obtain a target face fusion image.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that: When the processor executes the computer program, the face fusion method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the face fusion method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the face fusion method according to any one of claims 1 to 9 is implemented.

Images