A method and system for face replacement in a group photo

Abstract

The application relates to a group photo face replacement method and system, wherein the method comprises the following steps: performing similarity grouping on a group photo of a user to obtain group photo sets of different groups; performing face detection on all group photos in a target group photo set, determining faces with a face similarity greater than a preset threshold as the same person, and generating a person-face set mapping table; selecting a face with the highest comprehensive expression score from the person-face set mapping table as an optimal expression face of the corresponding person by using an expression quality evaluation model; and replacing an undesirable face in a to-be-edited group photo into the optimal expression face by using a trained group photo face replacement model. Through the application, automatic retrieval and grouping of group photos are realized, omission of effective materials can be effectively avoided, group photo face replacement editing effect is significantly improved, and the problem of how to improve group photo face replacement editing effect is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a face replacement method and system for group photos. BACKGROUND

[0002] With the popularity of digital photography technology and the development of social networks, digital photos have become an important carrier for recording scenes such as parties and celebrations. However, due to improper expression capture timing (such as closing eyes, expression stiffness), shooting angle deviation and other problems, the face state of some characters in the group photo is not good, which affects the overall effect of the group photo. Therefore, users need to manually find photos with good expressions from a large number of photos, and also need to use professional image editing tools (such as Photoshop) to replace faces. This process requires accurate adjustment of face size, angle, light and shadow and other parameters, which is tedious and requires high user skills.

[0003] However, existing face replacement schemes are generally difficult to adapt to the diverse face poses in group photos, such as matching the angles of front and side faces and complex lighting, and after replacement, problems such as edge splitting and abrupt light and shadow often occur; in addition, some schemes directly paste target face regions for replacement, which can easily lead to harsh integration of identity features and source images, such as mismatched expressions and facial muscle directions.

[0004] At present, there is no effective solution to the problem of how to improve the face replacement editing effect in group photos in the related art. SUMMARY

[0005] Embodiments of the present application provide a face replacement method and system for group photos to at least solve the problem of how to improve the face replacement editing effect in group photos in the related art.

[0006] In a first aspect, the embodiments of the present application provide a face replacement method for group photos, which comprises:

[0007] Grouping the group photos of a user according to similarity to obtain group photo sets of different groups;

[0008] Performing face detection on all group photos in the target group photo set, determining faces with a face similarity greater than a preset threshold as the same character, and generating a character-face set mapping table;

[0009] Selecting a face with the highest expression comprehensive score from the character-face set mapping table as the optimal expression face of the corresponding character using an expression quality evaluation model;

[0010] Replacing an undesirable face in the group photo to be edited with the optimal expression face by using a trained group face replacement model.

[0011] In some embodiments, replacing the non-ideal face in the to-be-edited group photo with the optimal expression face by the trained group photo face replacement model comprises:

[0012] constructing and training a group photo face replacement model to obtain a trained group photo face replacement model, wherein the group photo face replacement model comprises a multi-scale convolutional encoder, a Transformer mask generation module, an expression feature fusion module, and a UV image reconstruction module;

[0013] normalizing the non-ideal face in the to-be-edited group photo and the optimal expression face to unify the face posture and size;

[0014] inputting the normalized to-be-edited group photo and the optimal expression face into the trained group photo face replacement model to replace the non-ideal face in the to-be-edited group photo with the optimal expression face.

[0015] In some embodiments, constructing the multi-scale convolutional encoder in the group photo face replacement model comprises:

[0016] stacking a plurality of ConvBlock modules in sequence to construct the multi-scale convolutional encoder in the group photo face replacement model, wherein each ConvBlock module comprises a trunk branch and a residual connection, the residual connection is added to the trunk branch after adjusting the channel by convolution to output F uv16 feature maps, F uv8 feature maps, and F uv4 feature maps; the F uv16 feature maps are used to capture local expression details, the F uv8 feature maps are used to model medium structural changes, and the F uv4 feature maps are used to represent the whole.

[0017] inputting the output of the multi-scale convolutional encoder as the input of the Transformer mask generation module.

[0018] In some embodiments, constructing the Transformer mask generation module in the group photo face replacement model comprises:

[0019] stacking a plurality of Transformer encoders in multiple layers to construct the Transformer mask generation module in the group photo face replacement model, wherein each layer of the Transformer encoder comprises a self-attention mechanism and a non-linear transformation based on a feedforward neural network FFN, and a layer normalization LayerNorm operation is introduced after each sub-layer;

[0020] on the basis of the output of the multi-scale convolutional encoder, an identity feature vector fid participate in the calculation of attention weights as global guidance information to generate an identity preserving mask Mask id and an expression dynamic mask Mask exp wherein the identity preserving mask Mask id is used to identify an identity stable region, and the expression dynamic mask Mask exp is used to identify an expression region.

[0021] The output of the Transformer mask generation module is taken as the input of the expression feature fusion module.

[0022] In some embodiments, constructing the expression feature fusion module in the group photo face replacement model includes:

[0023] Combining a CLIP text encoder and a multi-layer perception MLP to construct the expression feature fusion module in the group photo face replacement model.

[0024] On the basis of the output of the multi-scale convolutional encoder and the Transformer mask generation module, a target expression description input by a user is introduced as semantic guidance to participate in expression feature fusion to generate expression fusion features.

[0025] The output of the expression feature fusion module is taken as the input of the UV image reconstruction module.

[0026] In some embodiments, constructing the UV image reconstruction module in the group photo face replacement model includes:

[0027] A decoder based on a UNet structure is used to construct the UV image reconstruction module in the group photo face replacement model.

[0028] In some embodiments, similarity grouping of the group photos of the user is performed to obtain group photo sets of different groups, including:

[0029] Obtaining group photos authorized by the user, extracting global features and local features of each group photo;

[0030] Grouping the group photos based on the global features and the local features to obtain group photo sets of different groups.

[0031] In some embodiments, face detection is performed on all group photos in a target group photo set, and a face with a face similarity greater than a preset threshold is determined as the same person to generate a person-face set mapping table, including:

[0032] Based on the selected to-be-edited group photo of the user, the target group photo set in which the to-be-edited group photo is located is determined from the group photo sets of different groups.

[0033] perform face detection on all group photos in the target group photo set, and calculate cosine similarity between different faces, and determine faces with cosine similarity greater than a preset threshold as the same person, to generate a person-face set mapping table.

[0034] In some embodiments, an expression quality evaluation model is used to select a face with the highest expression comprehensive score from the person-face set mapping table as the optimal expression face of the corresponding person, including:

[0035] based on an undesirable face selected by a user from the group photo to be edited, determining the person corresponding to the undesirable face through the person-face set mapping table;

[0036] using an expression quality evaluation model to score all faces in the person-face set mapping table of the person, and sorting the faces from high to low according to the expression comprehensive score, and selecting the face ranked first as the optimal expression face of the person.

[0037] In a second aspect, the embodiments of the present application provide a group photo face replacement system, which is used to execute the method of the first aspect, and includes a group photo grouping module, a face discrimination module, an expression scoring module, and a face replacement module.

[0038] The group photo grouping module is configured to group the group photos of a user according to similarity, to obtain group photo sets of different groups.

[0039] The face discrimination module is configured to perform face detection on all group photos in the target group photo set, and determine faces with face similarity greater than a preset threshold as the same person, to generate a person-face set mapping table.

[0040] The expression scoring module is configured to use an expression quality evaluation model to select a face with the highest expression comprehensive score from the person-face set mapping table as the optimal expression face of the corresponding person.

[0041] The face replacement module is configured to replace an undesirable face in a group photo to be edited with the optimal expression face through a trained group photo face replacement model.

[0042] Compared with the related art, the method and system for replacing faces in group photos provided by the embodiments of the present application group the group photos of users according to similarity, obtain group photo sets of different groups, perform face detection on all group photos in a target group photo set, determine faces with a face similarity greater than a preset threshold as the same person, and generate a person-face set mapping table. An expression quality evaluation model is used to select a face with the highest expression comprehensive score from the person-face set mapping table as an optimal expression face of the corresponding person. A trained group photo face replacement model is used to replace an undesirable face in an edited group photo with the optimal expression face, thereby achieving automatic retrieval and grouping of group photos, effectively avoiding omission of effective materials, accurately classifying faces of different persons based on the obtained group photo sets, further replacing an undesirable face of a specific person in an edited group photo with a face with the highest expression score, and significantly improving the editing effect of group photo face replacement, thereby solving the problem of how to improve the editing effect of group photo face replacement. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and help explain the present application together with the specification. The illustrative embodiments of the present application and their description serve to explain the present application. In the drawings:

[0044] Figure 1 is a step flowchart of a group photo face replacement method according to an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of a group photo face replacement method according to an embodiment of the present application;

[0046] Figure 3 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the present application clearer, the present application is described and explained below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0048] It is apparent that the drawings in the following description merely show some examples or embodiments of the present application, and for those skilled in the art, the present application can be applied to other similar situations without creative labor based on these drawings. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, some design, manufacture or production changes based on the technical content disclosed in the present application are only routine technical means for those skilled in the art related to the disclosure of the present application, and should not be understood as insufficient disclosure of the present application.

[0049] In the present application, the phrase "embodiment" means that the specific features, structures or properties described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in the present application should be understood as the usual meaning understood by those skilled in the art in the technical field to which the present application belongs. The terms "one", "a", "an", "the", and similar words involved in the present application do not represent quantity limitation, but can represent singular or plural. The terms "include", "contain", "have", and any variations thereof involved in the present application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device including a series of steps or modules (units) is not limited to the listed steps or units, but can also include steps or units not listed, or can also include other steps or units inherent to the process, method, product or device. The terms "connected", "connected", "coupled" and similar words involved in the present application are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The term "multiple" refers to two or more. The association between the associated objects is described by the term "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. The terms "first", "second", "third", and the like involved in the present application are only to distinguish similar objects, and do not represent a specific order for the objects.

[0051] The embodiment of the present application provides a face replacement method for group photos, Figure 1 is a step flowchart of the face replacement method for group photos according to the embodiment of the present application, as shown in Figure 1As shown, the method comprises the following steps:

[0052] Step S102, grouping the group photos of the user according to similarity to obtain group photo sets of different groups;

[0053] Step S102, specifically, obtaining the group photos authorized by the user, extracting the global features and local features of each group photo; grouping the group photos based on the global features and local features to obtain group photo sets of different groups.

[0054] It should be noted that the existing face replacement tool lacks a similar photo intelligent grouping and aggregation mechanism, and needs to rely on manual screening of similar photos, and it is difficult to automatically cluster based on scene semantics, character association or shooting time sequence, resulting in subsequent face retrieval in an unordered photo library, which is low in efficiency and easy to miss effective materials. In step S102, the user-authorized photos are grouped according to similarity, and the global features (scene semantic vectors) and local features (such as the face feature vectors of the main characters) of each photo are extracted; a density clustering algorithm is used to measure the feature similarity, and photos with similar content are aggregated into a group to ensure that the photos in the same group contain the same characters and have strong scene association, which can effectively avoid the omission of effective materials and significantly improve the subsequent group photo face replacement editing effect.

[0055] Step S104, performing face detection on all group photos in the target group photo set, and determining that the faces with a face similarity greater than a preset threshold are the same person to generate a person-face set mapping table;

[0056] Step S104 specifically comprises the following steps:

[0057] Step S1041, based on the selected to-be-edited group photo of the user, determining the target group photo set in which the to-be-edited group photo is located from the group photo sets of different groups;

[0058] Step S1042, performing face detection on all group photos in the target group photo set, and calculating the cosine similarity between different faces, and determining that the faces with a cosine similarity greater than a preset threshold are the same person to generate a person-face set mapping table.

[0059] It should be noted that the existing face replacement tool, even if the group is completed manually, but when matching the faces in the group, the face association accuracy of the same person in different photos is low due to the change of light and the difference in posture (such as side face and low head), it is difficult to completely extract all available expression materials of the person, and the range of expression selection is limited. And step S104, for each group of photo sets obtained by step S102, uses a face retrieval algorithm to detect the face area in all photos in the group, outputs the face coordinate frame and the face key point; extracts the face feature vector of each face through the face feature module; calculates the cosine similarity of all face feature vectors in the group, and determines the face with a similarity meeting a specified threshold as the same person, and aggregates to form a person-face set mapping table, further avoiding omission of effective materials, and significantly improving the subsequent group photo face replacement editing effect.

[0060] Step S106, using an expression quality evaluation model to select the face with the highest expression comprehensive score from the person-face set mapping table as the optimal expression face of the corresponding person;

[0061] Step S106 specifically includes the following steps:

[0062] Step S1061, based on the non-ideal face selected by the user from the to-be-edited group photo, determining the person corresponding to the non-ideal face through the person-face set mapping table;

[0063] Step S1062, using an expression quality evaluation model to score all faces in the person-face set mapping table of the person, and sorting them from high to low according to the expression comprehensive score, and selecting the first ranked face as the optimal expression face of the person.

[0064] It should be noted that the existing face replacement tool lacks expression ranking and optimization mechanism, and does not establish a quantitative evaluation system for multiple faces of the same person in the group, making it difficult to automatically select the "most natural" and "most suitable" expression in the group (such as a happy scene, it is not possible to preferentially select a smiling expression), which requires manual comparison one by one, and the operation is complicated and subjective. And step S106, based on the person-face set mapping table obtained by step S104, sorts the face set of each person according to the expression: constructs an expression quality evaluation model, inputs the face image, and outputs the expression comprehensive score, sorts in descending order according to the comprehensive score, selects the top 1 face as the "optimal expression face" of the person, establishes a face expression quantitative evaluation system, and automatically selects the optimal expression without manual comparison one by one.

[0065] Step S108, replacing the non-ideal face in the to-be-edited group photo with the optimal expression face through the trained group photo face replacement model.

[0066] Figure 2This is a schematic diagram of a face replacement method for group photos according to an embodiment of this application. Step S108 specifically includes the following steps:

[0067] Step S1081: Construct and train a group photo face replacement model to obtain a trained group photo face replacement model. The group photo face replacement model includes a multi-scale convolutional encoder, a Transformer mask generation module, an expression feature fusion module, and a UV image reconstruction module.

[0068] Specifically, step S1081:

[0069] ① Constructing the multi-scale convolutional encoder in the group photo face replacement model includes: stacking multiple ConvBlock modules sequentially to construct the multi-scale convolutional encoder in the group photo face replacement model. Each ConvBlock module contains a backbone branch and residual connections. The residual connections are adjusted by convolution and then added to the backbone branch to output F. uv16 Feature map, F uv8 Feature map and F uv4 Feature map; F uv16 Feature maps are used to capture local facial expression details, F uv8 Feature maps are used to model moderate structural changes, F uv4 The feature map is used to represent the whole; and the output of the multi-scale convolutional encoder is used as the input to the Transformer mask generation module.

[0070] It should be noted that the source AU vector v of the normalized non-ideal face src The target AU vector v of the optimal facial expression tgt (See subsequent step S1082) Before inputting the multi-scale convolutional encoder, the source image I needs to be converted using facial key points. src Mapping to 3D UV coordinate space establishes a unified spatial representation. Dynamic editing regions are defined within the 3D UV coordinate space and then input into a multi-scale convolutional encoder. This encoder consists of multiple stacked ConvBlock modules, each containing a backbone branch and residual connections. The residual connections are adjusted through convolution and added to the backbone output to enhance feature representation, outputting feature maps at three scales: F uv16 Used to capture local facial expression details, F uv8 Modeling moderate structural changes, F uv4 Represent the whole and ensure consistency in editing.

[0071] ②The Transformer mask generation module in the face replacement model includes: stacking the Transformer encoder in multiple layers to construct the Transformer mask generation module in the face replacement model, wherein each layer of the Transformer encoder includes a self-attention mechanism and a non-linear transformation based on a feedforward neural network FFN, and a layer normalization LayerNorm operation is introduced after each sub-layer; on the basis of the output of the multi-scale convolutional encoder, an identity feature vector f id participates in the calculation of attention weight as global guidance information to generate an identity-preserving mask Mask id and an expression dynamic mask Mask exp , wherein the identity-preserving mask Mask id is used to identify the identity stable region, the expression dynamic mask Mask exp is used to identify the expression region; and the output of the Transformer mask generation module is taken as the input of the expression feature fusion module.

[0072] It should be noted that the Transformer mask generation module takes the UV space feature map output by the multi-scale convolutional encoder as input, and respectively flattens each scale feature map (F uv16 , F uv8 , F uv4 ) into a two-dimensional visual Token sequence, and then performs concatenation after uniform projection to the same dimension to form a complete input Token sequence; at the same time, the identity feature vector f id is embedded as global guidance information participating in the calculation of attention weight. The output feature Token is reshaped into a spatial feature map, and is restored to the original UV image resolution through upsampling and channel mapping operations, and generates two binary mask maps Mask id (identity-preserving mask) and Mask exp (expression dynamic mask) through an activation function, wherein Mask id is used to identify the identity stable region, and Mask exp is used to identify the expression region, thereby effectively decoupling the identity and expression features in the spatial dimension, effectively solving the problem that the existing face replacement scheme directly pastes the target face region, resulting in identity features (such as face shape, skin color) and source image fusion being stiff, or expression and facial muscle direction being mismatched.

[0073] ③ The expression feature fusion module in the group photo face replacement model is constructed by combining CLIP text encoder and multilayer perceptron (MLP) to construct the expression feature fusion module in the group photo face replacement model; based on the output of the multi-scale convolutional encoder and Transformer mask generation module, the target expression description input by the user is introduced as semantic guidance to participate in expression feature fusion to generate expression fusion features; and the output of the expression feature fusion module is used as the input of the UV image reconstruction module.

[0074] It should be noted that the facial expression feature fusion module first starts from the Mask exp Extracting local facial expression features F from the region exp (F) exp = Mask exp *F uv16 ), It also incorporates user-inputted target facial expression descriptions, which are encoded into high-dimensional semantic vectors v using the CLIP text encoder. clip ∈R 512 Subsequently, it was compressed to the same size as F using a multilayer perceptron (MLP). exp Same channel dimension, denoted as v′ clip and with F exp Weighted fusion. This is combined with the reference graph dynamic vector v obtained from AU difference calculation. t tgt Construct a ternary fusion expression, the specific formula of which is:

[0075]

[0076] α, β, and γ are trainable weights, and this feature will be fed into the next UV image reconstruction module for dynamic region expression synthesis.

[0077] ④ The UV image reconstruction module in the group photo face replacement model is constructed by using a decoder based on the UNet structure.

[0078] It should be noted that the UV image reconstruction module uses a UNet-based decoder for feature upsampling during local expression transformation in the UV space. This decoder recovers the fused expression-driven feature map F layer by layer. fused The spatial dimensions are reduced to the original UV resolution. Each layer uses a transposed convolution operation to improve resolution and introduces a skip connection mechanism to extract features from the corresponding layer of the encoder to enhance local detail representation. This is combined with the mask output by the Transformer mask generation module. exp Limit the editing scope and output a UV space image I uv. Using linear interpolation, the pixels of I uv are mapped to a 2D image I gen (x, y). To avoid the appearance of holes at the boundaries of the UV mesh, the original image is blended into the hole region. The final image is a pixel-wise weighted blend, and the formula is:

[0079] I final (x, y) = Mask id (x, y) * I src (x, y) + (1 - Mask id (x, y)) * I gen (x, y)

[0080] Step S1082, the non-ideal face in the to-be-edited group photo and the optimal expression face are normalized to unify the face posture and size;

[0081] It should be noted that, as Figure 2 shown, the non-ideal face to be replaced in the to-be-edited group photo is positioned (which can be manually selected by the user, or automatically selected based on the score), and the image of the non-ideal face is defined as the source image I src . The AU vector of the non-ideal face is extracted by the OpenFace tool, and is defined as the source AU vector v src . The AU vector of the optimal expression face is extracted: the target AU vector v tgt . The source image I src is positioned by the face key point, and is aligned to the coordinate space of the standard face template by similarity transformation, so as to unify the face posture and scale, and normalize the source AU vector v tgt and the target AU vector v src .

[0082] Step S1083, as Figure 2 shown, the normalized to-be-edited group photo and the optimal expression face are input into the trained group face replacement model to replace the non-ideal face in the to-be-edited group photo with the optimal expression face.

[0083] Through the above steps in the embodiments of the present application, automatic retrieval and grouping of group photos is realized, which can effectively avoid omission of effective materials, and on the basis of grouping to obtain a group collection, the faces of different persons are accurately classified, further, an unideal face of a specific person in the to-be-edited group photo is replaced by a face with the highest expression score, which significantly improves the group face replacement and editing effect. In addition, a face expression quantification evaluation system is established, and the optimal expression is automatically selected without manual one-by-one comparison. The identity and expression features are effectively decoupled in the spatial dimension, effectively solving the problem that the existing face replacement scheme directly pastes the target face region, resulting in the fusion of identity features (such as face shape and skin color) and the source image being stiff, or the expression and facial muscle direction being mismatched.

[0084] In some embodiments, for the training of the group face replacement model constructed in step S1081, the training process specifically includes the following steps:

[0085] Training data description:

[0086] Collect the face images of authorized users under different expressions to construct an expression editing training set. Each pair of images includes: a source image I src , a target AU vector v tgt , a source AU vector v src , and an identity feature f id extracted by a face feature module, to obtain a neural network training data set AlltuFace, which is divided into a training set T train and a test set T test .

[0087] Model network training:

[0088] In this example, an Intel i7-14900k processor, an Ubuntu operating system, 128 GB of memory, 4 NVIDIA RTX 4090 graphics cards, a Pycharm 2020.1.1 editing tool, and a Pytorch 2.7.0 environment are used to train and test the model. The size of each batch of samples is set to 16, the maximum number of iterations is set to 200, the learning rate is set to lr 2e-5, the loss function value is calculated once per iteration, and the loss value of the validation set is updated once per iteration. When the number of iterations reaches the maximum number of iterations, the training is stopped and the model is saved.

[0089] Loss function of the network:

[0090]

[0091] Wherein, λ1 to λ5 are weight coefficients corresponding to each loss function.

[0092] Expression vector consistency loss LAU :

[0093]

[0094] where R AU denotes the pre-trained AU prediction network, input face image, output facial action unit vector (AU vector) to describe expression action, I gen denotes the generated image, the replacement output by the model, V tgt denotes the target AU vector, i.e. the expression features that the generated image is expected to have.

[0095] L1 pixel reconstruction loss L pixel :

[0096]

[0097] where I gt denotes the real image.

[0098] Identity preservation loss function L id (minimize the cosine distance between ):

[0099]

[0100] where ArcFace denotes the pre-trained face recognition model, input face image, output high-dimensional identity feature vector. I src denotes the original face image to be edited in the face replacement task. denotes the identity feature vector of the source image, denotes the identity feature vector of the generated image.

[0101] Perception loss L perc :

[0102]

[0103] where, denotes the feature extraction function of the l-th layer in the VGG19 network (conv1_2, conv2_2, conv3_2, etc. layers can be taken), which extracts the perception features of the image at this layer.

[0104] Semantic consistency loss L clip :

[0105]

[0106] where, is the image semantic vector extracted by the CLIP image encoder, is the target semantic vector extracted by the CLIP text encoder.

[0107] It should be noted that the steps shown in the above flow or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0108] The embodiment of the present application provides a face replacement system for group photos, which is used for executing the method of the above embodiment, and the system comprises a group photo grouping module, a face discrimination module, an expression scoring module and a face replacement module.

[0109] The group photo grouping module is used for grouping the group photos of a user according to similarity to obtain group photo sets of different groups.

[0110] The face discrimination module is used for performing face detection on all group photos in a target group photo set, judging faces with a face similarity greater than a preset threshold as the same person, and generating a person-face set mapping table.

[0111] The expression scoring module is used for selecting a face with the highest expression comprehensive score from the person-face set mapping table as an optimal expression face of the corresponding person by using an expression quality evaluation model.

[0112] The face replacement module is used for replacing an undesirable face in an edited group photo with the optimal expression face by using a trained group photo face replacement model.

[0113] Through the group photo grouping module, the face discrimination module, the expression scoring module and the face replacement module in the embodiment of the present application, automatic retrieval and grouping of group photos are realized, omission of effective materials can be effectively avoided, different faces of different persons are accurately classified on the basis of the grouped group photo sets, and an undesirable face of a specific person in an edited group photo is further replaced with a face with the highest expression score, so that the editing effect of group photo face replacement is significantly improved.

[0114] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented by software or hardware. For the module implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combination.

[0115] The embodiment provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the method embodiments.

[0116] Optionally, the electronic device further comprises a transmission device connected with the processor and an input and output device connected with the processor.

[0117] Optionally, the electronic device further comprises a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the electronic device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a method for replacing faces in a group photo. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0118] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and this embodiment will not be described here.

[0119] In addition, in combination with the method for replacing faces in a group photo in the above embodiments, the embodiments of the present application can provide a storage medium for implementation. The storage medium stores a computer program; the computer program is executed by the processor to implement any one of the methods for replacing faces in a group photo in the above embodiments.

[0120] In one embodiment, Figure 3 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, as Figure 3 shown, an electronic device is provided, which can be a server, and the internal structure diagram thereof can be as Figure 3 shown. The electronic device comprises a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is configured to provide computing and control capabilities, the network interface is configured to communicate with an external terminal through a network connection, the internal memory is configured to provide an environment for the operating system and the computer program to run, the computer program is executed by the processor to implement a method for replacing faces in a group photo, and the database is configured to store data.

[0121] Those skilled in the art can understand that, Figure 3The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM) and the like.

[0123] Those skilled in the art should understand that each technical feature of the above-mentioned embodiments can be combined arbitrarily, and in order to make the description simple, each technical feature of the above-mentioned embodiments is not described all possible combinations, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present application.

[0124] The above-mentioned embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A method of replacing a face in a group photo, characterized by, The method comprises: grouping the group photos of the user by similarity to obtain group photo sets of different groups; performing face detection on all group photos in the target group photo set, determining faces with a face similarity greater than a preset threshold as the same person, and generating a person-face set mapping table; selecting a face with the highest comprehensive expression score from the person-face set mapping table as the optimal expression face of the corresponding person using an expression quality evaluation model; constructing and training a group face replacement model to obtain a trained group face replacement model, wherein the group face replacement model comprises a multi-scale convolutional encoder, a Transformer mask generation module, an expression feature fusion module, and a UV image reconstruction module; normalizing the non-ideal face in the to-be-edited group photo and the optimal expression face to unify the face posture and size; inputting the normalized to-be-edited group photo and the optimal expression face into the trained group face replacement model to replace the non-ideal face in the to-be-edited group photo with the optimal expression face; wherein the Transformer mask generation module in the construction of the group face replacement model comprises: stacking the Transformer encoder in multiple layers to construct the Transformer mask generation module in the group face replacement model, wherein each layer of the Transformer encoder comprises a self-attention mechanism and a non-linear transformation based on a feedforward neural network FFN, and a layer normalization LayerNorm operation is introduced after each sub-layer; On the basis of the output of the multi-scale convolutional encoder, an identity feature vector f id Participate in the calculation of attention weight as global guidance information to generate identity preservation mask Mask id And expression dynamic mask Mask exp Wherein, the identity preservation mask Mask id For identifying identity stable area, the expression dynamic mask Mask exp For identifying expression area; the output of the Transformer mask generation module is used as the input of the expression feature fusion module.

2. The method of claim 1, wherein, The multi-scale convolutional encoder in the construction of the group face replacement model comprises: A plurality of ConvBlock modules are sequentially stacked to build a multi-scale convolutional encoder in the composite face replacement model, wherein each ConvBlock module contains a trunk branch and a residual connection, the residual connection is added to the trunk branch after adjusting the channel through convolution to output F uv16 feature maps, F uv8 feature maps and F uv4 feature maps; the F uv16 feature maps are used to capture local expression details, the F uv8 feature maps are used to model moderate structural changes, the F uv4 feature maps are used to represent the whole; the output of the multi-scale convolutional encoder is used as the input of the Transformer mask generation module.

3. The method of claim 1, wherein, The expression feature fusion module in the construction of the group face replacement model comprises: combining a CLIP text encoder and a multi-layer perceptron MLP to construct the expression feature fusion module in the group face replacement model; based on the outputs of the multi-scale convolutional encoder and the Transformer mask generation module, introducing a target expression description input by a user as semantic guidance to participate in expression feature fusion to generate expression fusion features; the output of the expression feature fusion module is used as the input of the UV image reconstruction module.

4. The method of claim 1, wherein, The UV image reconstruction module in the construction of the group face replacement model comprises: a UNet structure-based decoder is used to construct the UV image reconstruction module in the group face replacement model.

5. The method of claim 1, wherein, Grouping the group photos of the user by similarity to obtain group photo sets of different groups comprises: obtaining group photos authorized by the user, extracting global features and local features of each group photo; grouping the group photos based on the global features and the local features to obtain group photo sets of different groups.

6. The method of claim 1, wherein, Performing face detection on all group photos in the target group photo set, determining faces with a face similarity greater than a preset threshold as the same person, and generating a person-face set mapping table comprises: determine, based on the selected to-be-edited group photo, a target group photo set from the different group photo sets in which the to-be-edited group photo is located; perform face detection on all group photos in the target group photo set, and calculate cosine similarity between different faces, and determine faces with cosine similarity greater than a preset threshold as the same person, and generate a person-face set mapping table.

7. The method of claim 6, wherein, select, by using an expression quality evaluation model, a face with the highest expression comprehensive score from the person-face set mapping table as the optimal expression face of the corresponding person, including: based on a user-selected undesirable face in the to-be-edited group photo, determine the person corresponding to the undesirable face through the person-face set mapping table; score all faces in the person-face set mapping table of the person by using the expression quality evaluation model, and sort them from high to low according to the expression comprehensive score, and select the first-ranked face as the optimal expression face of the person.

8. A face replacement system for group photos, characterized by, The system is used to perform the method of any one of claims 1 to 7, and the system includes a group photo grouping module, a face discrimination module, an expression scoring module, and a face replacement module. The group photo grouping module is used to group the group photos of a user according to similarity, and obtain different group photo sets. The face discrimination module is used to perform face detection on all group photos in the target group photo set, determine faces with face similarity greater than a preset threshold as the same person, and generate a person-face set mapping table. The expression scoring module is used to select, by using an expression quality evaluation model, a face with the highest expression comprehensive score from the person-face set mapping table as the optimal expression face of the corresponding person. The face replacement module is used to replace an undesirable face in a to-be-edited group photo with the optimal expression face by using a trained group photo face replacement model.

Images