Face fusion method and system based on identity preservation and potential diffusion

Abstract

The application discloses a face fusion method and system based on identity preservation and potential diffusion. The method comprises the following steps: preprocessing a target base map and an identity reference map, performing face detection and segmentation, establishing an ordered reference queue and a mapping relationship, obtaining or generating an external mask map, and aligning and standardizing; extracting geometric structure and identity features from a mapping pair set to construct a composite conditional constraint model and impose conditions in a diffusion network; generating a new face image through a two-stage sampling method, and then optimizing the image quality through steps such as geometric matching, color and skin consistency and texture enhancement. The optimized face image is fused back into the target base map according to the transparency of the external mask map, and a final fusion image is obtained, and the entire process records relevant parameters. Through the method of the application, efficient, stable and high-quality face fusion can be realized in a multi-person scene, and various defects in the prior art are overcome to meet the growing demand for high fidelity.

Description

Technical Field

[0001] This invention relates to computer vision, and more specifically to a face fusion method and system based on identity preservation and potential diffusion. Background Technology

[0002] In the field of digital image processing, face replacement technology has made significant progress, mainly including two types of methods: one is the traditional face-swapping technology based on geometric alignment and feature mapping, such as single face replacement; the other is identity transfer technology using diffusion models, such as redrawing using only a single identity constraint model. These technologies can effectively achieve basic face-swapping effects in single-person scenarios, but they expose a series of problems when facing multi-person scenarios and high-fidelity requirements.

[0003] While the aforementioned technologies perform well in specific situations, they face challenges in multi-person scenarios. First, the "source face-target face" correspondence is unstable, leading to mismatches, face mixing, or disordered replacement order. Second, single-identity constraint schemes struggle to balance "identity consistency" and "texture naturalness," resulting in generated faces that lack realism and exhibit distorted skin texture. Furthermore, the mask boundary processing capabilities are insufficient for detailed areas such as the hairline, ears, and neck, easily resulting in hard edges, color bleeding, and unnatural transitions, further affecting the realism of the composite effect. Moreover, current methods heavily rely on manually drawn masks, resulting in low automation levels, and the parameter settings and workflow design lack reusability, increasing deployment costs. Simultaneously, during multiple rounds of local restoration, frequent overall tone drift occurs, causing inconsistencies between the blended area and the original background, disrupting the overall harmony of the image.

[0004] Therefore, it is necessary to design a new method to achieve efficient, stable and high-quality face fusion in multi-person scenarios, overcome the various shortcomings of existing technologies, and meet the growing demand for high fidelity. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a face fusion method and system based on identity preservation and potential diffusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a face fusion method based on identity preservation and potential diffusion, comprising:

[0007] Acquire and preprocess the target base map and identity reference map, and initialize the control parameters;

[0008] The identity reference image is subjected to face detection and segmentation, and the corresponding valid faces are filtered and sorted to form an ordered reference queue;

[0009] Face detection and segmentation are performed on the target base map, the corresponding valid faces are filtered and sorted, and a mapping relationship is established with the ordered reference queue to obtain a set of mapping pairs;

[0010] Obtain an external mask image, or generate an external mask image for valid faces on the set of mapping pairs;

[0011] Align the external mask image and perform normalization to obtain the processing result;

[0012] Based on the processing results, geometric structure and identity features are extracted for each mapping pair in the mapping pair set, a composite condition constraint model is constructed, and conditions are applied in the diffusion inference network to achieve consistency between structure and identity.

[0013] Based on the aforementioned conditions, a new facial image is generated using a two-stage sampling method.

[0014] The new facial image is subjected to geometric matching, color homogenization, and texture enhancement to obtain a processed facial image;

[0015] The processed facial image is fused back to the target base image based on the transparency of the outer mask image to obtain a fused image, and relevant parameters are recorded.

[0016] The further technical solution is as follows: The process of performing face detection and segmentation on the identity reference image, filtering and sorting the corresponding valid faces to form an ordered reference queue includes:

[0017] Perform face detection and semantic segmentation on the identity reference map to obtain candidate face frames and corresponding segmented regions;

[0018] Calculate the effective area of ​​each candidate face frame and filter them, retaining the candidate face frames that meet the threshold conditions, and construct an ordered reference identity queue according to a unified sorting rule.

[0019] The further technical solution is as follows: Face detection and segmentation are performed on the target base image, corresponding valid faces are filtered and sorted, and a mapping relationship is established with the ordered reference queue to obtain a set of mapping pairs, including:

[0020] Face detection and semantic segmentation are performed on the target base map to obtain candidate face boxes and corresponding segmentation regions. The effective area of ​​each candidate face box is calculated and filtered. Candidate face boxes that meet the threshold conditions are retained, and a target identity queue is constructed according to a unified sorting rule.

[0021] Establish a one-to-one mapping relationship between the ordered reference queue and the target identity queue. When the number of the two queues is inconsistent, the smaller number is used as the effective mapping number for processing, and the excess part is kept unchanged or processed according to a preset strategy.

[0022] The further technical solution is as follows: aligning the external mask image and performing standardization processing to obtain the processing result includes:

[0023] The external mask image is then subjected to size alignment, coordinate alignment, and transparency normalization. Finally, hole filling, edge refinement, shape expansion, edge smoothing, and transparency remapping are performed to obtain the processing result.

[0024] The further technical solution is as follows: Based on the processing result, geometric structure and identity features are extracted for each mapping pair in the mapping pair set, a composite conditional constraint model is constructed, and conditions are applied in the diffusion inference network to achieve consistency between structure and identity, including:

[0025] Based on the processing results, the target base map and identity reference map of each mapping pair in the mapping pair set are cropped with an expanded box and scaled to a uniform size to obtain two processed images.

[0026] The facial key points of the two processed images are located and the head pose is estimated, and then converted into a geometric conditional representation map.

[0027] The two geometric condition representation maps are aligned with the reference face through a similarity transformation and then input into a feature extraction network to obtain an identity vector;

[0028] The identity vector is combined with its geometric structure and identity features, and then fused into a composite condition according to weights.

[0029] The further technical solution is as follows: based on the processing result, extracting geometric structure and identity features for each mapping pair in the mapping pair set, constructing a composite conditional constraint model, and applying conditions in the diffusion inference network to achieve consistency between structure and identity, further includes:

[0030] If the difference between the two geometric condition representations does not meet the requirements, a rough face swap or geometric adjustment is first performed to create an initial reference face that is closer to the target structure, and then the identity vector is generated.

[0031] The further technical solution is as follows: Based on the aforementioned conditions, generating a new facial image using a two-stage sampling method includes:

[0032] The target base map is converted into a latent representation by an encoder, and a latent mask defining the variation region is generated based on the external mask map. Noise is added only within the region of the latent mask according to a specified random seed to obtain noisy latent variables.

[0033] Under the given conditions, the noisy latent variables are subjected to diffusion inversion processing to obtain the first feature, which is then decoded into a preliminary fused facial image.

[0034] The preliminary fused facial image is then refined by sampling high-sensitivity areas to obtain a new facial image.

[0035] The further technical solution is as follows: performing geometric matching, color unification, and texture enhancement on the new facial image to obtain the processed facial image includes:

[0036] The position and size of the new facial image are adjusted by calculating scaling, translation, or affine transformation parameters;

[0037] The color distribution of the new facial image is statistically analyzed and adjusted to match the skin tone and lighting conditions of the target base image.

[0038] Choose whether to perform super-resolution processing or detail restoration on the adjusted facial image as needed to obtain the processed facial image.

[0039] A further technical solution is as follows: the processed facial image is fused back to the target base image based on the transparency of the external mask image to obtain a fused image, and relevant parameters are recorded, including:

[0040] The write-back region is determined by the intersection of the extended box and the image boundary, and the effective fusion region is defined based on the non-zero value portion of the standard fusion mask;

[0041] Within the write-back area, the processed facial image and the target base image are weighted and fused at the pixel level according to the transparency of the external mask image to generate an intermediate image. For cases with multiple faces, the process is repeated and the intermediate image is updated. When overlap occurs, the write-back order is adjusted and the external mask image is optimized according to an area or order priority strategy to obtain a fused image.

[0042] Record all key parameters.

[0043] The present invention also provides a face fusion system based on identity preservation and potential diffusion, comprising:

[0044] The acquisition unit is used to acquire and preprocess the target base map and identity reference map, and initialize the control parameters;

[0045] The first processing unit is used to perform face detection and segmentation on the identity reference image, filter and sort the corresponding valid faces, and form an ordered reference queue.

[0046] The second processing unit is used to perform face detection and segmentation on the target base map, filter and sort the corresponding valid faces, and establish a mapping relationship with the ordered reference queue to obtain a set of mapping pairs.

[0047] A mask image forming unit is used to acquire an external mask image, or to generate an external mask image for valid faces on the set of mapping pairs;

[0048] An alignment processing unit is used to align the external mask image and perform normalization processing to obtain the processing result;

[0049] The mapping extraction unit is used to extract geometric structure and identity features for each mapping pair in the mapping pair set based on the processing results, construct a composite condition constraint model, and apply conditions in the diffusion inference network to achieve consistency between structure and identity.

[0050] A dual sampling unit is used to generate a new facial image based on the conditions using a two-stage sampling method;

[0051] An image processing unit is used to perform geometric matching, color unification, and texture enhancement on the new facial image to obtain a processed facial image.

[0052] The fusion unit is used to fuse the processed facial image back to the target base image according to the transparency of the external mask image to obtain a fused image, and record relevant parameters.

[0053] The advantages of this invention compared to existing technologies are as follows: This invention acquires and preprocesses a target base map and an identity reference map, uses face detection and segmentation techniques to filter and sort valid faces to form an ordered reference queue, and establishes a mapping relationship with the faces in the target base map. Then, an external mask map is generated automatically or manually and subjected to alignment and standardization processing. A composite conditional constraint model is constructed for each mapping pair, and structural and identity consistency conditions are applied in the diffusion inference network. A two-stage sampling method is used to generate new facial images. Subsequently, geometric matching, color unification, and texture enhancement are performed on the new facial images. Finally, based on the transparency of the external mask map, the optimized facial images are seamlessly fused back to the target base map, thereby achieving efficient, stable, and high-quality face fusion in multi-person scenarios. This method overcomes the shortcomings of existing technologies in terms of mismatch probability, identity preservation, and natural transition, meets the requirements of high fidelity, and ensures the authenticity and consistency of the synthesized results.

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A schematic flowchart illustrating the face fusion method based on identity preservation and potential diffusion provided in an embodiment of the present invention;

[0057] Figure 2 A schematic block diagram of a face fusion system based on identity preservation and potential diffusion provided in an embodiment of the present invention;

[0058] Figure 3 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0061] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0062] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0063] Please see Figure 1 , Figure 1This is a flowchart illustrating the face fusion method based on identity preservation and potential diffusion provided in an embodiment of the present invention. This face fusion method based on identity preservation and potential diffusion is applied in a server. Through a series of refined steps, this method achieves efficient, stable, and high-quality face fusion in multi-person scenarios, overcoming several limitations of existing technologies. First, face detection, segmentation, and ordered queue construction are performed on the target base map and identity reference map, establishing a mapping relationship to ensure accurate correspondence for each individual. Then, alignment and standardization are performed using an external mask image, and a composite conditional constraint model is constructed by combining geometric structure and identity feature extraction. Specific conditions are applied in the diffusion inference network to ensure consistency between structure and identity. Next, a two-stage sampling method is used to generate new facial images, which undergo geometric matching, color unification, and texture enhancement to optimize visual effects. Finally, the processed facial images are accurately fused back into the target base map based on the mask image transparency to obtain the final fused image. The entire process ensures stability and efficiency under high-fidelity requirements through precise parameter control and optimization strategies, while also meeting the increasing demands of practical applications.

[0064] Figure 1 This is a schematic flowchart of the face fusion method based on identity preservation and potential diffusion provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S190.

[0065] S110. Obtain and preprocess the target base map and identity reference map, and initialize the control parameters.

[0066] In this embodiment, the target base image refers to the underlying image for the face fusion operation. It typically contains one or more faces and serves as the basic canvas for the entire process. This base image can be a still image extracted from a video frame or a photograph. Before subsequent processing, the target base image needs to be preprocessed, including but not limited to resizing, color correction, and noise reduction, to ensure optimal processing results.

[0067] An identity reference map refers to a facial image provided to the system for transferring or replicating specific facial features. It can be a single image or a collection of images, depending on the complexity and diversity of the required identity features. These images are used to extract facial features, such as shape, structure, and texture, to achieve accurate face replacement or fusion on the target base map. Figure 1 Similarly, identity reference images also need to undergo corresponding preprocessing steps, such as face detection, segmentation, and quality assessment, to improve the accuracy of subsequent processing.

[0068] Control parameters refer to a series of initial settings defined to ensure the controllability of the entire processing and the repeatability of the results. These parameters cover all aspects from input data selection to the execution of the specific algorithm:

[0069] Face count parameter: Specifies the maximum number of faces allowed in this processing.

[0070] Detection threshold parameter: Set the confidence threshold for the face detection stage to filter out unreliable detection results.

[0071] Area threshold parameter: excludes candidate regions whose effective area is below a certain standard to avoid processing invalid face regions that are too small.

[0072] Sorting parameters: Determine how to sort the identified faces based on their position (such as horizontal coordinates) or size, thereby establishing a stable processing order.

[0073] Mask shape parameters: These involve setting the mask expansion radius, edge smoothness, and transparency remapping range, which affect the naturalness of the final blending effect.

[0074] Identity constraint weight parameter: Adjusts the balance between geometric structure constraints and identity feature constraints to achieve the ideal fusion effect.

[0075] Diffusion sampling parameters: control the specific number of sampling steps, denoising intensity, and local repair range in the first and second stages when generating a new facial image.

[0076] Blending parameters: determine how the target base image is blended with the processed new facial image at the pixel level, including blending intensity and feathering range.

[0077] Random seed parameter: ensures that the same results are produced under the same initial conditions each time the program is run, which facilitates experimental comparison and result verification.

[0078] First, three types of input data are acquired: a target base image, which is the foundational image for the fusion output, typically an original image containing one or more faces; an identity reference image, which provides the face images for transferring identity features, and can be single or multiple images; and an external mask image, which is an optional manually annotated fusion region image. Then, parameters are initialized, including at least: a face quantity parameter to limit the upper limit of faces processed in this round; a detection threshold parameter to control the confidence filtering of face candidates; an area threshold parameter to remove excessively small or invalid face regions; a sorting parameter to define a stable order based on horizontal position or area; a mask morphology parameter to control the mask expansion radius, edge smoothing intensity, and transparency remapping range; an identity constraint weight parameter to control the contribution ratio of structural constraints and identity constraints; a diffusion sampling parameter to control the number of sampling steps, denoising intensity, and local repair range in the first and second stages; a fusion parameter to control the region fusion intensity and feathering range; and a random seed parameter to ensure the reproducibility of the results. Through these definitions, each processing step in the subsequent process has clear inputs and controllable boundary conditions.

[0079] The detailed definitions and initializations provided above offer clear and specific operational guidelines and boundary conditions for each subsequent processing step, making the entire process more systematic and standardized.

[0080] S120. Perform face detection and segmentation on the identity reference image, filter and sort the corresponding valid faces to form an ordered reference queue.

[0081] In this embodiment, the ordered reference queue refers to a list of face information generated after a specific processing procedure. Each face in this list is represented by its corresponding bounding box (i.e., candidate face box) and segmented region. These candidate face boxes not only contain location information but are also filtered by calculating the effective area to ensure that only faces meeting certain quality standards are retained. Furthermore, these faces are sorted according to a unified rule (e.g., from left to right by horizontal coordinates), thus forming a structured set. This ordered reference queue allows subsequent processing steps to handle multiple face cases more systematically and automatically, avoiding inconsistencies in results caused by changes in the input order.

[0082] In one embodiment, step S120 described above may include steps S121 to S122.

[0083] S121. Perform face detection and semantic segmentation on the identity reference map to obtain candidate face frames and corresponding segmentation regions.

[0084] In this step, advanced face detection algorithms are first applied to identify the location of all faces in the identity reference image, and a bounding box (candidate face box) is assigned to each detected face. Next, semantic segmentation techniques are used to precisely extract the facial feature regions within each candidate face box, resulting in refined segmentation results. This process not only helps determine the approximate location of the faces but also provides more detailed facial contour information, laying the foundation for subsequent processing.

[0085] S122. Calculate the effective area of ​​each candidate face frame and filter them, retain the candidate face frames that meet the threshold conditions, and construct an ordered reference identity queue according to a unified sorting rule.

[0086] In this embodiment, firstly, based on the candidate face bounding boxes and their segmented regions obtained in the previous step, the effective area of ​​each candidate face is calculated. This typically involves assessing the actual space occupied based on the segmentation results and removing invalid small regions that may be caused by detection errors or image quality issues.

[0087] Then, an area threshold is set as a filtering criterion, retaining only candidate face outlines with an effective area greater than or equal to that threshold. This helps to eliminate low-quality or unnecessary data, focusing on processing the most representative and clear faces.

[0088] Finally, a uniform sorting rule (such as arranging from left to right according to the horizontal coordinate) is used to arrange the filtered face boxes in an orderly manner, creating a stable ordered reference queue. This sorting method is particularly suitable for processing in multi-person scenarios because it can ensure that the consistency and stability of the output can be maintained even if the input order changes between different runs.

[0089] Through the detailed steps described above, the "identity source" is standardized into a structured set that can be processed in a cyclical manner, avoiding the instability of the identity source order in multi-person scenarios. This enables efficient processing of the identity reference graph and ensures that the basic data for subsequent operations such as face fusion is accurate and orderly.

[0090] S130. Perform face detection and segmentation on the target base map, filter and sort the corresponding valid faces, and establish a mapping relationship with the ordered reference queue to obtain a set of mapping pairs.

[0091] In this embodiment, the mapping pair set refers to a one-to-one mapping relationship formed between an ordered reference queue (i.e., a list of face information obtained from the identity reference map) and a target identity queue (i.e., a list of face information obtained from the target base map). This set ensures that when performing operations such as face replacement, the correspondence of "who replaces whom" is clear and stable, thereby avoiding the problems of face matching or mismatch.

[0092] In one embodiment, step S130 described above may include steps S131 to S132.

[0093] S131. Perform face detection and semantic segmentation on the target base map to obtain candidate face boxes and corresponding segmentation regions, calculate the effective area of ​​each candidate face box and filter them, retain candidate face boxes that meet the threshold conditions, and construct a target identity queue according to a unified sorting rule.

[0094] In this embodiment, a face detection algorithm is first applied to the target base map to identify the location of all faces, and a bounding box (candidate face box) is assigned to each detected face. Then, semantic segmentation technology is used to extract the facial feature regions within each candidate face box.

[0095] Based on the segmentation results, the effective area of ​​each candidate face bounding box is calculated to evaluate its quality. An area threshold is set as a filtering criterion, retaining only candidate face bounding boxes with an effective area greater than or equal to the threshold, thus removing low-quality or unnecessary data.

[0096] By using the same sorting rules (e.g., horizontal coordinates from left to right), the filtered face bounding boxes are arranged in an orderly manner to create a target identity queue. This step ensures that the processing order remains consistent even in multi-person scenarios.

[0097] In this embodiment, the target identity queue refers to a structured list of face information generated on the target base map through the above steps. This list includes the bounding box and segmented regions of each face, and is sorted according to certain rules. This queue not only helps determine the specific location and range of each face in the target base map but also provides a basis for subsequent matching with the reference queue.

[0098] S132. Establish a one-to-one mapping relationship between the ordered reference queue and the target identity queue. When the number of the two queues is inconsistent, the smaller number is used as the effective mapping number for processing, and the excess part is kept unchanged or processed according to a preset strategy.

[0099] In this embodiment, a one-to-one mapping relationship is established based on the position information and sorting rules of the face bounding boxes in the two queues. This means that each member in the reference queue will find a corresponding one in the target identity queue for subsequent face fusion or other operations.

[0100] When the number of faces in the ordered reference queue and the target identity queue are different, the smaller number is selected as the effective mapping number. For the portion exceeding the effective mapping number, the original image can be left unchanged, or special processing can be performed according to a preset strategy, such as ignoring this extra facial information or applying specific processing methods.

[0101] Through the above steps, multiple faces can be accurately located and processed on the target base map, and a stable and reliable mapping relationship can be established with the reference queue, ensuring the accuracy and consistency of subsequent operations. This approach is particularly suitable for applications requiring complex image processing in multi-person scenarios, such as virtual reality and film and television special effects production.

[0102] S140. Obtain an external mask image, or generate an external mask image for valid faces on the mapping pair set.

[0103] In this embodiment, the external mask refers to a binarized or grayscale image used to define the image processing area. It specifies which parts should be preserved (original image) and which parts should be replaced (new image) during operations such as face fusion. If the user provides an external mask, it is used directly; otherwise, a mask is automatically generated based on the result of the target face segmentation. This mask not only defines the fusion boundary but also controls the transition effect during the fusion process through different transparency values ​​(ranging from 0 to 1), where 0 indicates that the original image area is completely preserved, 1 indicates that the fusion result is fully adopted, and intermediate values ​​represent gradual transition areas.

[0104] S150. Align the external mask image and perform normalization to obtain the processing result.

[0105] In this embodiment, the processing result refers to the standard fusion mask obtained after a series of processing steps. This mask has undergone processes such as size alignment, coordinate alignment, opacity normalization, hole filling, edge thinning, shape dilation, edge smoothing, and opacity remapping to ensure its seamless application in subsequent face fusion or other image processing tasks. Specifically:

[0106] Size alignment: Adjust the size of the outer mask image to match the size of the target base image and the reference image.

[0107] Coordinate alignment: The position of the external mask map is precisely mapped to the face position on the target base map to ensure that each face area that needs to be processed can be accurately covered.

[0108] Transparency normalization: Unify the transparency values ​​of the outer mask image so that the transparency values ​​of all pixels are within a specified range (e.g., between 0 and 1), which facilitates subsequent calculations and processing.

[0109] Hole filling: Detects and fills holes in the mask image, ensuring that no undefined areas affect the final result.

[0110] Edge refinement: Optimize the edges of the mask image to reduce jagged edges and improve fusion quality.

[0111] Shape expansion: Appropriately expand the effective area of ​​the mask as needed to avoid important features being accidentally clipped.

[0112] Edge smoothing: Further softens the edges of the mask image, making the blending transition more natural.

[0113] Transparency remapping: Redistributing the transparency values ​​of different areas in the mask image according to specific needs to achieve the best visual effect.

[0114] These processing steps work together to resolve issues such as harsh borders, unnatural hairlines, and abrupt skin color changes, ensuring that the merged face image looks more realistic and harmonious. Through this series of meticulous preprocessing steps, both automatically generated and externally provided mask images can effectively support high-quality image compositing and editing tasks.

[0115] Specifically, the external mask image is subjected to size alignment, coordinate alignment, and transparency normalization, followed by hole filling, edge refinement, shape expansion, edge smoothing, and transparency remapping to obtain the processing result.

[0116] S160. Based on the processing results, extract the geometric structure and identity features for each mapping pair in the mapping pair set, construct a composite condition constraint model, and apply conditions in the diffusion inference network to achieve consistency between structure and identity.

[0117] In one embodiment, step S160 described above may include steps S161 to S165.

[0118] S161. Based on the processing result, the target base map and identity reference map of each mapping pair in the mapping pair set are expanded and cropped to a uniform size to obtain two processed images.

[0119] This step involves cropping the target base image (Target, Ti) and the identity reference image (Reference, Ri) according to their respective bounding boxes B'_tgt,i and B'_ref,i, and adjusting them to a preset resolution size (such as 512 or 1024 pixels) to ensure scale consistency in subsequent processing. This step is crucial for reducing errors caused by differences in the original image size.

[0120] S162. Locate facial key points and estimate head pose for the two processed images, and convert them into geometric conditional representations.

[0121] In this embodiment, the geometric condition representation map refers to a graphical representation that can describe the location of facial key points and the relationships between them. It may include key point heatmaps, skeleton connection diagrams, or dense pose fields, etc. These representation methods are used to capture and express the geometric characteristics of the face, such as position, orientation, and expression structure, so as to maintain the original geometric structure of the face unchanged during subsequent face fusion.

[0122] S163. Align the two geometric condition representation maps with the reference face through a similarity transformation, and input them into the feature extraction network to obtain the identity vector.

[0123] In this embodiment, the identity vector refers to a high-dimensional vector extracted from a face image using a specific algorithm (such as a deep learning network) that uniquely identifies a person's identity characteristics. This vector is typically normalized to characterize the identifiable attributes of the person. By performing feature embedding operations on an aligned reference face, the impact of pose differences on identity feature extraction can be effectively reduced, thereby obtaining a more accurate identity vector.

[0124] The feature extraction network can be the deep learning network mentioned above, etc.

[0125] S164. The identity vector is combined with the geometric structure and identity features, and then fused into a composite condition according to the weights.

[0126] In this embodiment, the composite condition, or condition, refers to the constraint formed by combining the geometric condition GeoCond_i and the identity condition IdEmb_i according to their respective weights w_geo and w_id. This composite condition can guide the noisy prediction network ε̂_t at different sampling steps of the diffusion inference network, so that the generated new faces retain both the geometric structure consistency of the target ROI and the identity feature consistency of the reference ROI. Specifically, structural consistency can be constrained by applying residual control to the intermediate features of the U-Net through the control branch, and identity consistency can be constrained by changing the identity-related response across attention through the attention adapter.

[0127] S165. When the difference between the two geometric condition representation diagrams does not meet the requirements, a rough face swap or geometric adjustment is first performed to create an initial reference face that is closer to the target structure, and then the identity vector is generated.

[0128] When there are significant differences in pose or expression between the target ROI and the reference ROI, directly extracting identity features may lead to unsatisfactory fusion results. Therefore, this step proposes a pre-emptive identity adaptation mechanism, which first performs a coarse face swap or geometric adjustment operation to generate a "reference initialized face" that more closely resembles the target structure, and then generates the identity vector based on this adjusted image. This significantly reduces the difficulty of subsequent optimization stages while improving the stability of the entire process and the quality of the final synthesized image.

[0129] In this embodiment, the following processing is performed on each mapping pair (Ri, Ti) to achieve the joint constraint of maintaining geometry and identity consistency:

[0130] First, based on the target candidate bounding box B_tgt,i, the expanded bounding box B'_tgt,i is obtained by expanding it according to the ROI expansion parameter k_pad, and the intersection with the image boundary is calculated to avoid going out of bounds; second, the target ROI image Patch_tgt,i is cropped and scaled to a uniform size according to a preset resolution (such as 512 or 1024); finally, the reference face is also cropped to obtain Patch_ref,i, and normalized to the same scale is performed.

[0131] For Patch_tgt, i locates the set of facial keypoints K_i (e.g., 68 or 106 points) and can estimate the head pose vector P_i (e.g., yaw / pitch / roll); secondly, the keypoints K_i are converted into a structural condition representation GeoCond_i, such as a keypoint heatmap, skeleton connection diagram or dense pose field, to constrain the generated face to be consistent with the target ROI in terms of position, orientation and expression structure.

[0132] Patch_ref,i is aligned using a similarity transformation (e.g., aligned to the standard coordinate system based on the positions of the two eyes) to reduce the impact of pose differences on identity feature extraction. Secondly, the aligned reference face is input into the identity feature extraction network to obtain the identity vector IdEmb_i (usually a normalized high-dimensional vector), which is used to characterize the identifiable identity attributes of the person. Finally, when the identity reference image consists of multiple images or a single image contains multiple frames of faces, a weighted average or confidence-weighted fusion can be performed on multiple IdEmb images to obtain a stable IdEmb_i.

[0133] The geometric and identity conditions are fused together by weights to form a composite conditional constraint Cond_i. For example, Cond_i can be represented as Cond_i={GeoCond_i, IdEmb_i, w_geo, w_id}. Next, at each sampling step t of the diffusion inference, a dual-channel condition is applied to the noisy prediction network: ε̂_t=F(z_t|TextCond, GeoCond_i, IdEmb_i; w_geo, w_id), where GeoCond_i is used to constrain structural consistency (e.g., to apply residual control to intermediate features of the U-Net through control branches), IdEmb_i is used to constrain identity consistency (e.g., to change the identity-related response across attention through attention adapters / conditional injection), and w_geo and w_id are used to balance between "structural preservation" and "identity transfer".

[0134] When the target ROI and the reference ROI differ significantly in pose or expression, a preliminary identity adaptation (such as coarse alignment face swapping or geometric correction) can be performed first to generate a "reference initial face" that is closer to the target structure, thereby reducing the optimization difficulty in the subsequent diffusion stage and improving stability.

[0135] S170. Based on the conditions, a new facial image is generated using a two-stage sampling method.

[0136] In this embodiment, the new facial image refers to a face image generated after latent spatial coding, noise constraint, diffusion inversion, and local refinement, while preserving the geometric structure and identity features of the target ROI. This image retains the physical characteristics of the original target ROI (such as location, orientation, and expression structure) and incorporates the identity features of the reference ROI, thereby achieving the joint constraint of "geometric structure preservation and identity consistency preservation".

[0137] In one embodiment, step S170 described above may include steps S171 to S173.

[0138] S171. The target base map is converted into a latent representation by an encoder, and a latent mask defining the change region is generated according to the external mask map. Noise is added only within the region of the latent mask according to a specified random seed to obtain noisy latent variables.

[0139] In this embodiment, the noisy latent variable refers to the latent variable z_noisy formed by injecting noise ε only within the mask region after encoding the target ROI image Patch_tgt,i into a latent representation z0 via encoder Enc, combining it with the latent mask M_latent generated by the standard fusion mask M_norm, and under the control of a specified random seed. The noise injection follows the formula z_noisy=z0⊙(1-M_latent)+(z0+σ·ε)⊙M_latent, where ⊙ represents element-wise multiplication, and σ is related to the denoising intensity. The purpose of this is to limit the area of ​​"allowed change," allowing subsequent processing to focus more on the parts that need modification while keeping other parts unchanged.

[0140] S172. Under the given conditions, the noisy latent variables are subjected to diffusion inversion processing to obtain the first feature, and then decoded into a preliminary fused facial image.

[0141] In this embodiment, the initial fusion of facial images refers to performing diffusion inversion sampling on the noisy latent variable z_noisy under the constraint of the composite condition Cond_i, using a specific sampling number n1 and denoising intensity d1, to obtain an intermediate result z1. Then, z1 is decoded by the decoder Dec into the initial fusion result Patch_gen1,i. The main goal of this stage is to complete the identity transfer, ensuring that the generated result is consistent with the target ROI in its overall structure, that is, ensuring that the identity features of the subject are correctly transferred to the target facial structure.

[0142] S173. For the preliminary fused facial image, perform fine-sampling on highly sensitive areas to obtain a new facial image.

[0143] For locally sensitive areas (such as around the eyes, nose, mouth, and hairline) present in the initial fused facial image Patch_gen1,i, targeted refinement is performed in the second stage. This step can be achieved by re-encoding Patch_gen1,i or by continuing sampling directly in the latent space, performing two-stage refinement sampling (steps n2, intensity d2) within the mask area. This process prioritizes repairing areas prone to artifacts, ghosting, and texture collapse, ultimately outputting a new facial image Patch_gen2,i with higher clarity and a more natural appearance. This staged approach not only improves processing efficiency but also enhances the controllability and quality of the results. Furthermore, by fixing the random seed and recording the specific sampling parameters, good reproducibility of the output under the same parameters can be ensured, while also providing the possibility of balancing speed and quality.

[0144] In this embodiment, firstly, the target ROI image Patch_tgt,i is encoded into a latent representation z0 by encoder Enc; secondly, a latent mask M_latent (scaled to the latent scale and binarized / grayscaled) is generated from the standard fusion mask M_norm to define the "allowable variation" region; finally, noise ε is sampled according to the random seed seed, and noise is injected only in the mask region, for example: z_noisy=z0⊙(1-M_latent)+(z0+σ·ε)⊙M_latent, where ⊙ represents element-wise multiplication, and σ is related to the denoising intensity.

[0145] The first stage of sampling (subject identity transfer): Under the constraint of Cond_i, z_noisy is sampled by diffusion inversion using sampling steps n1 and denoising intensity d1 to obtain z1; z1 is then decoded to obtain the preliminary fusion result Patch_gen1,i. This stage focuses on completing the identity transfer to ensure that the generated result is consistent with the target ROI in overall structure.

[0146] The second stage of refinement (local stabilization and sharpness enhancement): Patch_gen1,i is re-encoded or sampling continues directly in the latent space. Second-stage refinement sampling (steps n2, intensity d2) is performed within the mask area. This stage prioritizes repairing locally highly sensitive areas (such as around the eyes, nose, mouth, and hairline boundaries) to reduce artifacts, ghosting, and texture collapse, outputting the refined result Patch_gen2,i.

[0147] By fixing the seed and recording specific sampling parameters, the output can be reproduced under the same parameters; by adjusting n1 / n2 and d1 / d2, a trade-off can be made between speed and quality.

[0148] S180. Perform geometric matching, color unification, and texture enhancement on the new facial image to obtain the processed facial image.

[0149] In this embodiment, the processed facial image refers to the result of geometric matching, color and skin texture consistency, and optional texture enhancement, ensuring that the generated new facial image blends naturally and visually with the target base image in terms of spatial location, size, color distribution, and detail clarity. This result not only solves the identity consistency problem but also significantly improves the consistency and realism of the visual effect.

[0150] In one embodiment, step S180 described above may include steps S181 to S183.

[0151] S181. Adjust the position and size of the new facial image by calculating scaling, translation, or affine transformation parameters.

[0152] This step aims to achieve geometry alignment:

[0153] Determine the transformation parameters: Based on the expanded bounding box B'_tgt,i of the target ROI and the current coordinates of the generated ROI, calculate the required scaling and translation transformation parameters. If more precise alignment is required, an affine transformation matrix can be calculated based on the keypoints K_i to further align the positions of important features such as eyes and mouths.

[0154] Transformation Application: The calculated transformation parameters are used to transform Patch_gen2,i to obtain Patch_align,i. This process ensures that the generated facial region perfectly matches the target ROI in spatial position and size, guaranteeing geometric consistency between the two facial regions.

[0155] S182. Statistically analyze and adjust the color distribution of the new facial image so that the color distribution of the new facial image matches the skin color and lighting conditions of the target base image.

[0156] This step involves color-skin harmonization:

[0157] Statistical color distribution: First, statistically analyze the color distribution within the effective fusion region or transition zone of the target ROI, such as the mean μ_t and variance σ_t per channel; simultaneously, statistically analyze the corresponding statistics μ_s and σ_s within the corresponding region of the new facial image.

[0158] Color transfer is performed using the formula Patch_color=(Patch_align-μ_s)*(σ_t / (σ_s+ε))+μ_t to adjust the color of the new facial image, where ε is a very small value used to avoid division by zero errors. Alternatively, histogram matching or matrix transformation-based methods can be used to achieve more subtle color transfer, ensuring that skin tone and lighting conditions are as consistent as possible between the old and new facial areas, thereby improving visual harmony.

[0159] S183. Select whether to perform super-resolution processing or detail restoration on the adjusted facial image as needed to obtain the processed facial image.

[0160] This step is texture enhancement (optional):

[0161] When enabled: When the texture enhancement function is enabled, super-resolution processing or facial detail restoration operations can be performed on Patch_color,i. The purpose is to improve the clarity of skin texture and the realism of details, making the final output facial image look more natural and realistic.

[0162] When disabled: If no additional texture enhancement is performed, Patch_color,i is directly output as the final result. Regardless of whether texture enhancement is performed, the final output is an optimized write-back region Patch_out,i, which has fully considered geometric matching, color consistency, and possible detail enhancement needs to ensure high integration and visual uniformity with the target base map.

[0163] In summary, these three steps effectively solve the problem of "identity correct but visual inconsistency," ensuring that the synthesized facial image maintains consistency in identity features while seamlessly integrating with the surrounding environment, thus greatly improving the overall visual quality and realism.

[0164] In this embodiment, to ensure natural consistency with the target base map, the following processing is performed:

[0165] First, based on the expanded bounding box B'_tgt,i of the target ROI and the current coordinates of the generated ROI, the scaling and translation transformations are calculated. Optionally, an affine transformation is calculated based on the key point K_i to further align the eye and mouth positions. Second, Patch_gen2,i is transformed to obtain Patch_align,i, so that it matches the target ROI in space.

[0166] Within the "effective fusion region" or transition zone of the target ROI, statistically analyze the color distribution (e.g., mean μ_t and variance σ_t per channel) and calculate the corresponding statistics (μ_s and σ_s) in the generated region. Next, perform color transfer on the generated region, for example, by channel: Patch_color=(Patch_align-μ_s)*(σ_t / (σ_s+ε))+μ_t, where ε is a minimum value to avoid division by zero. Histogram matching or color transfer based on matrix transformation can also be used to obtain more consistent skin tones and lighting.

[0167] When texture enhancement is enabled, super-resolution or facial detail restoration processing is performed on Patch_color,i to improve the clarity of skin texture and details; when disabled, Patch_color,i is output directly, and the final output is the enhanced region to be written back, Patch_out,i. This step is used to solve the problem of "identity is correct but visual inconsistency".

[0168] S190. The processed facial image is fused back to the target base image according to the transparency of the external mask image to obtain a fused image, and the relevant parameters are recorded.

[0169] In this embodiment, image fusion refers to the seamless integration of newly generated facial images back into the original target base image through precise control and optimization processing, ensuring a natural transition and visual consistency between the old and new images. The final output not only includes the harmonious integration of all facial regions but also ensures that the original appearance of the background and other elements remains unaffected. Furthermore, all key parameters used to generate the image are recorded for subsequent quality tracking and result reproduction.

[0170] In one embodiment, step S190 described above may include steps S191 to S193.

[0171] S191. Determine the write-back region by intersecting the extended box with the image boundary, and define the effective fusion region based on the non-zero value part of the standard fusion mask.

[0172] This step aims to precisely define the effective fusion area:

[0173] Determine the write-back region: First, calculate the intersection of the expanded bounding box B'_tgt,i and the image boundary to obtain the actual write-back region R_i. This region determines the specific location where the new facial image will be placed.

[0174] Define the effective fusion region: The "effective fusion region" is defined using the non-zero region of the standard fusion mask M_norm within R_i. Additionally, a threshold τ (e.g., 0.05) can be set as needed to remove edge noise regions with excessively low transparency, thereby improving fusion quality.

[0175] Keep the background unchanged: Outside the effective blending area, keep the original image pixels unchanged to prevent the background or irrelevant parts from being erroneously modified.

[0176] S192. Within the write-back area, the processed facial image and the target base image are weighted and fused at the pixel level according to the transparency of the external mask image to generate an intermediate image. For cases with multiple faces, the process is repeated and the intermediate image is updated. When overlap occurs, the write-back order is adjusted and the external mask image is optimized according to the area or order priority strategy to obtain a fused image.

[0177] This step involves weighted fusion write-back and conflict resolution:

[0178] Weighted fusion: Within the write-back region R_i, the processed facial image Patch_out,i and the target base image I_prev are weighted and fused at the pixel level according to the transparency of the outer mask image. For each pixel p, the formula I_new(p)=M_norm(p)*Patch_out,i(p)+(1-M_norm(p))*I_prev(p) is executed, where I_prev is the intermediate image from the previous round (initially the target base image), and I_new is the intermediate image after write-back.

[0179] Looping through multiple faces: For multiple faces, repeat steps from S160 to S190 for each (Ri,Ti) in the mapping order, and update the intermediate image after each round.

[0180] Handling overlapping areas: When the write-back areas of different faces overlap, the write-back order is determined by either an "area priority" or "sequence priority" strategy. Before write-back, mutual exclusion processing is performed on the mask to reduce artifacts caused by mutual coverage and ensure the best visual effect.

[0181] S193. Record all key parameters.

[0182] This step is to support global output and parameter logging:

[0183] Record information: After completing all fusion operations, output the final fused image. At the same time, record all key parameters in detail, including but not limited to mapping relationships, mask parameters, weight parameters, sampling parameters, and random seed information.

[0184] Supports quality traceability and result reproduction: The information recorded here helps users trace quality, verify every detail of the processing, and also provides a solid foundation for future result reproduction, ensuring that the same high-quality fusion effect can be reproduced under the same conditions.

[0185] In summary, these three meticulous steps not only achieve a perfect fusion between the new facial image and the target base image, but also effectively manage and optimize the fusion process between multiple faces, avoiding potential conflicts and artifacts, and ultimately producing a beautiful and realistic fused image. Furthermore, the detailed parameter recording provides significant convenience and support for subsequent work.

[0186] In this embodiment, the write-back region R_i is obtained by intersecting the extended bounding box B'_tgt,i with the image boundary; secondly, the non-zero region of the standard fusion mask M_norm within R_i is used as the "effective fusion region", and a threshold τ (such as 0.05) can be optionally set to remove edge noise regions with too low transparency; finally, the original image pixels outside the effective fusion region are kept unchanged to prevent the background from being incorrectly modified.

[0187] Within the write-back region R_i, a weighted fusion is performed based on the mask transparency. For example, the following is executed for each pixel p: I_new(p)=M_norm(p)*Patch_out, i(p)+(1-M_norm(p))*I_prev(p), where I_prev is the intermediate image from the previous round (initially the target base image), and I_new is the intermediate image after write-back.

[0188] First, steps S160 to S190 are repeated for each (Ri, Ti) in the mapping order, and the intermediate image is updated after each round. Second, when different face write-back areas overlap, the write-back order can be determined by the "area priority" or "order priority" strategy, and mutual exclusion processing can be performed on the mask before write-back to reduce artifacts caused by mutual coverage.

[0189] After the loop ends, the final fused image is output, and information such as mapping relationship, mask parameters, weight parameters, sampling parameters and random seed are recorded to support quality traceability and result reproduction.

[0190] This embodiment proposes a potential diffusion fusion mechanism of "dual identity constraints," which integrates InstantID structural constraints and PuLID identity feature constraints in the same process. By utilizing two different identity constraint methods, InstantID and PuLID, within a unified framework, they work together to maintain both individual identity characteristics and a natural appearance. This approach overcomes the limitations that may arise from single constraints in traditional methods, such as overemphasizing structure while neglecting texture, or vice versa.

[0191] The method in this embodiment proposes a "multi-person same-order mapping + circular write-back" mechanism: it achieves stable correspondence and continuous fusion of multiple targets through detection sorting and circular replacement.

[0192] This mechanism ensures stability and accuracy in multi-user scenarios through precise detection sorting and cyclic replacement strategies. It establishes reliable correspondences between multiple targets and progressively optimizes the fusion quality of each target through cyclic write-back, ensuring the consistency and integrity of the final result.

[0193] The method in this embodiment proposes an "automatic / manual masking unified fusion" mechanism: it is compatible with automatic segmentation of masks and outer masks in the same process, and performs unified edge refinement and transparency control.

[0194] The unique feature of this mechanism is its ability to process both automatically generated masks and user-provided manual masks simultaneously in a single workflow, and to uniformly refine their edges and adjust their transparency. This not only improves work efficiency but also allows for flexible adjustment of the mask quality and precision according to specific needs.

[0195] This embodiment proposes an integrated "sampling-enhancement-re-fusion" chain: two-stage latent sampling combined with texture enhancement and color adaptation improves the stability and usability of the results.

[0196] This process integrates multiple steps, including two-stage latent sampling, texture enhancement, and color adaptation, forming a complete processing workflow. This approach effectively improves image sharpness, detail, and color consistency, making the entire process more efficient and the results more reliable.

[0197] The method in this embodiment employs an innovative "multi-person sequential mapping + cyclic write-back" mechanism to ensure that each face in a multi-person scenario can be accurately matched and replaced with its corresponding position in the target base map. This mechanism significantly reduces mismatch problems caused by inaccurate face recognition or positional errors.

[0198] This embodiment introduces a unique "dual-identity constraint" latent diffusion fusion mechanism, combining InstantID structural constraints with PuLID identity feature constraints. This method not only effectively preserves individual identity features but also ensures the natural texture of the generated image, making the synthesized result more realistic and believable.

[0199] The innovative "unified fusion of automatic / manual masking" mechanism in this embodiment allows for the simultaneous use of automatic segmentation masking and external manual masking during processing, with unified edge refinement and transparency control. This not only improves the naturalness of the transition between the hairline and facial boundaries but also provides users with greater flexibility and precision.

[0200] The method in this embodiment employs an integrated "sampling-enhancement-re-fusion" chain, achieving end-to-end optimization from latent space to final output. By combining two-stage latent sampling with texture enhancement and color adaptation, the stability, clarity, and detail representation of the results are significantly improved, thereby enhancing the consistency of the overall visual experience.

[0201] The method in this embodiment can be deployed parametrically via the ComfyUI API, meaning it can be easily integrated into existing workflows and supports large-scale batch processing needs. This design greatly improves the applicability of the technology and the ease of operation.

[0202] The aforementioned face fusion method based on identity preservation and potential diffusion acquires and preprocesses a target base map and an identity reference map. It then uses face detection and segmentation techniques to filter and sort valid faces into an ordered reference queue, establishing a mapping relationship between these queues and the faces in the target base map. Next, an external mask map is generated automatically or manually and subjected to alignment and normalization. A composite conditional constraint model is constructed for each mapping pair, imposing structural and identity consistency conditions on the diffusion inference network. A two-stage sampling method is used to generate new facial images. Subsequently, the new facial images undergo geometric matching, color unification, and texture enhancement. Finally, based on the transparency of the external mask map, the optimized facial images are seamlessly fused back into the target base map, achieving efficient, stable, and high-quality face fusion in multi-person scenarios. This method overcomes the shortcomings of existing technologies in terms of mismatch probability, identity preservation, and natural transitions, meeting the requirements for high fidelity and ensuring the authenticity and consistency of the synthesized results.

[0203] Figure 2 This is a schematic block diagram of a face fusion system 300 based on identity preservation and potential diffusion provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above-described face fusion method based on identity preservation and potential diffusion, the present invention also provides a face fusion system 300 based on identity preservation and potential diffusion. This face fusion system 300 includes a unit for executing the above-described face fusion method based on identity preservation and potential diffusion, and the system can be configured in a server. Specifically, please refer to... Figure 2 The face fusion system 300 based on identity preservation and potential diffusion includes an acquisition unit 301, a first processing unit 302, a second processing unit 303, a mask image forming unit 304, an alignment processing unit 305, a mapping extraction unit 306, a dual sampling unit 307, an image processing unit 308, and a fusion unit 309.

[0204] The acquisition unit 301 is used to acquire and preprocess the target base image and the identity reference image, and initialize control parameters; the first processing unit 302 is used to perform face detection and segmentation on the identity reference image, filter and sort the corresponding valid faces to form an ordered reference queue; the second processing unit 303 is used to perform face detection and segmentation on the target base image, filter and sort the corresponding valid faces, and establish a mapping relationship with the ordered reference queue to obtain a mapping pair set; the mask image forming unit 304 is used to acquire an external mask image, or generate an external mask image for the valid faces on the mapping pair set; the alignment processing unit 305 is used to align the external mask image and perform standardization processing to... The processing results are obtained; the mapping extraction unit 306 is used to extract geometric structure and identity features for each mapping pair in the mapping pair set based on the processing results, construct a composite condition constraint model, and apply conditions in the diffusion inference network to achieve consistency of structure and identity; the dual sampling unit 307 is used to generate a new facial image based on the conditions using a two-stage sampling method; the image processing unit 308 is used to perform geometric matching, color unification, and texture enhancement on the new facial image to obtain a processed facial image; the fusion unit 309 is used to fuse the processed facial image back to the target base image according to the transparency of the external mask image to obtain a fused image and record relevant parameters.

[0205] In one embodiment, the first processing unit 302 includes:

[0206] The first detection and segmentation subunit is used to perform face detection and semantic segmentation on the identity reference map to obtain candidate face boxes and corresponding segmented regions; the reference queue construction subunit is used to calculate the effective area of ​​each candidate face box and filter it, retain the candidate face boxes that meet the threshold conditions, and construct an ordered reference identity queue according to a unified sorting rule.

[0207] In one embodiment, the second processing unit 303 includes:

[0208] The target queue construction subunit is used to perform face detection and semantic segmentation on the target base image, obtain candidate face boxes and corresponding segmented regions, calculate the effective area of ​​each candidate face box and filter them, retain the candidate face boxes that meet the threshold conditions, and construct the target identity queue according to a unified sorting rule; the mapping subunit is used to establish a one-to-one mapping relationship between the ordered reference queue and the target identity queue, wherein when the number of the two queues is inconsistent, the smaller number is used as the effective mapping number for processing, and the excess part keeps the original image unchanged or is processed according to a preset strategy.

[0209] In one embodiment, the alignment processing unit 305 is used to perform size alignment, coordinate alignment and transparency normalization on the external mask image, and then perform hole filling, edge refinement, shape expansion, edge smoothing and transparency remapping to obtain the processing result.

[0210] In one embodiment, the mapping extraction unit 306 includes:

[0211] The cropping subunit is used to crop the target base image and identity reference image of each mapping pair in the mapping pair set based on the processing result, and scale them to a uniform size to obtain two processed images. The transformation subunit is used to locate facial key points and estimate head pose in the two processed images, and convert them into geometric conditional representation maps. The extraction subunit is used to align the two geometric conditional representation maps with the reference face through similarity transformation and input them into the feature extraction network to obtain the identity vector. The fusion subunit is used to combine the identity vector with geometric structure and identity features, and fuse them into a composite conditional representation according to weights. The adjustment subunit is used to perform a rough face swap or geometric adjustment first when the difference between the two geometric conditional representation maps does not meet the requirements, to create an initial reference face that is closer to the target structure, and then generate the identity vector.

[0212] In one embodiment, the dual sampling unit 307 includes:

[0213] An addition subunit is used to convert the target base map into a latent representation through an encoder, and generate a latent mask defining the change region based on the external mask map. Noise is added only within the region of the latent mask according to a specified random seed to obtain noisy latent variables. An inversion subunit is used to perform diffusion inversion processing on the noisy latent variables under certain conditions to obtain a first feature, and decode it into a preliminary fused facial image. A refinement sampling subunit is used to refine sampling the preliminary fused facial image for high-sensitivity regions to obtain a new facial image.

[0214] In one embodiment, the image processing unit 308 includes:

[0215] The content adjustment subunit is used to adjust the position and size of the new facial image by calculating scaling, translation, or affine transformation parameters; the statistical adjustment subunit is used to statistically analyze and adjust the color distribution of the new facial image so that the color distribution of the new facial image matches the skin color and lighting conditions of the target background image; the restoration processing subunit is used to select whether to perform super-resolution processing or detail restoration on the adjusted facial image as needed to obtain the processed facial image.

[0216] In one embodiment, the fusion unit 309 includes:

[0217] The write-back region determination subunit is used to determine the write-back region by intersecting the image boundary with the extended box, and to define the effective fusion region based on the non-zero value portion of the standard fusion mask; the image fusion subunit is used to perform pixel-level weighted fusion of the processed facial image and the target base image within the write-back region according to the transparency of the outer mask image to generate an intermediate image. For cases with multiple faces, the process is repeated and the intermediate image is updated. When overlap occurs, the write-back order is adjusted according to an area or order priority strategy and the outer mask image is optimized to obtain the fused image; the recording subunit is used to record all key parameters.

[0218] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned face fusion system 300 based on identity preservation and potential diffusion and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0219] The aforementioned face fusion system 300 based on identity preservation and potential diffusion can be implemented as a computer program, which can be used in, for example... Figure 3 It runs on the computer device shown.

[0220] Please see Figure 3 , Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0221] See Figure 3 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0222] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a face fusion method based on identity preservation and potential diffusion.

[0223] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0224] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a face fusion method based on identity preservation and potential diffusion.

[0225] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0226] The processor 502 is used to run a computer program 5032 stored in the memory to implement all the steps of the face fusion method based on identity preservation and potential diffusion.

[0227] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0228] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0229] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all the steps of the face fusion method based on identity preservation and potential diffusion.

[0230] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0231] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0232] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0233] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0234] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0235] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A face fusion method based on identity preservation and potential diffusion, characterized in that, include: Acquire and preprocess the target base map and identity reference map, and initialize the control parameters; The identity reference image is subjected to face detection and segmentation, and the corresponding valid faces are filtered and sorted to form an ordered reference queue; Face detection and segmentation are performed on the target base map, the corresponding valid faces are filtered and sorted, and a mapping relationship is established with the ordered reference queue to obtain a set of mapping pairs; Obtain an external mask image, or generate an external mask image for valid faces on the set of mapping pairs; Align the external mask image and perform normalization to obtain the processing result; Based on the processing results, geometric structure and identity features are extracted for each mapping pair in the mapping pair set, a composite condition constraint model is constructed, and conditions are applied in the diffusion inference network to achieve consistency between structure and identity. Based on the aforementioned conditions, a new facial image is generated using a two-stage sampling method. The new facial image is subjected to geometric matching, color homogenization, and texture enhancement to obtain a processed facial image; The processed facial image is fused back to the target base image based on the transparency of the outer mask image to obtain a fused image, and relevant parameters are recorded.

2. The face fusion method based on identity preservation and potential diffusion according to claim 1, characterized in that, The step of performing face detection and segmentation on the identity reference image, filtering and sorting the corresponding valid faces to form an ordered reference queue includes: Perform face detection and semantic segmentation on the identity reference map to obtain candidate face frames and corresponding segmented regions; Calculate the effective area of ​​each candidate face frame and filter them, retaining the candidate face frames that meet the threshold conditions, and construct an ordered reference identity queue according to a unified sorting rule.

3. The face fusion method based on identity preservation and potential diffusion according to claim 1, characterized in that, The process of performing face detection and segmentation on the target base map, filtering and sorting the corresponding valid faces, and establishing a mapping relationship with the ordered reference queue to obtain a set of mapping pairs includes: Face detection and semantic segmentation are performed on the target base map to obtain candidate face boxes and corresponding segmentation regions. The effective area of ​​each candidate face box is calculated and filtered. Candidate face boxes that meet the threshold conditions are retained, and a target identity queue is constructed according to a unified sorting rule. Establish a one-to-one mapping relationship between the ordered reference queue and the target identity queue. When the number of the two queues is inconsistent, the smaller number is used as the effective mapping number for processing, and the excess part is kept unchanged or processed according to a preset strategy.

4. The face fusion method based on identity preservation and potential diffusion according to claim 1, characterized in that, The process of aligning the outer mask image and performing normalization to obtain the processing result includes: The external mask image is then subjected to size alignment, coordinate alignment, and transparency normalization. Finally, hole filling, edge refinement, shape expansion, edge smoothing, and transparency remapping are performed to obtain the processing result.

5. The face fusion method based on identity preservation and potential diffusion according to claim 1, characterized in that, Based on the processing results, geometric structure and identity features are extracted for each mapping pair in the mapping pair set, a composite conditional constraint model is constructed, and conditions are applied in the diffusion inference network to achieve consistency between structure and identity, including: Based on the processing results, the target base map and identity reference map of each mapping pair in the mapping pair set are cropped with an expanded box and scaled to a uniform size to obtain two processed images. The facial key points of the two processed images are located and the head pose is estimated, and then converted into a geometric conditional representation map. The two geometric condition representation maps are aligned with the reference face through a similarity transformation and then input into a feature extraction network to obtain an identity vector; The identity vector is combined with its geometric structure and identity features, and then fused into a composite condition according to weights.

6. The face fusion method based on identity preservation and potential diffusion according to claim 5, characterized in that, Based on the processing results, the process of extracting geometric structure and identity features for each mapping pair in the mapping pair set, constructing a composite conditional constraint model, and applying conditions in the diffusion inference network to achieve consistency between structure and identity also includes: If the difference between the two geometric condition representations does not meet the requirements, a rough face swap or geometric adjustment is first performed to create an initial reference face that is closer to the target structure, and then the identity vector is generated.

7. The face fusion method based on identity preservation and potential diffusion according to claim 1, characterized in that, The process of generating a new facial image based on the aforementioned conditions using a two-stage sampling method includes: The target base map is converted into a latent representation by an encoder, and a latent mask defining the variation region is generated based on the external mask map. Noise is added only within the region of the latent mask according to a specified random seed to obtain noisy latent variables. Under the given conditions, the noisy latent variables are subjected to diffusion inversion processing to obtain the first feature, which is then decoded into a preliminary fused facial image. The preliminary fused facial image is then refined by sampling high-sensitivity areas to obtain a new facial image.

8. The face fusion method based on identity preservation and potential diffusion according to claim 1, characterized in that, The process of performing geometric matching, color homogenization, and texture enhancement on the new facial image to obtain a processed facial image includes: The position and size of the new facial image are adjusted by calculating scaling, translation, or affine transformation parameters; The color distribution of the new facial image is statistically analyzed and adjusted to match the skin tone and lighting conditions of the target base image. Choose whether to perform super-resolution processing or detail restoration on the adjusted facial image as needed to obtain the processed facial image.

9. The face fusion method based on identity preservation and potential diffusion according to claim 1, characterized in that, The processed facial image is fused back to the target base image based on the transparency of the external mask to obtain a fused image, and relevant parameters are recorded, including: The write-back region is determined by the intersection of the extended box and the image boundary, and the effective fusion region is defined based on the non-zero value portion of the standard fusion mask; Within the write-back area, the processed facial image and the target base image are weighted and fused at the pixel level according to the transparency of the external mask image to generate an intermediate image. For cases with multiple faces, the process is repeated and the intermediate image is updated. When overlap occurs, the write-back order is adjusted and the external mask image is optimized according to an area or order priority strategy to obtain a fused image. Record all key parameters.

10. A face fusion system based on identity preservation and potential diffusion, characterized in that, include: The acquisition unit is used to acquire and preprocess the target base map and identity reference map, and initialize the control parameters; The first processing unit is used to perform face detection and segmentation on the identity reference image, filter and sort the corresponding valid faces, and form an ordered reference queue. The second processing unit is used to perform face detection and segmentation on the target base map, filter and sort the corresponding valid faces, and establish a mapping relationship with the ordered reference queue to obtain a set of mapping pairs. A mask image forming unit is used to acquire an external mask image, or to generate an external mask image for valid faces on the set of mapping pairs; An alignment processing unit is used to align the external mask image and perform normalization processing to obtain the processing result; The mapping extraction unit is used to extract geometric structure and identity features for each mapping pair in the mapping pair set based on the processing results, construct a composite condition constraint model, and apply conditions in the diffusion inference network to achieve consistency between structure and identity. A dual sampling unit is used to generate a new facial image based on the conditions using a two-stage sampling method; An image processing unit is used to perform geometric matching, color unification, and texture enhancement on the new facial image to obtain a processed facial image. The fusion unit is used to fuse the processed facial image back to the target base image according to the transparency of the external mask image to obtain a fused image, and record relevant parameters.

Images