Method and system for face replacement and skin color coordination of multi-target person image

Abstract

The invention provides a multi-target person image face replacement and skin color coordination method and system, and relates to the technical field of artificial intelligence image processing, and the method comprises the steps: receiving a source face image and a target image sequence containing a plurality of target persons; performing character detection on each target image, and retaining a character detection frame of which the confidence is higher than a threshold value; cutting the figure area based on the detection frame and expanding pixel redundancy to form an expanded cut image including a head and an adjacent skin area; zooming all the expanded and cut images to a uniform resolution ratio and then splicing the expanded and cut images into an optimized long image; and performing semantic segmentation on the optimized long graph, generating a multi-layer mask comprising a core mask layer, an extension mask layer and a coordination mask layer, and performing adaptive edge processing. Through a preprocessing strategy, multi-layer mask control, a style guiding mechanism and a skin color and illumination coordination technology, the quality, efficiency and practicability of a face changing result are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence image processing technology, in particular to a multi-target person image face replacement and skin color coordination method and system. BACKGROUND

[0002] With the in-depth application of artificial intelligence technology in the field of computer vision, face replacement technology is widely used in film and television production, advertising creation and other fields. Its core process includes face detection, feature extraction, feature mapping and image fusion, and the mainstream technical scheme is mainly based on generative adversarial network (such as DeepFakes, FaceSwap), 3D model (such as Face2Face) and single target diffusion model (such as InsightFace, GHOST).

[0003] However, these technologies may have the following deficiencies when processing multi-target person scenes (such as group photos, group video frames): Since the process is designed for a single face, the full process needs to be performed independently for each target when processing multiple targets, which may result in a linear increase in time with the number of targets, causing resource waste; the light, shooting style and other differences of different target original images may cause the replaced faces to have obvious differences in skin color, light and shadow, disrupting the overall harmony; focusing only on replacing the core area of the face may ignore the color and texture continuity with adjacent exposed skin, making it easy to appear as a mask; the highlights and shadows of the face replacement area may not match the original image environment light, causing the background to be disjointed and the realism to decrease. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a multi-target person image face replacement and skin color coordination method and system, which improves the quality, efficiency and practicality of face replacement results through preprocessing strategies, multi-layer mask control, style guiding mechanism and skin color and light coordination technology.

[0005] To solve the above technical problems, the technical scheme of the present application is as follows: In a first aspect, a multi-target person image face replacement and skin color coordination method is provided, which comprises: Step 1, receiving a source face image and a target image sequence containing multiple target persons; performing person detection on each target image, retaining the person detection frame with a confidence higher than a threshold; cropping the person region based on the detection frame and expanding the pixel redundancy to form an expanded cropped image containing the head and adjacent skin area; and scaling all expanded cropped images to a uniform resolution and then splicing them into an optimized long image; Step 2, performing semantic segmentation on the optimized long image to generate a multi-layer mask including a core mask layer, an expanded mask layer and a coordination mask layer, and performing adaptive edge processing; Step 3, extracting a style feature vector from the source face image with the optimized long image as a carrier and the core mask layer or the extended mask layer as a regional constraint; inputting the style feature vector into a diffusion model to perform face redrawing within the regional constraint range, and generating a style-unified face replacement result; Step 4, calculating a pixel-level average skin color value of the redrawing area based on the style-unified face replacement result; performing adaptive skin color adjustment on the exposed skin area in the operation domain of the coordination mask layer to match the chroma value with the average skin color value, obtaining an adjusted intermediate image; performing illumination estimation fusion and detail enhancement processing on the adjusted intermediate image, and obtaining a multi-target coordinated face swapping result.

[0006] The second aspect is a system for multi-target person image face replacement and skin color coordination, comprising: The splicing module is configured to receive a source face image and a target image sequence containing multiple target persons; perform person detection on each target image, and retain the person detection frame with a confidence value higher than a threshold; crop the person area based on the detection frame and expand the pixel redundancy to form an extended cropped image containing the head and the adjacent skin area; and splice all the extended cropped images into an optimized long image after scaling to a unified resolution. The multi-layer mask module is configured to perform semantic segmentation on the optimized long image to generate a multi-layer mask including a core mask layer, an extended mask layer, and a coordination mask layer, and perform adaptive edge processing. The redrawing module is configured to extract a style feature vector from the source face image with the optimized long image as a carrier and the core mask layer or the extended mask layer as a regional constraint; input the style feature vector into a diffusion model to perform face redrawing within the regional constraint range, and generate a style-unified face replacement result. The coordination module is configured to calculate a pixel-level average skin color value of the redrawing area based on the style-unified face replacement result; perform adaptive skin color adjustment on the exposed skin area in the operation domain of the coordination mask layer to match the chroma value with the average skin color value, obtain an adjusted intermediate image; perform illumination estimation fusion and detail enhancement processing on the adjusted intermediate image, and obtain a multi-target coordinated face swapping result.

[0007] The third aspect is a computing device, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0008] The fourth aspect is a computer-readable storage medium, the computer-readable storage medium stores a program, when the program is executed by a processor, the method is implemented.

[0009] The above scheme of the present application at least includes the following beneficial effects: By preprocessing multiple target person images into an optimized long image for batch processing, the number of model calls and repeated calculations is reduced, and the processing efficiency of the multi-target face changing scene is improved; by extracting the style feature vector of the source face and injecting it as a unified guide condition into the redraw process of the diffusion model, the visual style of all target person replaced faces is highly consistent; effectively solving the multi-target face skin color, lighting, texture difference problem caused by independent processing in the prior art, making the face changing result overall harmonious and natural; innovatively introducing a coordination mask layer to accurately position the exposed skin areas such as the neck and shoulders, and through an adaptive skin color adjustment algorithm, the color of these areas is matched with the average skin color of the replaced face, realizing seamless transition of the face and body skin; through multiple layers of masks, the face changing area is finely divided, and combined with adaptive edge processing, the precise control of the face changing area and the natural transition of the edge are ensured; through the lighting estimation model, the environmental lighting parameters are analyzed and shadow and highlight compensation is performed, so that the lighting environment of the face changing area is highly integrated with the original image; combined with the detail enhancement algorithm, the skin texture is restored, further improving the realism of the face changing result. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of a multi-target person image face replacement and skin color coordination method provided by an embodiment of the present application.

[0011] Figure 2 is a system diagram of a multi-target person image face replacement and skin color coordination system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0012] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0013] As Figure 1 shown, an embodiment of the present application proposes a multi-target person image face replacement and skin color coordination method, which includes the following steps: Step 1, receiving a source face image and a target image sequence containing multiple target persons; performing person detection on each target image, retaining the person detection frame with a confidence higher than a threshold; cropping the person region based on the detection frame and expanding the pixel redundancy to form an expanded cropped image containing the head and adjacent skin area; after scaling all expanded cropped images to a unified resolution, splicing them into an optimized long image; Step 2, performing semantic segmentation on the optimized long image to generate a multi-layer mask including a core mask layer, an extended mask layer and a coordination mask layer, and performing adaptive edge processing; Step 3, extracting a style feature vector from the source face image with the optimized long image as a carrier and the core mask layer or the extended mask layer as a region constraint; inputting the style feature vector into a diffusion model to perform face redrawing within the region constraint to generate a style-unified face replacement result; Step 4, calculating a pixel-level average skin color value of the redrawing region based on the style-unified face replacement result; performing adaptive skin color adjustment on the exposed skin region in the operation domain of the coordination mask layer to match the chroma value with the average skin color value to obtain an adjusted intermediate image; performing illumination estimation fusion and detail enhancement processing on the adjusted intermediate image to obtain a multi-target coordinated face swapping result.

[0014] In the embodiments of the present application, by screening high-confidence person detection boxes, the accuracy of the subsequent processing object is ensured, and invalid information interference is reduced; the head and adjacent skin area are extended and retained to provide sufficient context information for subsequent face replacement and skin color coordination, and to avoid harsh edges; the resolution is unified and spliced into an optimized long image to facilitate batch processing and model input, improve processing efficiency and consistency; the multi-layer mask realizes accurate division of different regions to provide clear region guidance for subsequent face redrawing and skin color adjustment; adaptive edge processing can make the mask edge more natural, reduce edge marks in subsequent processing, and improve the visual coherence of the final result; the optimized long image is used as a carrier, a specific mask is used as a region constraint, and the source face style features are combined to ensure that face redrawing is performed within the specified range while ensuring that the style is unified with the source face; the diffusion model is used to realize face replacement, which can generate a more natural and high-quality replacement effect and improve the authenticity of face swapping; the average skin color is calculated based on the replacement result and the exposed skin area is adjusted to keep the skin color consistent overall, avoiding the problem of abrupt skin color; illumination estimation fusion and detail enhancement processing further improve the overall texture and realism of the image, making the multi-target face swapping result more natural and coordinated in terms of illumination and details.

[0015] In a preferred embodiment of the present application, the above step 1, a source face image and a target image sequence containing a plurality of target persons are received; person detection is performed on each target image, and person detection boxes with a confidence higher than a threshold value are retained; the person region is cropped based on the detection box and the pixel redundancy is extended to form an extended cropped image containing the head and adjacent skin area; all the extended cropped images are scaled to a unified resolution and spliced into an optimized long image, including: Step 11, receiving a source face image and a target image sequence containing multiple target persons; calling a pre-trained target detection model to perform person detection on each target image to obtain a person detection box with a confidence value, specifically including: receiving a source face image and a target image sequence containing multiple target persons, wherein the source face image and the target image sequence are in a common image format (such as JPEG, PNG) and have sufficient initial resolution; for each target image, calling a pre-trained target detection model (such as a YOLOv8 model) to perform person detection.

[0016] In this embodiment, the pre-trained target detection model (such as the YOLOv8 model) used contains a backbone network for extracting image features, which processes the input target image step by step through multiple convolution operations, converting the original pixel information into feature maps containing different levels of semantic features. These feature maps not only retain the detailed information of the image (such as the edges and local features of the person), but also capture the global semantic information (such as the overall contour of the person).

[0017] At the same time, the model also includes a feature fusion network (Neck), which is responsible for fusing the different level feature maps output by the backbone network. Through upsampling, downsampling, and jump connection operations, the detailed information of the shallow features is combined with the semantic information of the deep features, generating more expressive fusion features to meet the detection needs of different scale person targets.

[0018] In addition, the model contains a detection head (Head), which performs prediction based on the fused feature maps, obtaining the detection results of the person target through convolution operations, specifically including the detection box (containing the upper left corner coordinates and the lower right corner coordinates) for positioning the person and the confidence value for representing the reliability of the detection box, achieving accurate identification and positioning of the person in the target image.

[0019] Step 12, retaining the detection box with a confidence value higher than a set threshold, specifically including: pre-setting a confidence threshold (for example, 0.8) for filtering valid detection results; for each person detection box with a confidence value obtained in step 11, comparing its confidence value with the pre-set threshold; if the confidence value of a certain detection box is greater than the threshold, retaining the detection box as a valid object for subsequent processing; if the confidence value of a certain detection box is less than or equal to the threshold, excluding the detection box to exclude ambiguous, incomplete, or misdetected person instances.

[0020] Step 13, based on each retained detection frame, taking the left upper corner coordinate and the right lower corner coordinate of the original detection frame as the reference; calculating the detection frame width and height; based on a preset redundancy ratio, expanding the width and height respectively by twice the redundancy ratio value to obtain the expanded width and height; calculating the new left upper corner coordinate and the new right lower corner coordinate according to the expanded width and height to obtain the new coordinate; according to the new coordinate, cutting to obtain an expanded cutting image containing the head and the adjacent skin area, specifically including: for each detection frame retained in step 12, taking its left upper corner coordinate (x1, y1) and right lower corner coordinate (x2, y2) as the reference, calculating the original width and the original height of the detection frame: wherein the original width is the coordinate difference of the left and right boundaries of the detection frame, that is, obtained by subtracting x1 from x2; the original height is the coordinate difference of the upper and lower boundaries of the detection frame, that is, obtained by subtracting y1 from y2.

[0021] A preset pixel redundancy ratio (for example, 10%) is set, and the original width and the original height are expanded based on the ratio: the expanded width is the original width plus the product of the original width and twice the redundancy ratio, and the expanded height is the original height plus the product of the original height and twice the redundancy ratio.

[0022] According to the expanded width and height, the new left upper corner coordinate and the new right lower corner coordinate are calculated: the new left upper corner horizontal coordinate is the original left upper corner horizontal coordinate (x1) minus the product of the original width and the redundancy ratio, and the new left upper corner vertical coordinate is the original left upper corner vertical coordinate (y1) minus the product of the original height and the redundancy ratio; the new right lower corner horizontal coordinate is the original right lower corner horizontal coordinate (x2) plus the product of the original width and the redundancy ratio, and the new right lower corner vertical coordinate is the original right lower corner vertical coordinate (y2) plus the product of the original height and the redundancy ratio.

[0023] In the process of calculating the new coordinate, it is necessary to ensure that the new coordinate does not exceed the boundary of the original target image; if the calculated new coordinate exceeds the boundary of the original image, the coordinates of the exceeding part are adjusted to the boundary coordinates of the original image; finally, the target image is cut according to the above new coordinate to obtain an expanded cutting image containing the head and the adjacent skin area (such as the neck and the shoulder) of the person.

[0024] Step 14, scaling all the expanded cutting images to a unified resolution through a bicubic interpolation algorithm to generate standardized images; based on the size data of the standardized images, calculating their arrangement position coordinates in the splicing canvas; pasting each standardized image to the calculated arrangement position coordinates in a preset order to generate an optimized long image, specifically including: for all the expanded cutting images obtained in step 13, performing scaling processing on them by using a bicubic interpolation algorithm to make all the images reach a preset unified resolution (for example, 1024x1024 pixels) to generate standardized images.

[0025] According to the size data (i.e. the width and height of the unified resolution) of the standardized images, the size of the splicing canvas is planned: if arranged in the preset order from left to right, the height of the canvas is consistent with the height of the standardized images, and the width of the canvas is the product of the width of the standardized images and the number of images.

[0026] The arrangement position coordinates of each standardized image in the splicing canvas are calculated: the top-left corner coordinates of the first standardized image are (0, 0), the top-left corner coordinates of the second standardized image are (the width of the standardized image, 0), and the top-left corner coordinates of the subsequent images are sequentially added to the horizontal coordinates of the top-left corner of the previous image based on the width of the standardized image, and the vertical coordinates are all 0.

[0027] According to the arrangement position coordinates calculated above, each standardized image is pasted into the splicing canvas one by one, and finally an optimized long image containing all the target persons is generated.

[0028] In this embodiment, by calling a pre-trained efficient target detection model, the accuracy and speed of person detection are ensured, providing a reliable initial object for subsequent processing; by using a confidence threshold for screening, low-quality detection results are effectively excluded, reducing the interference of invalid information on subsequent processes and improving processing accuracy; by accurately calculating the expanded cropping range, the core area of the head of the person is retained while the adjacent skin area is included, providing necessary context information for subsequent skin color coordination and avoiding harsh edges; by unifying the resolution and orderly splicing, multiple independent person images are integrated into an optimized long image, realizing the basis of batch processing, reducing the number of subsequent model calls, improving processing efficiency, and ensuring the consistency of the input, which is beneficial to the model to generate results of a unified style.

[0029] In a preferred embodiment of the present application, step 2 is performed on the optimized long image to generate a multi-layer mask including a core mask layer, an expanded mask layer and a coordinated mask layer, and adaptive edge processing is performed, including: Step 21: taking the optimized long image as input, performing pixel-level semantic segmentation by a pre-trained semantic segmentation model to obtain a preliminary segmentation image containing person, face, hair and skin categories, specifically including: taking the optimized long image as input, calling a pre-trained semantic segmentation model (such as Segment Anything Model, SAM) to perform pixel-level semantic segmentation.

[0030] The construction of the pre-trained semantic segmentation model (such as Segment Anything Model, SAM) can be divided into three levels of core component design, feature processing flow and model training optimization, specifically as follows: This model is composed of three core modules: image encoder, prompt encoder and mask decoder, and the functions and structure designs of each module are as follows: The image encoder adopts a Transformer-based Vision Transformer (ViT) architecture as the basic backbone. The input is a standardized image (such as an optimized long image, resized to a preset resolution). Feature extraction is achieved through the following steps: First, the input image is divided into fixed-size non-overlapping image blocks (such as 16x16 pixels / block). Each image block is converted into a feature vector (i.e., "image block embedding") through linear projection, and a position embedding is added to each image block to represent its spatial position information in the original image.

[0031] The above feature vectors are sent to a multi-layer Transformer encoder (including multi-head self-attention mechanism and feedforward neural network) to gradually convert the original pixel information into a feature map containing multi-scale semantics through multi-level global feature interaction and local detail capture. The deep Transformer layers focus on capturing global semantics (such as the overall outline of a person, the context of a scene), and the shallow Transformer layers retain local details (such as facial feature edges, hair texture).

[0032] The final output is a high-dimensional image feature mapping (usually 1 / 32 or 1 / 16 of the original image resolution), which contains both global context and local detail information, providing basic feature support for subsequent segmentation.

[0033] The prompt encoder is used to process external input prompts (such as bounding boxes, points, text descriptions, etc.) and convert them into vector representations that can be fused with image features. The specific design is as follows: For bounding box prompts (such as person detection box coordinates): generate position embedding vectors by encoding the top-left and bottom-right coordinates of the bounding box. Introduce relative position encoding to represent the spatial relationship between the box region and the entire image.

[0034] For point prompts (such as user-labeled foreground / background points): discretize the coordinates of each point to generate point position embeddings. Use binary embeddings to distinguish "foreground points" and "background points" (such as using different vectors to identify them).

[0035] For text prompts (such as "face" and "hair" semantic labels): use a pre-trained text encoder (such as the CLIP text encoder) to convert text into semantic embedding vectors, allowing the model to understand the visual features corresponding to the text description.

[0036] After uniform dimension conversion, the above different types of prompt embeddings are output as prompt feature vectors compatible with the image feature mapping.

[0037] The mask decoder is responsible for fusing the image features output by the image encoder and the prompt features output by the prompt encoder to generate a pixel-level segmentation mask. Its structure is designed as follows: The input is an image feature map (upsampled to the target resolution) and a prompt feature vector. The two types of information are exchanged through a cross-attention mechanism (the image features focus on the regions indicated by the prompt, and the prompt features guide the image features to focus on the target semantics).

[0038] The exchanged features are fed into a multi-layer Transformer decoder and a convolutional layer: the Transformer decoder further models global dependencies, and the convolutional layer refines local edges; finally, an output head (with a sigmoid activation function) generates multiple candidate segmentation masks (usually 3) and simultaneously generates confidence values for each mask.

[0039] The overall model implements semantic segmentation through an end-to-end process of "input → feature extraction → prompt fusion → mask generation", as follows: The input image is processed by the image encoder to generate an image feature map containing global and local features; the external prompt (such as user-specified semantic indicators such as "person" and "face") is converted into a prompt feature vector by the prompt encoder; the mask decoder fuses the image feature map and the prompt feature vector, generates candidate segmentation masks through cross-attention and multi-layer processing; based on the confidence scores of the candidate masks, the optimal mask is selected as the output, achieving pixel-level semantic segmentation of the input image (each pixel is labeled with the corresponding semantic category).

[0040] To achieve "zero-shot generalization" capability (i.e., accurate segmentation of unseen targets or categories), the model is constructed through large-scale data training and optimization strategies, including: using a dataset containing a vast number of images (covering different scenes, lighting, and resolutions) and corresponding mask annotations (a combination of manual annotation and automatic generation) to cover multiple semantic targets such as "person", "face", "hair", and "skin"; using "generate segmentation results consistent with the annotated mask" as the loss function, minimizing the difference between the predicted mask and the true mask (such as using a combination of cross-entropy loss and Dice loss); through data augmentation (such as random cropping, rotation, and color jitter), the model's adaptability to different image deformations is improved; at the same time, diverse prompts (different types and precision of prompt information) are introduced during training to enable the model to learn the general ability to accurately segment targets based on any prompt.

[0041] The model extracts features from the optimized long image using the image encoder to generate a feature map containing global and local features; then, through the collaborative processing of the prompt encoder and the mask decoder, the semantic category of each pixel in the feature map is identified and determined to obtain a preliminary segmentation map.

[0042] In the preliminary segmentation map, each pixel is labeled as a preset semantic category, at least including four categories of "person", "face", "hair", and "skin"; wherein, the "face" category pixel corresponds to the five feature regions of eyes, nose, and lips, the "hair" category pixel corresponds to the head hair region, the "skin" category pixel corresponds to the face, neck and other exposed skin regions, and the "person" category pixel covers all the above-mentioned regions related to the person.

[0043] Step 22, based on the preliminary segmentation map, a multi-layer mask including a core mask layer, an extended mask layer and a coordinated mask layer is obtained; wherein, the core mask layer is to extract the face category pixel region in the preliminary segmentation map, and to perform a morphological erosion operation based on the face key point detection result, to accurately locate to the five feature regions; the extended mask layer is to perform a morphological dilation operation on the core mask layer, to fuse the hair category region of the preliminary segmentation map, to form a mask covering the complete head and hairline; the coordinated mask layer is to extract the skin category pixel region of the preliminary segmentation map, to exclude the clothing covered region by logical NOT operation, to retain the neck and exposed skin region, and specifically includes: extracting all pixels labeled as "face" category from the preliminary segmentation map obtained in step 21 to form an initial face region mask; calling a face key point detection model (such as a key point detector based on a pre-trained model) to locate the five feature key points (such as the corners of the eyes, the tip of the nose, and the endpoints of the lip line) in the initial face region; performing a morphological erosion operation on the initial face region mask based on these key points as a reference - shrinking the mask edge pixel by pixel through a preset size structure element (such as a 3x3 pixel rectangular structure element), to remove the non-core region of the face edge (such as the fuzzy pixels outside the face contour), and finally to obtain the core mask layer accurately located to the five feature regions.

[0044] Based on the core mask layer, a morphological dilation operation is performed on it - the edge of the core mask layer is expanded pixel by pixel through a preset size structure element (such as a 5x5 pixel rectangular structure element), so that it covers the complete face and head contour basic region; the "hair" category pixels are extracted from the preliminary segmentation map to form a hair region mask; the dilated core mask layer and the hair region mask are subjected to logical "or" operation, the two regions are fused to generate the extended mask layer covering the complete head (including five features, face contour, hairline and hair).

[0045] All pixels labeled as the "skin" category are extracted from the initial segmentation map to form an initial skin region mask; the human body range is determined by the overall human segmentation result (e.g., the "human" category pixel region in step 21), and a clothing coverage region mask is generated in combination with clothing detection (e.g., identifying the clothing contour through a preset clothing semantic category or an edge detection algorithm); the initial skin region mask and the clothing coverage region mask are subjected to logical "not" operation to remove the skin pixels blocked by the clothing and retain the pixels of the neck, shoulder, and other exposed skin regions, and finally obtain a coordinated mask layer.

[0046] In step 23, based on the multi-layer mask, a 5-8 pixel wide gradient transition band is generated at the junction of the core mask layer and the extended mask layer, and an 8-15 pixel wide gradient transition band is generated at the junction of the extended mask layer and the coordinated mask layer; based on the gradient transition band, an edge detail protection algorithm is applied to the hairline region to retain the hair detail features while eliminating jagged edges and unnatural boundaries, which specifically includes: for the junction region of the core mask layer and the extended mask layer: determine the edge overlap range of the two through pixel coordinate comparison, and generate a 5-8 pixel wide gradient transition band in the range - for the pixels in the transition band, assign a gradient weight according to the distance from the edge of the core mask layer (e.g., the weight of the pixels close to the core mask layer is 1, and the weight of the pixels close to the outside of the extended mask layer decreases linearly to 0.5), to achieve smooth connection of the two types of masks.

[0047] For the junction region of the extended mask layer and the coordinated mask layer (e.g., the junction of the neck and the head): similarly, determine the edge overlap range of the two to generate an 8-15 pixel wide gradient transition band, and make the edge of the head region and the exposed skin region transition naturally through similar gradient weight distribution.

[0048] For the hairline region contained in the extended mask layer, the edge detail protection algorithm is applied: the hair detail features (e.g., hair edge) of the hairline are identified through an edge detection operator (e.g., Canny operator) to mark the key detail pixels to be retained; the non-key detail pixels in the transition band are subjected to smoothing processing (e.g., Gaussian blur) to eliminate jagged edges and unnatural boundaries; at the same time, the marked key detail pixels are kept as the original pixel values to ensure that the hair, eyelashes, and other fine features are not blurred, and finally obtain a multi-layer mask with natural edges and complete details.

[0049] In this embodiment, pixel-level class labeling is achieved by pre-training a semantic segmentation model, providing a reliable semantic basis for the accurate generation of multi-layer masks, ensuring that subsequent masks can accurately correspond to different regions of the person (face, hair, skin, etc.); multi-layer masks are generated through morphological operations and logical operations, achieving fine division of the core area (facial features), extended area (head), and skin tone coordination area (exposed skin) of face swapping, providing clear operation boundaries for face redrawing and skin tone adjustment; through gradual transition zones and edge detail protection, both the harsh boundaries at the junction of different mask layers are eliminated, and key details such as hair are preserved, improving the naturalness of the mask edges and laying a foundation for the realism of subsequent face replacement and skin tone fusion.

[0050] In a preferred embodiment of the present application, step 3 extracts a style feature vector from the source face image, with an optimized long image as a carrier, a core mask layer or an extended mask layer as a regional constraint; the style feature vector is input into a diffusion model to perform face redrawing within the regional constraint range, generating a style-unified face replacement result, including: Step 31 calls a pre-trained image encoder on the source face image to extract a style feature vector representing lighting, color, and texture features, specifically including: preprocessing the source face image: adjusting the source face image to the input resolution (e.g., 224x224 pixels) required by the pre-trained image encoder (e.g., CLIP image encoder), and normalizing the pixel values of the image (e.g., normalizing the RGB channel values to a pre-set range).

[0051] The pre-trained image encoder is called to perform feature extraction on the preprocessed source face image: the encoder first divides the source face image into fixed-size image blocks, and maps the image blocks through multi-layer convolution and Transformer structure to capture local textures (e.g., pores, wrinkles) and global attributes (e.g., lighting intensity, color bias) of the source face.

[0052] Through global average pooling operation, the high-dimensional feature map output by the encoder is converted into a fixed-dimensional style feature vector (e.g., 768 dimensions), which integrates the lighting, color, texture, and other style information of the source face.

[0053] Step 32, input the optimized long figure into the diffusion model with the style feature vector as the style guide condition, and the optimized long figure as the input carrier, the core mask layer or the extended mask layer as the area constraint; within the range defined by the area constraint, perform denoising processing that balances structure preservation and detail generation; generate a redrawn optimized long figure that is consistent with the style of the source face through iterative denoising, which specifically includes: the diffusion model is fine-tuned based on the Flux.1FillDev architecture, which is centered on realizing image redrawing under conditional constraints through a rectified flow mechanism, and the specific construction can be divided into four levels: network architecture design, input processing module, conditional guidance mechanism, and denoising process optimization, which are as follows: The diffusion model is based on a rectified flow transformer with 12 billion parameters as the backbone, and the overall structure adopts a three-section structure of "encoder-processor-decoder": the image encoder receives the optimized long figure adjusted to the preset size (such as 1024x1024 pixels), and through multi-layer convolution and self-attention mechanism, it performs down-sampling processing on the image, converting the original pixel information into multi-scale feature maps (including low-resolution global features and high-resolution local details); the rectified flow processor, as the core of the model, contains 64 layers of Transformer blocks, each of which models the long-distance dependency between features through cross-attention mechanism, and introduces the rectified flow dynamics equation (used to describe the continuous change process from "complete pollution" to "complete removal" of noise), realizing the gradual correction of noise and fine adjustment of features; the image decoder receives the denoised features output by the processor, and through up-sampling and convolution operations, it restores the feature maps to the input image size, finally generating the redrawn optimized long figure.

[0054] The model design has a special input interface to standardize multiple source inputs and ensure information compatibility and effective fusion: The optimized long figure is scaled to the model's preset input size (such as 1024x1024 pixels), and the pixel values are normalized (such as mapped to the range [-1, 1]) to eliminate the interference of brightness and contrast differences on the model.

[0055] The core mask layer or the extended mask layer is imported, and the mask is converted to a 0-1 matrix (1 represents the face area that needs to be redrawn, and 0 represents the background / non-face area that does not need to be modified) through binary operation; then, the matrix is multiplied with the feature maps of the optimized long figure pixel by pixel to clearly define the model's redrawing operation range and avoid mis-modification of non-target areas.

[0056] The style feature vector (such as 768 dimensions) is received and converted into a vector matching the dimension of the image feature map through linear projection, providing a basic signal for conditional guidance.

[0057] To realize the accurate guidance of style features on the redrawing process, the model designs a special conditional embedding layer and a cross-modal attention mechanism: the style feature vector (containing the source face's lighting, color, and texture information) is converted into a "style guide signal" - through a multi-layer perceptron (MLP) to perform nonlinear mapping on the feature vector, generating a guide tensor with the same dimension as the image feature map, so that the style information can be directly interpreted by the Transformer layer; in each Transformer block of the rectilinear flow processor, a "image feature-style guide" cross-modal attention mechanism is introduced: the image feature is the "query" (Query), the style guide signal is the "key" (Key) and "value" (Value), and through attention weight calculation, the model prioritizes the features that match the source face style (such as the color range of the skin tone and the shadow gradient formed by the lighting) when processing each pixel feature, ensuring that the redrawing process always follows the style standards of the source face.

[0058] To balance the "structure preservation-style transfer" requirements of face redrawing, the model optimizes the denoising process through parameter settings and iterative mechanisms: set the denoising intensity parameter (such as 0.55) to control the amount of noise added to the region constraint range: the higher the parameter value, the more noise added to the region constraint range, the greater the model's freedom to modify the original facial structure; the lower the parameter value, the less noise added, the more complete the preservation of the original structure; the value of 0.55 is verified through experiments, which can ensure the effective transfer of source face style details (such as skin texture and lighting effects) while preserving the original pose and expression of the target person (such as the curvature of the mouth and the direction of the eyes).

[0059] Pre-set a fixed number of denoising iterations (such as 50 times), and each iteration is performed according to the following process: based on the current iteration step and the denoising intensity, add noise conforming to the Gaussian distribution to the image features within the region constraint range; the rectilinear flow processor calculates the noise correction amount based on the style guide signal and the image features through Transformer and convolution operations, gradually reducing the proportion of noise in the features; pass the corrected features to the next layer and repeat the above process until 50 iterations are completed, so that the facial features within the region constraint range gradually converge to the fusion features of "source face style + target person structure".

[0060] Step 33, based on the arrangement position coordinates, locate the region boundaries of each independent person in the optimized long image after redrawing; perform image segmentation according to the region boundaries to extract the face replacement result of each target person; the sequence of face replacement images composed of all face replacement results is the face replacement result with uniform style, which specifically includes: retrieving the arrangement position coordinates (such as the top-left corner coordinates, width, and height) of each standardized image recorded in step 14 in the splicing canvas, and based on the coordinate information, locating the region boundaries of each independent target person in the optimized long image after redrawing (i.e., the rectangular range of each target person in the long image).

[0061] According to the above region boundary, an image segmentation operation is performed on the redrawn optimized long image: the face replacement image sequence is formed by arranging all the cropped single-person face replacement images in the order of the original target image sequence, and the sequence is the face replacement result with the unified face style of all target persons.

[0062] According to the above region boundary, an image segmentation operation is performed on the redrawn optimized long image: the face replacement image sequence is formed by arranging all the cropped single-person face replacement images in the order of the original target image sequence, and the sequence is the face replacement result with the unified face style of all target persons.

[0063] In this embodiment, the style feature vector of the source face is accurately extracted by the pre-trained image encoder, providing a unified benchmark for the style consistency of multi-target face replacement, and avoiding the style deviation problem caused by incomplete style feature extraction in the prior art; with the help of the conditional denoising mechanism of the diffusion model, the style-guided face replacement is realized within the region constraint range, which not only preserves the original pose and expression structure of the target person, but also ensures that all target faces are consistent with the source face style, solving the technical pain point of style splitting in multi-target face replacement; based on the preset coordinates, the single-person face replacement result is accurately segmented, ensuring the independence and integrity of each target image after batch processing, providing a clear processing object for subsequent skin color coordination and other post-processing steps, and improving the standardization and efficiency of the overall process.

[0064] In a preferred embodiment of the present application, step 32 inputs the optimized long image as the input carrier, the core mask layer or the extended mask layer as the region constraint, and the style feature vector as the style guide condition into the diffusion model; within the range defined by the region constraint, denoising processing is performed to balance structure preservation and detail generation; and an optimized long image redrawn with a unified style with the source face is generated through iterative denoising, including: Step 321 inputs the optimized long image as the input carrier into the diffusion model input channel; maps the core mask layer or the extended mask layer as the region constraint into a binary region constraint matrix; and injects the style feature vector as the style guide condition into the model condition guide layer; to obtain an initialized diffusion model state, which specifically includes: preprocessing the optimized long image as the input carrier: adjusting the image size to the standard size required by the diffusion model input channel (such as 1024x1024 pixels), and normalizing the RGB channel pixel values of the image (such as mapping to the range of [-1, 1]), and then loading the preprocessed optimized long image into the image input channel of the diffusion model to complete the input preparation of image data.

[0065] For the core mask layer or the extended mask layer, perform binary mapping: mark the pixels in the mask layer that belong to the face redrawn area (such as facial features, complete head) as numerical value 1, and mark the pixels in the non-redrawn area (such as background, clothes) as numerical value 0, to form a binary region constraint matrix; the matrix is consistent with the size of the optimized long image, and the redrawing operation range of the model is clearly defined through the 0-1 distribution of the matrix elements.

[0066] Process the style feature vector: through the linear conversion module built in the conditional guidance layer of the model, convert the style feature vector (such as 768 dimensions) into a guidance signal (such as a tensor with the same dimension as the image feature map) matching the feature processing dimension of the diffusion model; then inject the guidance signal into the conditional guidance layer, so that the style feature information is recognized and called by the feature processing module of the model.

[0067] Through the above operation, the diffusion model completes the integration of the input image, the region constraint and the style guidance, and enters the initialization ready state.

[0068] Step 322, based on the initialization completed diffusion model state, add controllable noise in the non-zero pixel region defined by the region constraint matrix; perform a preset number of denoising iterations, in each iteration, calculate the cosine similarity between the current noise image state and the style feature vector; dynamically adjust the denoising intensity parameter according to the similarity value; maintain the original image structure of the non-constrained area unchanged, and generate an updated noise image state, which specifically includes: based on the initialization completed diffusion model state, locate the non-zero pixel region (i.e. the face area to be redrawn) with a value of 1 in the binary region constraint matrix, and add controllable noise conforming to a preset distribution (such as Gaussian distribution) to the image pixels in the region: the noise addition amount is set according to the initial denoising intensity parameter (such as 0.55), to ensure that the noise can provide space for the model to generate new features, and also does not destroy the original face structure (such as posture, expression contour) of the target person.

[0069] Set the total number of denoising iterations (such as 50 times), and start the iterative denoising process: in each iteration, first extract the feature vector of the current noise image (obtained through the feature extraction submodule built in the model), and calculate the cosine similarity between the feature vector and the source face style feature vector extracted in step 31-the higher the similarity value, the closer the style of the current noise image to the source face style; the lower the similarity value, the greater the style difference.

[0070] According to the cosine similarity value, the denoising strength parameter of the current iteration is dynamically adjusted: if the similarity value is lower than a preset threshold (such as 0.7), the denoising strength is increased (such as from 0.55 to 0.6), and the model's ability to remove noise and generate new style features is enhanced; if the similarity value is higher than or equal to the preset threshold, the denoising strength is reduced (such as from 0.55 to 0.5), and the modification of existing style features is reduced, and the generated matching style is preferentially retained.

[0071] In the denoising process of each iteration, the feature locking is performed on the pixel region with a value of 0 in the binary region constraint matrix (non-constrained region): the model is prevented from modifying the pixels in this region through a mask shielding mechanism, and the original image structure (such as background and clothing texture) is maintained unchanged; only the non-zero pixel region is subjected to denoising and feature updating, and the updated noise image state is generated and passed to the next iteration.

[0072] Step 323, when the number of iterations reaches a preset threshold, the denoising process is terminated; the noise image state generated by the final iteration is decoded into an optimized long picture after redrawing, which contains facial features consistent with the source face style in the constrained region, specifically including: continuously performing the denoising iteration process of step 322, and monitoring the number of iterations in real time: when the number of iterations reaches a preset total number (such as 50), the denoising process is terminated, and at this time the noise image state has completed the convergence from the "noisy initial state" to the "style-matched stable state".

[0073] The decoder module of the diffusion model is called to decode the noise image state generated by the final iteration: the low-resolution feature map is restored to the original size of the optimized long picture (such as 1024x1024 pixels) through upsampling operation, and the pixel value is de-normalized (mapped back to the RGB pixel range of 0-255), generating an optimized long picture after redrawing.

[0074] In the optimized long picture after redrawing, all target human faces in the non-zero pixel region (face region) defined by the binary region constraint matrix have consistent illumination, color and texture features with the source face; the non-constrained region maintains the original image structure, achieving the coordination of style consistency and structural integrity.

[0075] In this embodiment, by standardizing input processing and integrating multi-source information, it is ensured that the diffusion model accurately receives the optimized long graph, regional constraint and style guide conditions, laying a foundation for the accuracy of the subsequent denoising process and avoiding the deviation caused by incompatible input formats; by adding controllable noise, adjusting the dynamic denoising intensity and locking the non-constrained area, while retaining the original facial structure of the target person, the noise image is forced to converge to the source face style, solving the balance problem of "structure reservation and style unification" and ensuring the consistency and naturalness of multi-target face retouching; by iteration termination and decoding processing, the retouching result meeting the size requirement and style unification is generated, providing high-quality input for the segmentation of the subsequent single-person face changing result, and improving the coherence and reliability of the multi-target face changing process.

[0076] In a preferred embodiment of the present application, step 4, based on the style-unified face replacement result, calculates the pixel-level average skin color value of the retouching area; taking the coordination mask layer as the operation domain, performing adaptive skin color adjustment on the exposed skin area to make its chroma value match the average skin color value, obtaining an adjusted intermediate image; performing illumination estimation fusion and detail enhancement processing on the adjusted intermediate image to obtain a multi-target coordinated face changing result, including: Step 41, based on the style-unified face replacement result, positioning the retouching area according to the face core mask layer or the extended mask layer in the retouching area; converting the sample pixels from the RGB color space to the Lab color space; calculating the arithmetic mean value in the a channel and the b channel respectively as the a component and the b component of the average skin color value, specifically including: based on the style-unified face replacement result, positioning the retouching area through the face core mask layer or the extended mask layer generated in step 2: performing pixel-level matching between the mask layer and the face replacement result, the area in the mask layer with a pixel value of 1 is the retouching area to be processed (such as the core area of the five organs or the complete head area), and the boundary range of the area is determined.

[0077] Select all pixels in the above retouching area as sample objects, and convert the RGB color space values of each pixel to Lab color space: according to the preset color conversion rule (such as the conversion protocol conforming to the CIE standard), the red (R), green (G), and blue (B) channel values of each pixel are mapped to the lightness (L), green-red (a), and blue-yellow (b) channel values of Lab space respectively, obtaining pixel data containing L, a, and b components.

[0078] Statistical sorting of L channel values (lightness) of all converted pixels: arrange the L values from small to large, and eliminate the pixels in the highest 5% quantile (over-bright area) and the lowest 5% quantile (over-dark area) to exclude abnormal pixels such as highlights, shadows or spots that interfere with skin color calculation.

[0079] For the remaining 90% of pixels, the arithmetic mean of the a channel value and the b channel value is calculated respectively: the a component of the average skin color value is obtained by summing the a channel values of all remaining pixels and dividing by the number of pixels; the b component of the average skin color value is obtained by summing the b channel values of all remaining pixels and dividing by the number of pixels, which together constitute the average skin color value representing the typical skin color of the redrawn area.

[0080] Step 42: In the coordinate mask layer, the a component and the b component of the average skin color value are taken as the target reference, and the global color conversion is completed by converting all pixels in the operation domain from the RGB color space to the Lab color space; the difference Δa between the a channel value of each pixel in the operation domain and the target a component is calculated, and the difference Δb between the b channel value of each pixel in the operation domain and the target b component is calculated; the a value of the pixel is updated to the original a value plus Δa multiplied by the mapping coefficient, and the b value of the pixel is updated to the original b value plus Δb multiplied by the mapping coefficient; the L channel value of all pixels is kept unchanged to maintain the illumination, and the adjusted intermediate image is generated, which specifically includes: taking the coordinate mask layer as the operation area, which is the exposed skin area (such as neck, shoulder, etc., excluding the face redrawn area and the clothing covered area) in the mask layer with pixel value of 1, to clearly define the spatial range of the operation domain.

[0081] Convert the RGB color space values of all pixels in the operation domain to the Lab color space: through the consistent color conversion rule as in step 41, the color space conversion of all pixels is completed, and the L, a, b channel values of each pixel are obtained.

[0082] Taking the a component of the average skin color value calculated in step 41 as the target a reference and the b component as the target b reference, the following adjustments are performed for each pixel in the operation domain: calculate the difference Δa between the a channel value of the current pixel and the target a reference (i.e. Δa = target a reference - current a value); calculate the difference Δb between the b channel value of the current pixel and the target b reference (i.e. Δb = target b reference - current b value); A preset mapping coefficient (such as 0.8, which can be adjusted adaptively according to the skin color difference) is used to update the a channel value of the pixel to "original a value + Δa × mapping coefficient", and the b channel value to "original b value + Δb × mapping coefficient", so that the a and b values of the pixels in the operation domain gradually approach the target reference.

[0083] During the adjustment process, the L channel value (brightness) of all pixels is kept unchanged to maintain the brightness level and illumination characteristics of the original skin area; after the adjustment is completed, the Lab values of the pixels in the operation domain are converted back to the RGB color space, and are spliced with the non-operation domain pixels (such as background, clothing) to generate the adjusted intermediate image.

[0084] Step 43, based on the adjusted intermediate image, call the light estimation model to analyze the ambient light parameters of the intermediate image, specifically including: input the adjusted intermediate image into the pre-trained light estimation model (such as a light analysis model based on deep convolutional network), the pre-trained light estimation model is based on deep convolutional network architecture, aiming to realize accurate estimation of ambient light parameters through hierarchical extraction and fusion of image features, its construction process can be divided into three levels of model architecture design, core module function implementation and training optimization, as follows: The model adopts an end-to-end architecture of "input layer-feature extraction network-feature fusion module-output layer", and the whole is based on a deep convolutional neural network as the basic backbone, and each layer is designed as follows: the input layer receives the adjusted intermediate image (such as the image generated in step 42), which is standardized to a preset size (such as 256x256 pixels), and the pixel value is normalized (such as mapping to the range of [0, 1]); the feature extraction network includes shallow convolutional layers and deep convolutional layers, which realize multi-scale feature extraction through multi-level convolution operations, and is the core part of the model to capture light information; the feature fusion module connects the output of the feature extraction network, which is responsible for integrating feature information at different levels to generate fusion features with local details and global features; the output layer is composed of fully connected layers, which map the fusion features to specific ambient light parameters to obtain quantitative results related to light.

[0085] The shallow convolutional layer is composed of 3-5 convolutional layers, which uses a small size convolution kernel (such as 3x3) with a stride of 1 and a ReLU activation function; this part focuses on local light detail extraction of the image: through layer-by-layer convolution of the input image, it captures the intensity and range of skin surface reflection, the clarity and distribution of object edge shadow, and the local area brightness mutation, etc. Granularity features, generate low-dimensional feature maps containing local light features; the deep convolutional layer is located after the shallow convolutional layer, which contains 5-8 convolutional layers and pooling layers (such as maximum pooling), and uses a gradually increasing receptive field (such as through increasing the number of convolution kernels or hierarchical depth); this part focuses on global light feature extraction: through further abstraction and aggregation of shallow features, it captures global features such as average brightness value, color temperature bias (such as the difference between the proportion of red channel and blue channel pixel values), and the distribution ratio of light and dark areas in the picture, etc. Generate high-dimensional feature maps containing global light information.

[0086] The feature fusion module integrates local and global features through skip connection and attention mechanism: first, the local feature maps output by the shallow convolutional layer are adjusted to the same size as the global feature maps output by the deep convolutional layer through upsampling operation (such as bilinear interpolation); then, the two are merged into a fusion feature map through channel concatenation; the spatial attention mechanism is introduced to assign weights to the features in different regions of the fusion feature map (such as the weight of the illumination change region is higher than that of the smooth region), to strengthen the attention to the features of the key region of the illumination estimation, and finally to obtain the optimized fusion features.

[0087] The output layer includes multiple parallel fully connected sub-layers corresponding to different environmental illumination parameters: for "light source direction" (such as azimuth angle), the fusion features are mapped to angle values (such as 0°-360° range) through the fully connected sub-layer; for "illumination intensity" (such as average brightness), the normalized brightness coefficient (such as 0-1 range, corresponding to the gradient from dark to bright) is output through the fully connected sub-layer; for "color temperature type" (such as warm / cold color tone proportion), the ratio of warm color tone pixel proportion to cold color tone pixel proportion is output; for "highlight region distribution", the bounding box coordinates of the high brightness pixel concentration region (such as the top-left and bottom-right coordinates) are obtained.

[0088] In order to enable the model to accurately estimate the illumination parameters of complex scenes, the following training process is used to optimize the parameters: a large image dataset containing different illumination conditions is used, covering indoor / outdoor, daytime / nighttime, front light / back light, etc.; the real illumination parameters (such as light source direction, illumination intensity, color temperature, etc.) of each image are manually labeled or measured by professional equipment to construct "image-illumination parameter" paired samples.

[0089] A multi-task loss function is used to calculate the difference between the predicted value and the true value of each illumination parameter: for continuous value parameters (such as illumination intensity, azimuth angle), mean square error (MSE) loss is used, for ratio type parameters (such as color temperature proportion), cross-entropy loss is used, and the total loss is obtained by weighted summation to ensure balanced estimation accuracy of each parameter.

[0090] Random gradient descent (SGD) or Adam optimizer is used to iteratively optimize the model parameters: a batch of samples is randomly selected from the dataset at each iteration, input into the model to obtain the predicted illumination parameters; the loss between the predicted value and the true value is calculated, and the weights of each layer are updated through back propagation; data augmentation strategies (such as random cropping, brightness adjustment, color jitter) are introduced to improve the generalization ability of the model to different illumination changes; after a predetermined number of rounds (such as 100 rounds) of training, when the loss function value tends to be stable and the validation set accuracy meets the standard, the training is terminated, and the pre-trained illumination estimation model is obtained.

[0091] The model analyzes the ambient light parameters through the following process: the model first performs multi-scale feature extraction on the intermediate image, captures local light details (such as skin surface reflection, object edge shadow) through shallow convolutional layers, and captures global light features (such as overall picture brightness average, color temperature bias) through deep network; then the local and global features are integrated through the feature fusion module to obtain the key parameters representing the ambient light, including but not limited to: light source direction (such as the azimuth angle relative to the center of the picture), light intensity (such as the average brightness value of the picture), color temperature type (such as the proportion of warm / cold color tone), and high light area distribution (such as the concentration position of high brightness pixels in the picture).

[0092] Step 44, based on the light parameters, performing shadow and highlight compensation on the face-changing area to generate a light-optimized image, specifically including: based on the ambient light parameters, locating the face-changing area in the intermediate image (i.e. the union of the face redrawing area in step 3 and the exposed skin adjustment area in step 42), and performing shadow and highlight compensation on the area: The shadow compensation determines the theoretical shadow area in the face-changing area according to the light source direction (such as the area below the nose wings and near the jawline), and if the current area shadow intensity is lower than the shadow reference in the ambient light parameters (such as the average brightness of the shadow area is higher than the preset threshold), the brightness value of the corresponding position pixel is reduced (such as the L channel value is proportionally adjusted down), and the shadow level is enhanced; if the shadow intensity is too high, the brightness value is appropriately increased to avoid detail loss.

[0093] The highlight compensation adjusts the brightness and hue of the highlight area (such as forehead, cheekbone) in the face-changing area according to the light intensity and color temperature parameters: if the highlight brightness is lower than the environmental highlight reference, the L channel value is increased; if the color temperature difference is too large (such as the highlight of the face-changing area is cold and the environmental highlight is warm), the a and b channel values are fine-tuned to make the highlight hue consistent with the environment.

[0094] Step 45, based on the light-optimized image, applying a detail enhancement algorithm to restore skin texture details to obtain a multi-target coordinated face-changing result, specifically including: applying a detail enhancement algorithm (such as a texture restoration algorithm based on a residual network) to the light-optimized image generated in step 44: the algorithm first identifies the micro detail features (such as pores, fine lines, hair texture) of the skin area in the image through a texture detection module, and marks the target area that needs to be enhanced; then the local contrast of the target area is enhanced: by adjusting the brightness difference of adjacent pixels (such as increasing the L value difference between the texture edge and the surrounding area), the texture clarity is enhanced; at the same time, the enhanced area is smoothed (such as weakening the noise in the non-texture area through Gaussian filtering), to avoid artifacts caused by excessive enhancement.

[0095] After processing, a multi-target coordinated face-changing result with clear texture, coordinated light, and uniform skin color is obtained.

[0096] In this embodiment, by excluding extreme brightness pixels and calculating the typical skin color mean value, the interference of highlights and shadows on the skin color reference is avoided, the average skin color value is more representative, and accurate quantitative standards are provided for subsequent skin color adjustment; based on difference mapping and mapping coefficient control, the skin color smooth transition of the exposed skin area and the face redraw area is realized, while the brightness is kept unchanged, the problem of light imbalance in the adjustment process is avoided, and the "masking" is effectively eliminated; the ambient light parameters are extracted through the light estimation model, data support is provided for the light fusion of the face changing area, and the pertinence and accuracy of the light adjustment are ensured; shadow and highlight compensation makes the face changing area integrate into the overall ambient light, solves the problem of "strong synthesis", and improves the realism of the result.

[0097] As shown in Figure 2 The embodiment of the present application also provides a multi-target person image face replacement and skin color coordination system, which comprises: A splicing module is configured to receive a source face image and a target image sequence containing multiple target persons; perform person detection on each target image, and retain the person detection frame with a confidence higher than a threshold; crop the person area based on the detection frame and expand the pixel redundancy to form an expanded cropped image containing the head and adjacent skin area; and splice all the expanded cropped images into an optimized long image after scaling to a uniform resolution. A multi-layer mask module is configured to perform semantic segmentation on the optimized long image to generate a multi-layer mask including a core mask layer, an expanded mask layer and a coordination mask layer, and perform adaptive edge processing. A redraw module is configured to extract a style feature vector from the source face image with the optimized long image as the carrier and the core mask layer or the expanded mask layer as the area constraint; input the style feature vector into a diffusion model to perform face redraw within the area constraint range, and generate a style-unified face replacement result. A coordination module is configured to calculate a pixel-level average skin color value of the redrawn area based on the style-unified face replacement result; perform adaptive skin color adjustment on the exposed skin area in the operation domain of the coordination mask layer to match the chroma value with the average skin color value, and obtain an adjusted intermediate image; and perform light estimation fusion and detail enhancement processing on the adjusted intermediate image to obtain a multi-target coordinated face changing result.

[0098] It should be noted that the system corresponds to the above method, and all implementation manners in the above method embodiment are applicable to this embodiment and can achieve the same technical effects.

[0099] The above is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles described in the present application, can also be made several improvements and refinements, these improvements and refinements should also be considered the scope of protection of the present application.

Claims

1. A method for multi-target person image face replacement and skin color coordination, characterized in that, The method comprises: Step 1, receiving a source face image and a target image sequence containing multiple target persons; performing person detection on each target image, and retaining the person detection frame with a confidence value higher than a threshold; based on the detection frame, cutting the person region and expanding the pixel redundancy to form an expanded cutting image containing the head and the adjacent skin region; after scaling all the expanded cutting images to a unified resolution, splicing them into an optimized long image; Step 2, performing semantic segmentation on the optimized long image to generate a multi-layer mask including a core mask layer, an expanded mask layer and a coordination mask layer, and performing adaptive edge processing; Step 3, extracting a style feature vector from the source face image with the optimized long image as the carrier and the core mask layer or the expanded mask layer as the region constraint; inputting the style feature vector into a diffusion model to perform face redrawing within the region constraint range and generating a style-unified face replacement result; Step 4, based on the style-unified face replacement result, calculating the pixel-level average skin color value of the redrawn region; taking the coordination mask layer as the operation domain, performing adaptive skin color adjustment on the exposed skin region to make its chroma value match the average skin color value, and obtaining an adjusted intermediate image; performing illumination estimation fusion and detail enhancement processing on the adjusted intermediate image to obtain a multi-target coordination face changing result.

2. The method for multi-target character image face replacement and skin color coordination according to claim 1, characterized in that, Step 1, receiving a source face image and a target image sequence containing multiple target persons; performing person detection on each target image, and retaining the person detection frame with a confidence value higher than a threshold; based on the detection frame, cutting the person region and expanding the pixel redundancy to form an expanded cutting image containing the head and the adjacent skin region; After scaling all the expanded cutting images to a unified resolution, splicing them into an optimized long image, including: Receiving a source face image and a target image sequence containing multiple target persons; calling a pre-trained target detection model to perform person detection on each target image to obtain a person detection frame with a confidence value; Retaining the detection frame with a confidence value higher than a set threshold; Based on each retained detection frame, taking the upper left corner coordinates and the lower right corner coordinates of the original detection frame as the reference; calculating the detection frame width and height; based on a preset redundancy ratio, expanding the width and height outward by twice the redundancy ratio value respectively to obtain the expanded width and height; calculating the new upper left corner coordinates and the new lower right corner coordinates according to the expanded width and height to obtain new coordinates; cutting the expanded cutting image containing the head and the adjacent skin region according to the new coordinates; Scaling all the expanded cutting images to a unified resolution by a bicubic interpolation algorithm to generate a standardized image; based on the size data of the standardized image, calculating its arrangement position coordinates in the splicing canvas; pasting each standardized image to the calculated arrangement position coordinates in a preset order to generate an optimized long image.

3. The method for multi-target character image face replacement and skin color coordination of claim 2, characterized in that, Step 2, performing semantic segmentation on the optimized long image to generate a multi-layer mask including a core mask layer, an expanded mask layer and a coordination mask layer, and performing adaptive edge processing, including: Taking the optimized long image as the input, performing pixel-level semantic segmentation by a pre-trained semantic segmentation model to obtain a preliminary segmentation image containing the categories of person, face, hair and skin; Based on the preliminary segmentation map, a multi-layer mask including a core mask layer, an extended mask layer and a coordinated mask layer is obtained; wherein the core mask layer is to extract a facial category pixel region in the preliminary segmentation map, and perform a morphological erosion operation based on a facial key point detection result to accurately locate to a facial feature region; the extended mask layer is to perform a morphological dilation operation on the core mask layer, and fuse a hair category region of the preliminary segmentation map to form a mask covering a complete head and a hairline; and the coordinated mask layer is to extract a skin category pixel region of the preliminary segmentation map, and exclude a clothing covering region through a logical NOT operation to retain a neck and exposed skin region; Based on the multi-layer mask, a 5-8 pixel width gradual transition zone is generated at the junction of the core mask layer and the extended mask layer, and an 8-15 pixel width gradual transition zone is generated at the junction of the extended mask layer and the coordinated mask layer; based on the gradual transition zone, an edge detail protection algorithm is applied to the hairline region to retain hair detail features while eliminating jaggies and unnatural boundaries.

4. The method for multi-target figure image face replacement and skin color coordination according to claim 3, characterized in that, Step 3, taking the optimized long picture as a carrier and the core mask layer or the extended mask layer as a regional constraint, a style feature vector is extracted from the source face image; the style feature vector is input into a diffusion model to perform face redrawing within the regional constraint range to generate a style-unified face replacement result, including: A pre-trained image encoder is called for the source face image to extract a style feature vector representing illumination, tone and texture features; Taking the optimized long picture as an input carrier, the core mask layer or the extended mask layer as a regional constraint, and the style feature vector as a style guide condition, the optimized long picture and the style feature vector are input into the diffusion model; within the range defined by the regional constraint, denoising processing balancing structure preservation and detail generation is performed; through iterative denoising, a redrawing optimized long picture with a unified style with the source face is generated; Based on the arrangement position coordinates, the regional boundaries of each independent person in the redrawing optimized long picture are located; according to the regional boundaries, image segmentation is performed to extract the face replacement result of each target person; a face replacement image sequence composed of all face replacement results is the style-unified face replacement result.

5. The method for multi-target figure image face replacement and skin color coordination according to claim 4, characterized in that, Taking the optimized long picture as an input carrier, the core mask layer or the extended mask layer as a regional constraint, and the style feature vector as a style guide condition, the optimized long picture and the style feature vector are input into the diffusion model; within the range defined by the regional constraint, denoising processing balancing structure preservation and detail generation is performed; through iterative denoising, a redrawing optimized long picture with a unified style with the source face is generated, including: Taking the optimized long picture as an input carrier, it is loaded into the diffusion model input channel; taking the core mask layer or the extended mask layer as a regional constraint, it is mapped into a binary regional constraint matrix; taking the style feature vector as a style guide condition, it is injected into the model condition guide layer; an initialized diffusion model state is obtained; Based on the initialized diffusion model state, controllable noise is added in the non-zero pixel region defined by the region constraint matrix; a preset number of denoising iterations are performed, in each iteration, the cosine similarity between the current noise image state and the style feature vector is calculated; the denoising strength parameter is dynamically adjusted according to the similarity value; the original image structure of the non-constrained region is maintained unchanged, and an updated noise image state is generated; When the number of iterations reaches the preset threshold, the denoising process is terminated; the noise image state generated in the final iteration is decoded into a redrawn and optimized long image, which contains facial features consistent with the source face style in the constrained region.

6. The method for multi-target figure image face replacement and skin color coordination according to claim 5, characterized in that, Step 4, based on the style-unified face replacement result, calculate the average skin color value of the redrawn region at the pixel level; take the coordination mask layer as the operation domain, perform adaptive skin color adjustment on the exposed skin area to make its chroma value match the average skin color value, and obtain an adjusted intermediate image, including: Based on the style-unified face replacement result, the redrawn region is located in the redrawn region according to the face core mask layer or the extended mask layer; the sampling pixels are converted from the RGB color space to the Lab color space; the arithmetic mean values of the remaining pixels in the a channel and the b channel are calculated respectively as the a component and the b component of the average skin color value; Take the coordination mask layer as the operation region, and the a component and the b component of the average skin color value as the target reference, convert all pixels in the operation domain from the RGB color space to the Lab color space to complete the global color conversion; calculate the difference Δa between the a channel value of each pixel in the operation domain and the target a component, and the difference Δb between the b channel value and the target b component, update the a value of the pixel to the original a value plus Δa multiplied by the mapping coefficient, and update the b value of the pixel to the original b value plus Δb multiplied by the mapping coefficient; keep the L channel value of all pixels unchanged to maintain the illumination, and generate an adjusted intermediate image.

7. The method for multi-target figure image face replacement and skin color coordination of claim 6, characterized in that, Perform illumination estimation fusion and detail enhancement processing on the adjusted intermediate image to obtain a multi-target coordinated face swapping result, including: Based on the adjusted intermediate image, call the illumination estimation model to analyze the ambient lighting parameters of the intermediate image; Based on the lighting parameters, perform shadow and highlight compensation on the face swapping region to generate an illumination optimized image; Based on the illumination optimized image, apply a detail enhancement algorithm to restore skin texture details to obtain a multi-target coordinated face swapping result.

8. A multi-target figure image face replacement and skin color coordination system, which implements the method of any one of claims 1 to 7, characterized in that, Including: The splicing module is configured to receive a source face image and a target image sequence containing a plurality of target persons; Perform person detection on each target image, and retain the person detection frame with a confidence value higher than a threshold; crop the person region based on the detection frame and extend the pixel redundancy to form an extended cropped image containing the head and the adjacent skin region; scale all the extended cropped images to a unified resolution and splice them into an optimized long image; The multi-layer mask module is configured to perform semantic segmentation on the optimized long image to generate a multi-layer mask including a core mask layer, an extended mask layer, and a coordination mask layer, and perform adaptive edge processing. The re-drawing module is configured to extract a style feature vector from a source face image with an optimized long image as a carrier, a core mask layer or an extended mask layer as a region constraint; input the style feature vector into a diffusion model, perform face re-drawing in the region constraint range, and generate a style-unified face replacement result; The coordination module is configured to calculate a pixel-level average skin color value of a re-drawing region based on the style-unified face replacement result; perform adaptive skin color adjustment on a bare skin region in an operation domain of a coordination mask layer, so that a chroma value of the bare skin region matches the average skin color value, and obtain an adjusted intermediate image; and perform illumination estimation fusion and detail enhancement processing on the adjusted intermediate image, and obtain a multi-target coordinated face changing result.

9. A computing device, comprising: The method comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by the processor to implement the method according to any one of claims 1 to 7.

Images