A portrait synthesis method, apparatus, medium and electronic device

Abstract

This application provides a method, apparatus, medium, and electronic device for portrait synthesis. The method includes: acquiring an initial synthesized image; performing depth estimation on the initial synthesized image to obtain a depth map; acquiring a shadow generation region mask based on the depth map, the shadow generation region mask being used to limit the pixel region for light and shadow generation during iterative denoising using a diffusion model; encoding the depth map into a structure guidance signal, wherein the structure guidance signal is used to control the geometric shape of the generated light and shadow; inputting the initial synthesized image, the shadow generation region mask, and the structure guidance signal into a generation network for iterative denoising to generate an initial light and shadow synthesized image; such that light and shadow are generated within the region limited by the shadow generation region mask under the constraint of the structure guidance signal, and at least the features of the initial synthesized image are retained in the region outside the region limited by the shadow generation region mask. The embodiments of this application effectively improve the shadow effect of the synthesized portrait image.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, embodiments of this application relate to a method, apparatus, medium, and electronic device for human portrait synthesis. Background Technology

[0002] In the modern digital image production chain, portrait compositing technology has evolved from traditional static background replacement to complex scene semantic reconstruction. This technology not only requires the removal of the human subject from the original environment, but also requires it to be seamlessly and naturally embedded into a new background.

[0003] In practical applications, using green screens for portrait compositing is currently the mainstream solution for achieving high-quality subject removal. By acquiring a portrait under specific chroma key conditions, the system can embed the person into a virtual digital scene. For modern commercial applications, the upper limit of portrait compositing quality is no longer merely about clean cutouts, but rather about achieving deep uniformity between the foreground and background in terms of geometric perspective, ambient light interaction, and texture frequency distribution. Especially in the production of ultra-high-definition commercial posters or film-grade composite videos, how to solve the color bleeding interference caused by green screens and achieve lossless preservation of portrait features has become a pressing technical problem in the field of portrait compositing. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, medium, and electronic device for portrait synthesis. The embodiments of this application determine a shadow generation region mask using a depth map obtained from an initial synthesized image, and control a diffusion model (e.g., including a control module controlnet and a generation network (i.e., the main network of the diffusion model)) based on the shadow generation region mask and the depth map. This allows the diffusion model to redraw the image in shadow areas while retaining at least the features of the original image in non-shadow areas, thereby effectively improving the generation of light and shadow effects on the synthesized image while better preserving the detailed features of the original portrait image. The embodiments of this application can seamlessly blend a portrait taken in front of a green screen (i.e., the original portrait image) into a target background image, and solve problems such as color bleeding, edge blending, and inconsistent lighting during the fusion process.

[0005] In a first aspect, embodiments of this application provide a portrait synthesis method, the portrait synthesis method comprising: acquiring an initial synthesized image, wherein the initial synthesized image is obtained by fusing a foreground image and a target background image of an original portrait image; performing depth estimation on the initial synthesized image to obtain a depth map, wherein the pixel values ​​in the depth map are used to characterize the depth information of each pixel point on the initial synthesized image; acquiring a shadow generation region mask based on the depth map, wherein the shadow generation region mask is used to limit the pixel region where light and shadow are generated during the iterative denoising process of a diffusion model; and encoding the depth map into a structure guidance signal, wherein the structure guidance... The signal is used to control the geometric shape of the generated light and shadow; the initial synthesized image, the shadow generation region mask, and the structure guidance signal are input into the generation network for iterative denoising to generate the initial synthesized light and shadow image; wherein, in the iterative denoising process, according to the shadow generation region mask, the latent variables predicted by the generation network and the latent variables obtained by adding noise to the initial synthesized image are weighted and fused, so that light and shadow are generated within the area defined by the shadow generation region mask with the structure guidance signal as a constraint, and at least some of the original features of the initial synthesized image are retained in the area outside the area defined by the shadow generation region mask.

[0006] The embodiments of this application achieve precise control over the generation of physically realistic shadows on the target background image by inputting both the depth map and the shadow generation mask into the diffusion model: the shadow generation mask limits the shadow generation area at the pixel level, ensuring that the shadow is only projected onto a reasonable plane (such as the ground or wall) in the target background with a depth not lower than that of the portrait; the depth map constrains the shape of the shadow at the geometric level, ensuring that the projection direction, length, and attenuation law of the shadow are strictly consistent with the three-dimensional spatial structure of the target background. The two work together to ensure that the final generated shadow closely fits the ground plane in the target background and fades naturally with distance, fundamentally eliminating the floating shadow effect and harsh edge problems common in synthetic images, achieving depth uniformity between the portrait and the target background in the light and shadow dimension.

[0007] In some embodiments, generating a shadow generation region mask based on the depth map includes: determining a depth constraint mask based on the depth map, wherein the depth constraint mask is used to indicate a spatial region with a depth not lower than the depth level of the human figure; determining an extended shadow mask based on the geometric prior of the human figure, wherein the extended shadow mask is used to indicate a shadow candidate region with natural attenuation characteristics in the scene, and the geometric prior includes the position and / or size of the human figure; solving for the union of the spatial regions defined by the depth constraint mask and the extended shadow mask to obtain a shadow prediction region mask; and obtaining the shadow generation region mask based on the shadow prediction region mask.

[0008] Through the above steps, this application incorporates depth information into shadow region determination, using a depth constraint mask to exclude physically unreasonable areas such as those in the air. Simultaneously, an extended shadow mask simulates the natural attenuation of shadows. By solving the intersection of these two methods, both physical plausibility and geometric possibility are verified, ensuring that the area defined by the final shadow generation region mask conforms to three-dimensional spatial logic (located only on a background plane with a depth not less than that of the human figure) and possesses natural penumbra transition characteristics. This shadow generation region mask accurately defines the shadow generation position and intensity distribution during the diffusion model generation process, thereby eliminating the common problems of shadow floating and harsh edges in synthetic images. It ensures that the generated shadows closely adhere to the background plane, achieving a highly realistic spatial fusion effect.

[0009] In some embodiments, obtaining the shadow generation region mask based on the shadow prediction region mask includes: taking the union of the shadow region defined by the shadow prediction region mask and the portrait dilation region defined by the portrait dilation region mask to obtain an initial shadow generation region, wherein the portrait dilation region mask is obtained by morphological dilation of the portrait mask and is used to cover the semi-transparent area of ​​the portrait edge; subtracting the protection region defined by the protection region mask from the initial shadow generation region to obtain a target shadow generation region, wherein the protection region mask is taken from the portrait mask or obtained by semantic segmentation of the initial synthesized image, and the shadow generation region mask is used to define the target shadow generation region.

[0010] The embodiments of this application first merge the shadow prediction region with the portrait expansion region to ensure that semi-transparent areas at the edges of the portrait (such as hair strands and gauze skirts) are included in the candidate range for shadow generation, avoiding abrupt transitions between shadows and the foreground due to edge protection. Then, the core portrait protection region obtained from portrait masking or semantic segmentation is removed from the merged region to ensure that key features such as the face and hands are not affected by the shadow generation process. This mechanism ensures a natural transition between shadows and foreground edges while reducing or avoiding modifications to the core portrait region, resolving the contradiction in traditional methods where edge protection results in broken shadows or shadow generation damages the portrait, thus achieving a composite result that combines the realism of the shadows with the integrity of the portrait.

[0011] In some embodiments, the portrait synthesis method further includes: performing semantic segmentation on the original portrait image and extracting a first feature mask corresponding to a target category, wherein the target category includes at least one of face, skin, and clothing; performing target detection and instance segmentation on the original portrait image and extracting a second feature mask corresponding to the target category; taking the union of the first feature mask and the second feature mask, and obtaining the protected region mask after dilation and smoothing processing.

[0012] Some embodiments of this application integrate the pixel-level classification capabilities of semantic segmentation with the contour extraction capabilities of instance segmentation to ensure that key feature regions such as faces, skin, and clothing are accurately and completely identified. The semantic segmentation path provides global category attribution, avoiding missed detections; the detection and instance segmentation paths accurately locate the contours of faces and hands, compensating for the shortcomings of semantic segmentation in edge details. After the union of the two is dilated and smoothed, a protective region mask with natural transition boundaries is obtained, providing accurate spatial guidance for subsequent high-frequency feature transfer. This mechanism effectively solves the problem of missing protective regions or harsh boundaries that may be caused by single recognition methods, ensuring that the core features of the portrait are completely preserved during the shadow generation process, so as to achieve the generation of light and shadow without loss of detail.

[0013] In some embodiments, determining the depth constraint mask based on the depth map includes: determining a foreground subject region based on a portrait mask, wherein the foreground subject region is used to characterize the set of pixels in the portrait mask corresponding to the portrait whose confidence level is higher than a preset confidence threshold; extracting depth values ​​corresponding to the foreground subject region from the depth map and calculating the average of all extracted depth values ​​to obtain a depth threshold; and performing binarization processing on the depth map based on the depth threshold to obtain the depth constraint mask.

[0014] Some embodiments of this application utilize valid pixel regions in the portrait mask with confidence levels higher than a preset confidence threshold as depth statistical samples. This eliminates the interference of semi-transparent pixels at the edges of the portrait on the depth mean calculation, ensuring the accuracy of the depth threshold. Based on this threshold, the depth map is binarized to accurately identify all spatial regions with depths not lower than the portrait, i.e., the background plane that physically allows shadow projection. This depth constraint mask serves as one of the dual verification bases for subsequent shadow generation, eliminating unreasonable positions such as those in the air from a three-dimensional spatial perspective. This ensures that the generated shadows always conform to the ground or wall, eliminating the floating shadow effect commonly seen in synthetic images and providing spatial constraints for the subsequent diffusion model to generate physically realistic shadows.

[0015] In some embodiments, the step of binarizing the depth map according to the depth threshold to obtain the depth constraint mask includes: for each pixel in the depth map, determining whether the depth value of the corresponding pixel is greater than or equal to the depth threshold; if the determination result is yes, then marking the corresponding pixel as a first value; if the determination result is no, then marking the corresponding pixel as a second value; after marking all pixels, a binary image is obtained, and the binary image is used as the depth constraint mask.

[0016] The embodiments of this application convert a continuously distributed depth map into a binary depth constraint mask. This mask accurately identifies all physical spatial regions in the scene with a depth no less than that of the human figure. This conversion transforms three-dimensional spatial information into pixel-level control signals, providing a physical constraint for subsequent shadow generation. Only regions marked as 1 by the depth constraint mask D are allowed to cast shadows; regions with a depth less than that of the human figure (such as the air or the foreground) are forcibly excluded. This depth constraint mask serves as the first layer of protection in the dual verification mechanism for shadow generation, preventing logical errors such as generated shadows floating in the air and ensuring that subsequently generated shadows always conform to reasonable planes such as the ground or walls, thus improving the spatial realism of the resulting composite image.

[0017] In some embodiments, determining the extended shadow mask based on the geometric prior of the portrait includes: determining a portrait region based on the portrait mask, wherein the portrait region is used to characterize the overall spatial range of the portrait; generating a preliminary shadow region based on the geometric features of the portrait region and a preset scaling factor; and sequentially performing dilation and blurring processing on the preliminary shadow region to obtain the extended shadow mask.

[0018] Some embodiments of this application utilize geometric priors of the human figure (figure position and size) to quickly estimate the possible projection range of the shadow, generating an initial shadow region. This region is then expanded through dilation to cover the natural diffusion range of the shadow, and further blurred to give the region edges a smooth grayscale distribution, thus obtaining an expanded shadow mask with natural attenuation characteristics. This expanded shadow mask serves as the geometric prior for the dual verification mechanism of shadow generation, providing reasonably positioned candidate regions for subsequent intersection operations with the depth constraint mask. This ensures that the final generated shadow conforms to the geometric laws of light source illumination (extending outwards from the bottom of the figure) and possesses a natural penumbra transition effect, avoiding the problems of harsh shadow edges and distorted shapes in traditional methods.

[0019] In some embodiments, the portrait synthesis method further includes: obtaining a detail restoration region mask, wherein the detail restoration mask is the complement of the shadow generation region mask; performing low-pass filtering on the initial synthesized image to obtain a low-frequency component; extracting high-frequency texture features based on the initial synthesized image and the low-frequency component; and superimposing the high-frequency texture features onto the light and shadow image to be synthesized according to the mask weight of the detail restoration region mask to obtain a target light and shadow synthesized image, wherein the light and shadow image to be synthesized includes the initial light and shadow synthesized image.

[0020] Some embodiments of this application further introduce a frequency domain detail compensation mechanism after the diffusion model generates the light and shadow fusion image: First, the initial synthesized image is low-pass filtered to separate the high-frequency texture features that carry the microscopic details of the original portrait; then, based on the weights of the detail recovery region mask Mprot, these high-frequency features are precisely superimposed back to the corresponding regions of the light and shadow fusion image. The detail recovery region mask, as the complement of the shadow generation region mask, covers all non-shadow regions, including the core region of the portrait and the background preservation region, ensuring that regions not modified during shadow generation can obtain forced restoration of original details in the final output stage. This mechanism achieves environmental lighting uniformity while losslessly preserving the biological feature details of the original portrait (such as skin pores and hair texture), solving the identity feature offset problem caused by the latent space reconstruction characteristics of the diffusion model, and making the synthesis result have both commercial-grade lighting realism and biological feature fidelity.

[0021] In some embodiments, the step of inputting the initial synthesized image, the shadow generation region mask, and the structure guidance signal into a generator network for iterative denoising to generate an initial composite lighting image includes: performing controlled iterative denoising in the latent space. For each denoising step, the generator network is configured to: estimate the current prediction latent variable based on the structure guidance signal; obtain a noisy latent variable matching the current noise level obtained by deterministic denoising of the initial synthesized image; perform weighted fusion of the prediction latent variable and the noisy latent variable based on the shadow generation region mask to obtain the current fused latent variable, which is used as the input for the next denoising step; iterate until denoising is completed, and decode the final fused latent variable to obtain the initial composite lighting image.

[0022] The embodiments of this application employ the aforementioned latent space weighted fusion mechanism. In each denoising step of the diffusion model, the latent variables predicted by the model and the noisy latent variables of the original image are weighted and mixed pixel-by-pixel using a shadow generation region mask. Within regions where the mask value of the shadow generation region mask is 1 (i.e., the region defined by the corresponding mask), the generation features predicted by the diffusion model are used to ensure that shadows are generated within the shadow generation region. Within regions where the mask value of the shadow generation region mask is 0 (i.e., regions not defined by the corresponding mask), the noisy features of the original image are forcibly retained, ensuring that non-shadow areas (including the portrait and background) are unaffected by the generation process. This mechanism leverages the inherent characteristics of image rewriting using the diffusion model to redraw the light and shadow features of the shadow generation region. Simultaneously, by introducing noisy latent variables carrying the original portrait features and matching the current noise level in each fusion step, the consistency of the fusion result distribution is ensured, enabling subsequent denoising steps to execute correctly. The final decoded light and shadow fusion image exhibits physically consistent shadow effects within the shadow generation region and fully preserves the original features of the original image within the non-shadow region.

[0023] In some embodiments, the step of weightedly fusing the predicted latent variable and the noisy latent variable according to the shadow generation region mask to obtain the current fused latent variable includes: using the value of the predicted latent variable within the region defined by the shadow generation region mask; and using the value of the noisy latent variable within the region outside the region defined by the shadow generation region mask.

[0024] In each denoising step of the diffusion model, the image space is divided into a generation region and a protection region based on the shadow generation region mask. Within the generation region, the model-predicted latent variables are fully utilized to ensure free shadow generation. In the protection region, the noisy latent variables of the initial synthesized image are forcibly used to ensure that the original features of the portrait and background regions are not affected by the generation process. This mechanism leverages the inherent characteristic of full-image rewriting in traditional diffusion models to solve the problem of generative models easily modifying non-target regions during synthesis, ultimately resulting in a high-quality synthesized image with realistic shadows and a lossless portrait.

[0025] In some embodiments, the portrait synthesis method further includes: obtaining a detail restoration region mask, wherein the detail restoration region mask is used to identify regions where original portrait features need to be preserved; estimating the current prediction latent variable based on the structure guidance signal includes: predicting the prediction latent variable based on the structure guidance signal and a spatial mask, wherein the spatial mask is constructed based on the shadow generation region mask and the detail restoration region mask, and is used to limit feature interactions between the region defined by the shadow generation region mask and the region defined by the detail restoration region mask during the computation of the generative network.

[0026] In some embodiments of this application, during the computation of the diffusion model, a visibility matrix is ​​constructed based on the shadow generation region mask and the detail recovery region mask, and injected into the attention layer. This restricts the feature interaction between the shadow generation region and the detail recovery region within the model. This mechanism complements the result control of latent space weighted fusion, achieving dual protection of process control and result correction. This solves the problem of identity feature shift caused by feature cross-regional flow during the synthesis process in the diffusion model, ensuring that the final synthesized image obtains realistic shadows while fully preserving the biometric details of the human portrait.

[0027] In some embodiments, obtaining the initial composite image includes: sampling pixels from the green screen region of the original portrait image, calculating the average chromaticity of the sampled pixels as a background reference color; for each pixel in the original portrait image, calculating the color distance between the color vector of the corresponding pixel and the background reference color; mapping the color distance to a preliminary transparency mask according to a start threshold, a stop threshold, and the following principles: if the color distance is less than or equal to the start threshold, the corresponding pixel is marked as completely transparent; if the color distance is greater than or equal to the stop threshold, it is marked as completely opaque; if the color distance is less than or equal to the stop threshold, it is marked as completely opaque. Between the starting threshold and the ending threshold, the image is marked as semi-transparent using linear interpolation; wherein the starting threshold and the ending threshold are determined based on the mean and variance of the chromaticity distribution on the original portrait image, and the starting threshold is less than the ending threshold; the preliminary transparency mask is smoothed to obtain a green screen desaturation mask; foreground content is extracted from the original portrait image based on the portrait mask, and the foreground content is merged with the green screen desaturation mask to generate a foreground layer with a transparency channel; the foreground layer is fused with the target background image to obtain the initial composite image.

[0028] Some embodiments of this application upgrade the processing of green screen overflow areas from simple color replacement to the generation of semi-transparent materials: First, a high-quality green screen desaturation mask α_smooth is generated using dual threshold mapping and Gaussian smoothing. This mask preserves the subject in completely opaque areas, removes the background in completely transparent areas, and achieves a natural transition in semi-transparent areas. Then, the green screen desaturation mask α_smooth is merged with the original portrait image as an alpha channel to generate a foreground layer with accurate transparency information. This foreground layer serves as the input base for subsequent processing, effectively eliminating green screen overflow while fully preserving the original details of semi-transparent objects such as hair and gauze skirts. It provides high-quality materials with complete information for depth estimation, adaptive mask generation, and diffusion model shadow generation. The residual slight green will be automatically fine-tuned and eliminated by the diffusion model during shadow generation, forming a systematic collaborative processing.

[0029] In some embodiments, when a target type image needs to be output, the portrait synthesis method further includes: dividing the initial light and shadow synthesis image into multiple local blocks with overlapping regions; performing detail enhancement and resolution improvement processing on each local block respectively, and then seamlessly stitching the processed local blocks to obtain a high-definition light and shadow synthesis image; and superimposing the high-frequency texture features onto the corresponding region of the high-definition light and shadow synthesis image according to the weight of the detail recovery region mask to obtain the target light and shadow synthesis image.

[0030] This application addresses the need for ultra-high-definition output by segmenting the light and shadow fusion image (i.e., the initial light and shadow composite image) into local blocks adapted to GPU memory for independent processing. This overcomes the memory limitations of single inference, enabling smooth processing of 4K / 8K images on conventional computing platforms. After enhancement, each block is seamlessly stitched together using linear weighted blending, eliminating block boundary artifacts and ensuring global visual continuity. Furthermore, based on detail recovery region masks, high-frequency texture features are compensated to corresponding areas of the high-definition light and shadow composite image, ensuring that while resolution is improved, microscopic details such as skin pores and hair strands in portraits are restored without loss. This mechanism achieves ultra-high-definition output while balancing computational efficiency and image quality fidelity, giving this solution strong engineering feasibility and meeting the dual requirements of high-resolution, high-fidelity compositing in commercial pipelines.

[0031] Secondly, some embodiments of this application provide a portrait synthesis apparatus, the portrait synthesis apparatus comprising: an initial synthesized image tiling processing module configured to acquire an initial synthesized image, wherein the initial synthesized image is obtained by fusing the foreground and target background images of an original portrait image; a depth map acquisition module configured to perform depth estimation on the initial synthesized image to obtain a depth map, wherein the pixel values ​​in the depth map are used to characterize the depth information of each pixel point on the initial synthesized image; a shadow generation region mask acquisition module configured to acquire a shadow generation region mask based on the depth map, wherein the shadow generation region mask is used to limit the pixel region where light and shadow are generated during the diffusion model iterative denoising process; and a spatial guidance feature acquisition module configured to... The depth map is encoded as a structure guidance signal, which is used to control the geometric shape of the generated light and shadow. The light and shadow fusion image acquisition module is configured to input the initial synthesized image, the shadow generation region mask, and the spatial guidance features into the generation network for iterative denoising to generate an initial synthesized light and shadow image. In the iterative denoising process, according to the shadow generation region mask, the latent variables predicted by the generation network and the latent variables obtained by adding noise to the initial synthesized image are weighted and fused, so that light and shadow are generated within the area defined by the shadow generation region mask with the structure guidance signal as a constraint, and at least some of the original features of the initial synthesized image are retained in the area outside the area defined by the shadow generation region mask.

[0032] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the human image synthesis method as described in any of the embodiments included in the first aspect above.

[0033] Fourthly, some embodiments of this application provide an electronic device including a memory and a processor, wherein the memory is used to store a computer program, and the processor can read the computer program from the memory and execute it to implement the human image synthesis method as described in any of the embodiments included in the first aspect above.

[0034] Fifthly, some embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the human image synthesis method as described in any of the embodiments included in the first aspect above. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is an architecture diagram of the human portrait synthesis system provided in the embodiments of this application.

[0037] Figure 2 This is one of the flowcharts for the portrait synthesis method in this application embodiment.

[0038] Figure 3 The second flowchart of the portrait synthesis method provided in the embodiments of this application.

[0039] Figure 4 A block diagram illustrating the composition of the human portrait synthesis apparatus provided in the embodiments of this application.

[0040] Figure 5 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0042] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0043] The inventors of this application discovered the following limitations in the related portrait synthesis methods during their research: (i) Traditional synthesis methods based on chroma key mapping and linear interpolation.

[0044] Such methods include Chroma Keying, Alpha Matting, Poisson Image Editing, and Global Color Transfer. These methods are highly dependent on the lighting quality of the shooting environment, struggle to handle color spilling caused by green screens, and lack awareness of three-dimensional spatial geometry, resulting in composite results that often lack realistic shadow interactions.

[0045] (II) Feature Alignment Synthesis Method Based on Convolutional Neural Network (CNN)

[0046] CNN methods incur significant memory overhead when processing ultra-high-definition (4K) images. Furthermore, the shadows they generate often exhibit flattened characteristics, lacking realistic physical depth logic, making it difficult to meet the texture requirements of high-end commercial photography.

[0047] (III) Generative Synthesis Methods Based on Diffusion Models

[0048] Diffusion models generate an image by performing multiple denoising iterations in the image's coding space and then decoding the output using a decoder. These methods reconstruct the synthesized region by performing denoising inference in the latent space. While diffusion models address the issue of consistent lighting, they are essentially pixel reconstructions rather than pixel preservation, making them highly susceptible to altering the original biological characteristics of the portrait (e.g., excessive skin smoothing, subtle changes to facial features), leading to distortion in the synthesized result under commercially viable requirements.

[0049] In other words, existing portrait compositing solutions generally suffer from problems such as edge color bleeding, lack of spatial geometric logic, generative feature shift, and inadequate detail protection when dealing with high-resolution commercial images. To meet the creative needs of professional commercial photography, high-end e-commerce visual displays, virtual studio live streaming, and high-precision AIGC, there is an urgent need to develop a novel portrait compositing technology. The portrait compositing method provided in this application can achieve: spatial logic stability at extremely high resolution; multi-dimensional differentiated control and green screen depth optimization; and a refined guiding path based on high- and low-frequency decoupling. The embodiments of this application can effectively avoid AI molding and feature drift while performing generative background enhancement and light and shadow transfer. This solution must ensure that the realistic biological texture of the skin, high-frequency edges of the hair, and the geometric structure of complex props are preserved at the pixel level. While maintaining high-precision compositing quality, it significantly reduces the memory overhead during ultra-large-scale image processing through block reconstruction and adaptive mask calculation optimization.

[0050] The embodiments of this application specifically address the aforementioned industry technical challenges by proposing a portrait compositing system and its implementation method based on an adaptive masking and diffusion model. By introducing modules such as RGBA channel encapsulation, depth field spatial constraints, adaptive mask generation, and high- and low-frequency feature transfer, the system systematically solves the key problems of uncontrollable portrait compositing effects, severe edge residue, and easy loss of details in high-resolution scenes, providing highly reliable and efficient technical support for commercial portrait post-processing.

[0051] Notes to relevant terms used in the embodiments of this application:

[0052] The original portrait image, a three-channel RGB image, contains the portrait and the original background, serving as the source for extracting the foreground to be composited.

[0053] The target background image, as a pre-determined background to be synthesized, retains its semantic content unchanged during the processing of this application. This application limits the generation area of ​​the diffusion model by using a shadow generation mask, so that shadows are generated only in local areas of the target background that are in contact with the human figure. At the same time, it ensures that the remaining areas of the background retain their original features during the iterative denoising process through latent space weighted fusion, thereby achieving accurate shadow synthesis without changing the background content.

[0054] A portrait mask identifies the pixel region belonging to the main subject of a person in the original portrait image. It can be a binary or grayscale mask and can be used to extract portrait content from the original image. If the portrait mask is grayscale, its pixel value represents the confidence level that the pixel belongs to the person, ranging from 0 to 1 (or 0 to 255). A higher pixel value indicates a higher confidence level that the pixel belongs to the person, while a lower pixel value indicates a lower confidence level that it belongs to the background. In the embodiments of this application, when calculating the depth threshold, only pixels with a confidence level higher than a preset confidence threshold are used to exclude interference from semi-transparent pixels at the edges on depth statistics.

[0055] A portrait is an image containing only the subject of a person, extracted from the original portrait image. It is in RGB or RGBA format and is obtained by extracting (multiplying pixel by pixel) the portrait from the original image using a portrait mask. For example, the portrait can be obtained by multiplying the portrait mask with the original portrait image and then blackening the background.

[0056] A green screen desaturation mask, used to identify the transparency weight of the green screen area, is merged with the foreground content as an alpha channel to generate a foreground layer with a semi-transparent effect. Embodiments of this application generate this grayscale mask using chroma keying, double thresholding, and Gaussian smoothing.

[0057] The foreground layer, a four-channel RGBA image obtained by merging the foreground content (RGB) with the green screen desaturation mask (Alpha), is used to fuse with the target background image to obtain the initial composite image.

[0058] The initial synthesized image, a three-channel RGB image obtained by fusing the foreground layer and the target background image, is used as input data for subsequent depth estimation and adaptive mask generation.

[0059] A depth map is a single-channel grayscale image obtained by depth estimation of the initial synthesized image. The pixel value represents the relative distance between the point and the camera, providing three-dimensional spatial constraints for shadow generation.

[0060] Depth constraint mask, a mask obtained by binarizing the depth map based on a depth threshold, identifies physical spatial regions with a depth not lower than the depth level of the human figure, serving as the first layer of verification (physical rationality) for shadow generation.

[0061] Extended shadow mask, a grayscale mask generated based on the geometric features of the human face (minimum bounding rectangle) and obtained through dilation and blurring, simulates the natural decay pattern of shadows and serves as a second verification (geometric probability) for shadow generation.

[0062] The shadow prediction region is the intersection of the depth constraint mask and the extended shadow mask, identifying shadow candidate regions that simultaneously satisfy physical plausibility and geometric possibility.

[0063] The portrait expansion area is a mask obtained by morphologically expanding the portrait mask. It is used to cover the semi-transparent area at the edge of the portrait to ensure that the shadows blend naturally with the foreground.

[0064] Protected area mask is used to preserve the core area of ​​the human face, which is usually taken from the human face mask or key areas such as the face and hands obtained through semantic segmentation. These areas are excluded in the shadow mask generation to ensure that the core of the human face is not affected.

[0065] The shadow generation mask, which is ultimately used to limit the shadow generation region, serves as a weight in the latent space fusion of the diffusion model to control the generation region.

[0066] The detail recovery region mask, the complement of the shadow generation mask, is used to identify all non-shaded regions and to provide region guidance during high-frequency feature transfer.

[0067] The structure guidance signal, which encodes the depth map to obtain a multi-scale feature map, is used to control the geometric shape of the generated light and shadow. It is injected into the generation network to ensure that the generated shadow conforms to the three-dimensional spatial structure.

[0068] The latent space mask is a mask that is downsampled from the shadow generation mask to the same size as the latent space mask, and is used as a weight in the latent space weighted fusion.

[0069] Predict latent variables, generate latent variables predicted by the network in the current denoising step, and apply them in the shadow generation region.

[0070] Noise-adding latent variables are latent variables obtained by deterministic noise addition to the initial synthesized image and matched with the noise level of the current step. They are used in non-shaded areas.

[0071] The image is a blended image, and the final latent variables are decoded by VAE to obtain an RGB image, which is the result of core shadow generation.

[0072] A binary mask has pixel values ​​that take only two values: 0 or 1. A pixel value of 1 indicates that the pixel belongs to the target area defined by the mask, such as a human figure; a pixel value of 0 indicates that the pixel does not belong to the target area.

[0073] A grayscale mask has pixel values ​​continuously distributed between 0 and 1. A pixel value of 1 indicates that the pixel belongs entirely to the target area identified by the mask, or has the highest weight; a pixel value of 0 indicates that the pixel does not belong to the target area at all, or has a weight of zero; a pixel value between 0 and 1 indicates that the pixel partially belongs to the target area, or has an intermediate weight between the lowest and highest weight, and is usually used to characterize features that require smooth transitions, such as edge transition areas, semi-transparent areas, or intensity decay.

[0074] The diffusion model includes a generator network and a control module. The generator network is the denoising network of the diffusion model, which can adopt a U-Net architecture (such as Stable Diffusion XL) or a Transformer architecture (such as DiT, PixArt-α) for iterative denoising in the latent space. The control module is used to encode external conditions (such as depth maps) as guiding features and inject them into the generator network, and can adopt ControlNet or its alternatives (such as IP-Adapter, training free editing methods, etc.). It should be noted that in the diffusion model of this application embodiment, the attention layer of its generator network is supplemented with a low-rank adaptation (LoRA) weight module obtained through fine-tuning. This LoRA module is connected in parallel with the original weight matrix in a bypass manner. During the fine-tuning process, the original weights remain frozen, and only the low-rank matrix parameters of the LoRA module are updated. This LoRA module is obtained by fine-tuning through training data containing real shadows and is used to guide the generator network to output shadow effects that conform to physical laws during the iterative denoising process.

[0075] For example, in this embodiment, the shadow generation mask is a binary mask, and its pixel values ​​are used as weights in the latent space fusion of the diffusion model. The region with a pixel value of 1 is the shadow core generation region (i.e., the region defined by the shadow generation region mask), which fully utilizes the generation features predicted by the diffusion model (i.e., within the region defined by the shadow generation region mask, light and shadow are generated under the constraint of the structure-guided signal, without fusing the original features); the region with a pixel value of 0 is the non-shadow region (the region outside the region defined by the shadow generation region mask), which fully retains the original features of the original image (i.e., the region outside the region defined by the shadow generation region mask retains the complete original features of the initial synthesized image, without superimposing any features generated by the generative network).

[0076] For example, in this embodiment, the shadow generation mask is a grayscale mask, and its pixel values ​​are used as weights in the latent space fusion of the diffusion model. Specifically, the region with a pixel value of 1 is the shadow core generation region (i.e., the region defined by the shadow generation region mask), which fully utilizes the generation features predicted by the diffusion model (i.e., within the region defined by the shadow generation region mask, light and shadow are generated under the constraint of the structure-guided signal, without fusing the original features); the region with a pixel value of 0 (as the first region outside the region defined by the shadow generation region mask) is a non-shadow region, which fully retains the original features of the original image (i.e., in the first region, the complete original features of the initial synthesized image are retained, without superimposing any features generated by the generative network); the region with pixel values ​​between 0 and 1 (as the second region outside the region defined by the shadow generation region mask) is a shadow edge transition region, achieving a smooth blending of generated features and original features (i.e., in the second region, some of the original features of the initial synthesized image are retained; for example, the pixel values ​​of pixels in this second region are obtained by weighted fusion of the original features and the features generated by the generative network, and the weighting coefficient is the weight value in the shadow generation mask).

[0077] Please refer to Figure 1 , Figure 1 This application provides an architecture diagram of a portrait synthesis system according to some embodiments. The system includes: a preliminary synthesized image acquisition module 110, a depth map estimation module 120, and an adaptive mask extraction module (including at least...). Figure 1 The system includes a shadow generation region mask acquisition module 151 and a detail restoration region mask acquisition module 152, a high-frequency feature extraction module 141, a diffusion model 130, and a protected region feature replacement module 140.

[0078] The initial composite image acquisition module 110 is configured to generate an initial composite image based on the original portrait image, the target background image, and the portrait mask.

[0079] In some embodiments of this application, the initial synthesized image acquisition module 110 includes:

[0080] The data alignment unit is configured to verify the spatial resolution of the original portrait image and the portrait mask to ensure that the pixel coordinates are consistent.

[0081] The channel merging unit is configured to merge the portrait mask as the alpha channel with the RGB channels of the original portrait image to construct a four-channel image.

[0082] The green screen processing unit is configured to identify the green screen area through chroma key analysis, generate a semi-transparent mask (i.e., a green screen desaturation mask), and optimize the foreground edges to obtain the processed foreground layer.

[0083] The initial fusion unit is configured to linearly blend the processed foreground layer with the target background image 102 to obtain an initial composite image 104.

[0084] The depth map estimation module 120 is configured to recover three-dimensional spatial geometric information from the initial synthetic image and generate a depth map for physical constraints of subsequent shadow generation.

[0085] For example, in some embodiments of this application, the depth estimation unit of the depth estimation module 120 employs a monocular depth estimation model (such as DepthAnythingV2) to infer from the initial synthetic image and generate a dense depth map. In some embodiments of this application, the depth estimation unit further includes a depth normalization unit, which is used to perform numerical normalization and scale recalibration on the original depth values ​​to generate a high-precision depth map with a consistent scale.

[0086] The shadow generation region mask acquisition module 151 is configured to generate a mask for defining the pixel region for light and shadow generation based on the depth map and the portrait mask.

[0087] In some embodiments of this application, the shadow generation region mask module 151 includes: a depth constraint extraction unit, which is configured to determine a threshold based on the depth mean of the portrait mask region and generate a depth constraint mask to identify the background region where shadows are physically allowed to be cast. In some embodiments of this application, the shadow generation region mask module includes: a depth constraint extraction unit, a geometric projection calculation unit, and a logic fusion unit. The depth constraint extraction unit is configured to determine a threshold based on the depth mean of the portrait mask region and generate a depth constraint mask to identify the background region where shadows are physically allowed to be cast. The geometric projection calculation unit is configured to: generate a geometric projection region based on the minimum bounding rectangle of the portrait mask according to a preset scaling factor, and obtain an extended shadow mask through dilation and blurring processing. The logic fusion unit is configured to take the intersection of the extended shadow mask and the depth constraint mask as a shadow prediction mask to characterize the shadow prediction region. Subsequently, a shadow generation region mask is obtained based on the shadow prediction mask. For example, in some embodiments of this application, a protection region mask also needs to be obtained, and then the protection region mask is subtracted from the shadow prediction mask to obtain a shadow sound mask. In some other embodiments of this application, it is also necessary to obtain a protection region mask and a portrait dilation region mask. The shadow region defined by the shadow prediction region mask and the portrait dilation region mask are combined to obtain an initial shadow generation region. The portrait dilation region mask is obtained by morphological dilation of the portrait mask and is used to cover the semi-transparent area of ​​the portrait edge. The protection region defined by the protection region mask is subtracted from the initial shadow generation region to obtain the target shadow generation region. The protection region mask is taken from the portrait mask or obtained by semantic segmentation of the initial synthesized image. The shadow generation region mask is used to define the target shadow generation region.

[0088] For example, in some embodiments of this application, the protected region mask acquisition module (not shown in the figure) includes: a semantic segmentation unit, an instance segmentation unit, and a fusion post-processing unit. The semantic segmentation unit is configured to extract the segmentation mask corresponding to the face category using a semantic segmentation model (such as SegFormer); the instance segmentation unit is configured to locate the face and hand detection boxes using an object detection model (such as YOLO), and then use an instance segmentation model (such as SAM) to perform fine segmentation on the detection box region to obtain the face and hand instance mask; the fusion post-processing unit is configured to take the union of the semantic segmentation mask and the instance segmentation mask, and then perform dynamic dilation and Gaussian smoothing to obtain the protected region mask.

[0089] The detail restoration region mask acquisition module 152 is configured to generate a detail restoration region mask for identifying key feature regions such as faces and hands based on the original portrait image and depth map. For example, in some embodiments of this application, the detail restoration region mask is the complement of the shadow generation region mask.

[0090] The high-frequency feature extraction module 141 is configured to extract high-frequency texture features (such as microscopic details like skin pores and hair strands) from the initial synthesized image to provide material for subsequent detail compensation.

[0091] In some embodiments of this application, the high-frequency feature extraction module 141 includes a low-pass filtering unit, a high-frequency extraction unit, and a feature storage unit. The low-pass filtering unit is configured to apply Gaussian blur to the initial synthesized image to extract low-frequency components. The high-frequency extraction unit is configured to subtract the low-frequency components from the initial synthesized image to obtain a high-frequency texture feature map 108. The feature storage unit is configured to save the extracted high-frequency texture features for use by the protected area feature replacement module.

[0092] The diffusion model 130 is configured to generate light and shadow within the area defined by the shadow generation region mask under the guidance of the depth map, and output an initial light and shadow composite image 109.

[0093] In some embodiments of this application, a control module (e.g., a ControlNet module) is configured to encode a depth map into a structure guidance signal and inject the structure guidance signal into the generator network and fine-tuning model to control the geometry and position of the generated shadow content. For example, in some embodiments of this application, the control module includes: a locked feature extraction network, a conditional encoding network, and a feature fusion unit, wherein the locked feature extraction network is used to lock the pre-trained U-Net weights and extract multi-scale original tiling features from the initial synthesized image; the conditional encoding network is used to encode the depth map into multi-scale depth features; and the feature fusion unit is used to fuse the depth features with the original tiling features to generate spatial guidance features, which are then injected into the corresponding layer of the generator network.

[0094] The generator network 132 (i.e., the diffusion model main network U-Net) is configured to perform iterative denoising in the latent space to generate lighting effects. For example, in some embodiments of this application, the generator network includes: an encoder-decoder structure, a spatial mask injection unit, and a latent space weighted fusion unit. The encoder-decoder structure consists of multiple downsampling layers, intermediate layers, and upsampling layers, with residual connections and attention mechanisms between layers. The spatial mask injection unit is configured to construct a visibility matrix in the attention layer based on the shadow generation region mask and the detail recovery region mask to restrict feature interactions between the shadow area and the protected area. The latent space weighted fusion unit is configured to perform the following operations: downsample the shadow generation region mask to the latent space size to obtain the latent space mask Mlatent corresponding to the shadow generation region mask; generate the noisy latent variable Zcompnoisy corresponding to the current denoising step of the preliminary synthesized image; perform weighted fusion according to the following algorithm: z_final = Mlatent ⊙ Zpred + (1-Mlatent) ⊙ Zcompnoisy, ensuring that the shadow area adopts the generated feature Zpred predicted by the diffusion model, and the non-shadow area retains the original feature Zcompnoisy; iterate until denoising is completed, and output the initial light and shadow synthesized image through the VAE decoder.

[0095] The protected region feature replacement module 140, based on the detail restoration region mask, compensates the high-frequency texture features obtained by the high-frequency feature extraction module back to the synthesized image, ensuring the fidelity of details in key portrait features. For example, in some embodiments of this application, the protected region feature replacement module 140 includes: an alignment unit, a feature transfer unit, and a consistency calibration unit. The alignment unit is used to ensure spatial alignment between the current synthesized image, the detail restoration region mask, and the high-frequency features. The feature transfer unit is used to perform element-wise multiplication of the high-frequency texture features and the detail restoration region mask, then multiply by the dynamic gain coefficient β, and superimpose it onto the current synthesized lighting image to obtain a new lighting composite image. The consistency calibration unit is used to perform histogram matching and edge sharpening on the fused image to eliminate possible artifacts and output the final synthesized image (e.g., the target lighting composite image 100).

[0096] In other words, in the embodiments of this application, the original portrait image 101, the target background image 102, and the portrait mask 103 are input into the initial composite image acquisition module 110. This module 110 extracts foreground content (including the portrait itself) from the original portrait image 101 using the portrait mask, and then stitches the foreground content with the target background image 102 to obtain the initial composite image 104. The initial composite image 104 is then input into the depth map estimation module 120 to obtain the depth map 105. The depth map 105 and the initial composite image 104 are then input into the shadow generation region mask acquisition module 151 to obtain the shadow generation region mask 106. The depth map 105 is then input into the control module 131 of the diffusion model 130 to obtain spatial guidance features (i.e., structural guidance signals). The spatial guidance features and the shadow generation region mask 106 are input into the generation network of the diffusion model 130 (e.g., the generation network is the main network U-Net of the diffusion model) to fine-tune the model (i.e., the shadow fine-tuning model), and an initial light and shadow composite image 109 is obtained through iterative denoising. In the iterative denoising process, the generation network performs weighted fusion of the predicted latent variables and the latent variables obtained by adding noise to the initial composite image according to the shadow generation region mask, so that light and shadow are generated within the area defined by the shadow generation region mask with the spatial guidance features as constraints, and at least some of the original features of the initial composite image are retained in the area outside the area defined by the shadow generation region mask. In some embodiments of this application, in order to further improve the quality of the synthesized image, a detail restoration region mask 107 is obtained through a detail restoration region mask acquisition module 152; the initial synthesized image 104 is input into a high-frequency feature extraction module 141 to obtain a high-frequency texture feature 108; the detail restoration region mask 107, the high-frequency texture feature 108, and the initial light and shadow synthesized image 109 are input into a protection region feature replacement module 140 to obtain a target light and shadow synthesized image 100; wherein, the protection region feature replacement module 140 is configured to use the high-frequency texture feature 108 as a feature of the region corresponding to the detail restoration region mask 107 on the initial light and shadow synthesized image 109 for compensation.

[0097] It should be noted that in some embodiments of this application, the shadow generation region mask is a spatial redraw magnitude weight map used to control the redraw magnitude of the diffusion model. The pixel value of each pixel on this map is a decimal between 0 and 1. In the latent space fusion step of the diffusion model, the region is divided based on the shadow generation region mask: the features predicted by the diffusion model are used in the shadow area indicated by the shadow generation region mask (i.e., the area with a weight of 1), while the noisy features of the initial synthesized image are forcibly used in the non-shadow area (i.e., the area with a weight of 0, including the background preservation area and the portrait area). Since the key portrait feature area covered by the mask corresponding to the detail recovery region mask belongs to the non-shadow area, it is indirectly protected in this fusion step, and its feature value is not covered by the diffusion model prediction. In addition, to further strengthen the protection, embodiments of this application also introduce a spatial mask in the attention mechanism of the diffusion model to limit the feature interaction between the shadow area and the non-shadow area, and achieve the final lossless restoration of details through high-frequency texture feature transfer after the generation process is completed. In other words, embodiments of this application force the predicted values ​​of the blending diffusion model and the original features extracted from the initial synthesized image by weighting the shadow generation region mask, and input the depth map into ControlNet to constrain the shape of the shadows generated by the diffusion model. Simultaneously, to prevent features from contaminating each other in the attention mechanism, a spatial mask is also needed to limit the information flow between shadow and non-shadow areas. Finally, if there are still a small amount of residual error, embodiments of this application also use high-frequency texture transfer as a final compensation method to force the original details back onto the image. Some embodiments of this application form a complete protection system for the composite portrait image from process to result, from macro to micro, through the following four-layer mechanism: region forced blending, structure-guided constraints, feature interaction restrictions, and detail compensation of high-frequency texture features.

[0098] The following is combined Figure 2 This application provides an exemplary embodiment of a portrait compositing method, which includes:

[0099] S110, Obtain an initial composite image, wherein the initial composite image is obtained by fusing the foreground of the original portrait image with the target background image.

[0100] S120, depth estimation is performed on the initial synthesized image to obtain a depth map, wherein the pixel values ​​in the depth map are used to characterize the depth information of each pixel point on the initial synthesized image.

[0101] S130, Obtain a shadow generation region mask based on the depth map, wherein the shadow generation region mask is used to limit the pixel region where light and shadow are generated during the diffusion model iterative denoising process.

[0102] It should be noted that in some embodiments of this application, the shadow generation region mask is the same size as the original portrait image, while in some embodiments, the shadow generation region mask input to the diffusion model is an image smaller than the original portrait image. This is because the diffusion model (such as Stable Diffusion) operates in the latent space, which is the low-dimensional feature space of the original image after VAE encoding, and its size is smaller (e.g., 64x64). Therefore, it is necessary to downsample the shadow generation mask region mask (e.g., 512x512) generated in the pixel space to the size of the latent space to obtain the shadow generation mask in the latent space, i.e., the latent space mask, so that it can be used in each step of latent space fusion.

[0103] S140, the depth map is encoded into a structure guidance signal, wherein the structure guidance signal is used to control the geometry of the generated light and shadow.

[0104] For example, in some embodiments of this application, the encoding process is implemented using ControlNet: the depth map is input into the conditional encoding network of ControlNet and fused with the locked features of the initial synthesized image to generate a multi-scale structure-guided feature map. In other embodiments, the depth map can also be directly encoded into a single-scale or multi-scale structure-guided signal using other encoding networks (such as convolutional neural networks). The structure-guided signal is injected into each layer of the U-Net of the diffusion model (i.e., the generator network) to ensure that the generated shadows and other lighting effects conform to the three-dimensional spatial structure defined by the depth map.

[0105] S150, the initial synthesized image, the shadow generation region mask, and the structure guidance signal are input into the generation network for iterative denoising to generate the initial light and shadow synthesized image.

[0106] For example, in some embodiments of this application, the generator network is a diffusion model, U-Net, which is used for iterative denoising to generate image features. In some embodiments of this application, the generator network includes... Figure 1 The generative network and fine-tuning model are described. The generative network can be a diffusion model, U-Net, used for iterative denoising and generating image features. The fine-tuning module can be a shadow LoRA model.

[0107] It should be noted that, in the diffusion model iterative denoising process of the embodiments of this application, the latent variables predicted by the generating network and the latent variables obtained by deterministic noise addition processing of the initial synthesized image are weighted and fused according to the shadow generation region mask, so that light and shadow are generated within the area defined by the shadow generation region mask with the structure guidance signal as a constraint, and the original features of the initial synthesized image are preserved in the area outside the area defined by the shadow generation region mask (e.g., including part of the background area).

[0108] The following example illustrates... Figure 2 The implementation process of the relevant steps.

[0109] In some embodiments of this application, S110 is used to parse multimodal input data and establish pixel references. On the other hand, it can also perform optimizations to address edge color bleeding and transparency loss issues in green screen shooting environments, thereby providing clean and realistic layer tensors for subsequent high-precision compositing.

[0110] Some embodiments of this application can perform preliminary color overflow elimination by executing S110, that is, to achieve preliminary correction of obvious color overflow areas in the pixel space, providing a clean base for subsequent processing. The processing of S110 includes the following three sub-steps: color overflow identification, that is, automatically identifying green color overflow areas affected by green screen reflection by analyzing the tonal distribution of foreground edge pixels in the HSV color space (pixels with H channel in the green range and S and V above the threshold); color compensation, that is, performing color compensation on the affected pixels using a preset desaturation operator (for example, adjusting the green component towards the color direction of adjacent non-color overflow pixels), to initially eliminate the green edge phenomenon; and Alpha channel optimization, that is, performing sub-pixel-level gradient smoothing processing on the transparency channel to preserve the semi-transparent features of high-frequency areas such as hair strands.

[0111] For example, in some embodiments of this application, obtaining the initial synthesized image in step S110 includes:

[0112] The first step is to sample pixels from the green screen area in the original portrait image and calculate the average chromaticity of the sampled pixels as the background reference color.

[0113] The second step is to calculate the color distance between the color vector of each pixel in the original portrait image and the background reference color for each pixel. That is, to determine the color vector of any pixel P in the original portrait image, and then calculate the color distance between each color vector and the background reference color.

[0114] For example, the first and second steps mentioned above include: sampling pixels from the green screen region of the input image (i.e., the original portrait image), calculating its average chromaticity as the background reference color Cref, and simultaneously calculating the variance of the chromaticity distribution for subsequent threshold setting. For any pixel P in the original portrait image, its color vector is C(P), and the color distance between it and the background reference color Cref is calculated.

[0115] The third step is to map the color distance to a preliminary transparency mask based on the start threshold and the stop threshold. If the color distance is less than or equal to the start threshold, the corresponding pixel is marked as completely transparent; if the color distance is greater than or equal to the stop threshold, it is marked as completely opaque; if the color distance is between the start threshold and the stop threshold, it is marked as semi-transparent by linear interpolation. The start threshold and the stop threshold are determined based on the mean and variance of the chromaticity distribution on the original portrait image, and the start threshold is less than the stop threshold.

[0116] In other words, in some embodiments of this application, the third step includes: traversing each color distance and obtaining a preliminary transparency mask according to the following principles: if it is confirmed that the value of the corresponding color distance is less than or equal to the starting threshold, then the pixel value of the pixel corresponding to the color distance being compared is set to a first value; if it is confirmed that the value of the corresponding color distance is greater than or equal to the cutoff threshold, then the pixel value of the pixel corresponding to the color distance being compared is set to a second value; if it is confirmed that the value of the corresponding color distance is between the starting threshold and the cutoff threshold, then the value of the pixel corresponding to the color distance being compared is set to a number between the first value and the second value, wherein the starting threshold is less than the cutoff threshold, and both the starting threshold and the cutoff threshold are determined by the mean and variance of the chromaticity distribution on the original portrait image.

[0117] The fourth step is to smooth the initial transparency mask to obtain the green screen desaturation mask α. smooth .

[0118] For example, the processing steps for the third and fourth steps are as follows:

[0119] A dual-threshold adjustment mechanism is used to map color distance to an initial transparency mask α. prelim To achieve a smooth transition from hard edges to soft edges, the specific implementation process includes:

[0120] Introducing a soft edge initiation threshold T low and cutoff threshold T high Map the color distance to an initial transparency mask α prelim :

[0121] α prelim (P) = min(1, max(0, (ΔC - T low ) / (T high - T low )))

[0122] Wherein: T low = mean - 2σ (mean minus 2 variances); T high= mean + 2σ (mean plus 2 variances); when ΔC ≤ T low : Pixels belong entirely to the background, α=0; when ΔC ≥ T high : The pixel belongs entirely to the foreground, α=1; when T low <ΔC < T high : The pixel is in the transition region, and α is linearly interpolated between 0 and 1.

[0123] Gaussian smoothing is applied to the initial transparency mask to obtain the green screen desaturation mask α. smooth This step is used to further soften the edges, making the semi-transparent transition more natural.

[0124] After performing the four steps from step one to step four above, the output is an optimized four-channel layer. In other words, the generated green screen desaturation mask α smooth Saved as The alpha channel is read and used in subsequent steps.

[0125] The fifth step is to merge the original portrait image with the green screen desaturation mask to generate a foreground layer with a transparency channel.

[0126] The sixth step is to fuse the foreground layer with the target background image to obtain the initial composite image.

[0127] For example, in some embodiments of this application, the foreground of the original portrait image and the target background image are fused according to the green screen desaturation mask to obtain the initial composite image. For example, using the green screen desaturation mask α smooth The median value is replaced by the value taken from the Mhuman image mask. In other words, adaptive tiling processing is performed between the foreground and background images. The system then applies the optimized four-channel cutout layer according to the preset layout. Overlay onto target background image Above, and based on the pixel weights in the alpha channel Linear blending of layers generates a preliminary composite image. .

[0128] To improve the accuracy of the obtained shadow generation region mask, in some embodiments of this application, the process of generating the shadow generation region mask based on the depth map includes:

[0129] The first step is to determine a depth constraint mask based on the depth map, wherein the depth constraint mask is used to indicate a spatial region with a depth not lower than that of the human figure.

[0130] For example, in some embodiments of this application, the first step includes:

[0131] The foreground subject region is determined based on the portrait mask, wherein the foreground subject region is used to represent the set of pixels in the portrait with a confidence level higher than a preset threshold. For example, the portrait mask Mr is obtained, and the positions of all pixels with pixel values ​​greater than a preset threshold (e.g., 0.1) are located to obtain the foreground subject region.

[0132] The depth values ​​corresponding to the foreground subject region are extracted from the depth map, and the average of all extracted depth values ​​is calculated to obtain a depth threshold. For example, this step includes: extracting the corresponding depth values ​​of these locations in the foreground subject region in the depth map, and calculating the arithmetic mean of these depth values ​​as a depth threshold τ, which is used to quantify the average depth level of the portrait.

[0133] The depth constraint mask is obtained by binarizing the depth map according to the depth threshold. For example, the process of obtaining the depth constraint mask by binarizing the depth map according to the depth threshold includes: for each pixel in the depth map, determining whether the depth value of the corresponding pixel is greater than or equal to the depth threshold; if the determination result is yes, then marking the corresponding pixel as a first value; if the determination result is no, then marking the corresponding pixel as a second value; after marking all pixels, a binary image is obtained, and the binary image is used as the depth constraint mask. That is, in some embodiments of this application, for each pixel in the depth map, the following operations are performed respectively: if it is determined that the depth value of the corresponding pixel is greater than or equal to the depth threshold, then the pixel value of the corresponding pixel is assigned a first value (for example, the first value is the number 1), otherwise it is assigned a second value (the second value is the number zero), resulting in a binary image, and the binary image is used as the depth constraint mask D. For example, this step includes: binarizing the depth map, and for each pixel location, assigning a value of 1 if its depth value satisfies depth ≥ τ, and assigning a value of 0 otherwise. The resulting binary image is the depth constraint mask D, which is used to indicate spatial regions in the scene with a depth no less than that of a human figure that may be suitable for shadow casting.

[0134] The second step is to determine an extended shadow mask based on the geometric prior of the portrait, wherein the extended shadow mask is used to indicate shadow candidate regions with natural attenuation characteristics in the scene, and the geometric prior includes the position and / or size of the portrait.

[0135] For example, in some embodiments of this application, the second step includes: determining a portrait region based on a portrait mask (the portrait region refers to the overall spatial range of the portrait identified by the portrait mask, usually represented by a minimum bounding rectangle, which may include semi-transparent edges (which will be expanded later)), wherein the portrait region is used to characterize the overall spatial range of the portrait; obtaining a preliminary shadow region based on the geometric features of the portrait region (such as the minimum bounding rectangle) and a preset scaling factor; and sequentially performing expansion and blurring processing on the preliminary shadow region to obtain the extended shadow mask. For example, in some embodiments of this application, the second step includes: obtaining a minimum bounding rectangle (which outlines the area where the person is located) based on the portrait mask (a mask reflecting the location of the person), generating a geometric shadow region on the upper side (corresponding to the top of the person's head) and the lower side (corresponding to the bottom of the person's feet) of the rectangle according to a preset scaling factor, and obtaining the extended shadow mask Sexp after the region is expanded and blurred.

[0136] In other words, the extended shadow mask Sexp in this application embodiment refers to an intermediate mask obtained by generating a preliminary projection area based on the minimum bounding rectangle of the portrait mask according to a preset scaling factor, and then performing morphological dilation and Gaussian blurring. This extended shadow mask is used to characterize the approximate range where shadows may exist, and can be logically fused with the depth constraint mask to generate the final shadow generation area mask. The value of each pixel in the extended shadow mask represents the degree to which the position belongs to the candidate shadow area (0 or 1 in binary mode, and a continuous value of 0 to 1 in grayscale mode).

[0137] The third step is to solve for the intersection of the spatial regions defined by the depth constraint mask and the extended shadow mask to obtain the shadow prediction region mask.

[0138] For example, in some embodiments of this application, the shadow prediction region mask S is obtained by taking the union of the extended shadow mask and the depth constraint mask: .

[0139] The fourth step is to obtain the shadow generation region mask based on the shadow prediction region mask.

[0140] For example, in some embodiments of this application, the fourth step includes: taking the union of the shadow region defined by the shadow prediction region mask and the portrait dilation region defined by the portrait dilation region mask to obtain an initial shadow generation region, wherein the portrait dilation region mask is obtained by morphological dilation of the portrait mask and is used to cover the semi-transparent area of ​​the portrait edge; subtracting the protection region defined by the protection region mask from the initial shadow generation region to obtain a target shadow generation region, wherein the protection region mask is taken from the portrait mask or obtained by semantic segmentation of the initial synthesized image, and the shadow generation region mask is used to define the target shadow generation region.

[0141] For example, in some embodiments of this application, the process of obtaining the protected region mask includes: performing semantic segmentation on the original portrait image to extract a first feature mask corresponding to a target category (e.g., including face, hand, or hair), wherein the target category includes at least one of face, skin, and clothing; performing target detection and instance segmentation on the original portrait image to extract a second feature mask corresponding to the target category; taking the union of the first feature mask and the second feature mask, and then performing dilation and smoothing processing to obtain the protected region mask. For example, in some embodiments of this application, the portrait synthesis method further includes the following process for determining the above-mentioned protected region mask: performing semantic segmentation on the initial synthesized image to extract a first feature mask corresponding to a target category, wherein the target category includes one or more of the following categories: face, hand, hair, and transparent clothing; performing target detection and instance segmentation on the initial synthesized image to extract a second feature mask corresponding to the target category; taking the union of the first feature mask and the second feature mask, and then performing dilation and smoothing processing to obtain the protected region mask.

[0142] It should be noted that the extended shadow mask Sepf is a projection area based on the portrait mask at the preset position of the shadow under the feet. In other words, the embodiments of this application assume, through preset hyperparameters, that a target area calculated based on the portrait mask below the feet or knees of the foreground mask is a potential shadow-generating area. This area is then combined with the depth-constrained area defined by the depth-constrained mask calculated from the depth map to obtain a merged area, which is the shadow prediction region mask S. The physical meaning of the shadow prediction region mask is the potential shadow-generating area. The shadow generation region mask Mshadow is the mask for the input diffusion model obtained from the shadow prediction region mask. The physical meaning of this mask is all areas that need to be redrawn by the diffusion model. For example, in some embodiments of this application, the conversion formula between the shadow generation region mask Mshadow and the shadow prediction region mask S is: This formula characterizes, in some embodiments of this application, the shadow generation area For the shadow prediction region S and the human figure dilation region mask The union of (the regions used to represent the expanded human image area) minus the protective region mask. (Used to characterize the region where the original features should be preserved).

[0143] In the embodiments of this application, the physical meaning of the shadow generation region mask Mshadow is all the regions that need to be redrawn by the diffusion model.

[0144] In some embodiments, the portrait synthesis method further includes: constructing a spatial constraint operator based on the depth map and the portrait mask, wherein the spatial constraint operator includes: the shadow casting start position, the shadow casting direction, and the shadow attenuation law; determining the extended shadow mask based on the geometric prior of the portrait includes: determining the extended shadow mask based on the spatial constraint operator.

[0145] In order to further preserve high-frequency texture features and avoid the influence of the diffusion model on these regional features, in some embodiments of this application, it is also necessary to obtain a detail recovery region mask and paste the high-frequency texture features on the initial synthetic image back to the light and shadow synthetic image output by the diffusion model according to the detail recovery region mask.

[0146] For example, in some embodiments of this application, the portrait synthesis method further includes:

[0147] The first step is to obtain the detail recovery region mask, wherein the detail recovery mask is the complement of the shadow generation region mask.

[0148] For example, in some embodiments of this application, the detail recovery region mask can be determined by the following formula:

[0149]

[0150] in, Generate a region mask for the shadow. For detail recovery region mask. That is, the detail preservation region of the final high-frequency detail overlap in these embodiments of this application. This is the inverse value of the shadow generation area.

[0151] It should be noted that the detail restoration area mask can be a binary mask of the same size as the original portrait image, using the number 1 to represent the pixels that need to be protected for portrait features, and using the number 0 to represent the areas that do not need to be protected for portrait features.

[0152] The second step is to perform low-pass filtering on the initial synthesized image to obtain low-frequency components.

[0153] For example, in some embodiments of this application, a low-pass filter is used to perform low-pass filtering on the initial synthesized image to obtain low-frequency components.

[0154] The third step is to extract high-frequency texture features based on the initial synthesized image and the low-frequency components. For example, the high-frequency texture features are obtained by calculating the difference between the initial synthesized image and the low-frequency components.

[0155] The fourth step involves superimposing the high-frequency texture features onto the image to be synthesized into a lighting and shadow image based on the mask weights of the detail recovery region mask, thereby obtaining the target lighting and shadow composite image. The image to be synthesized into a lighting and shadow image includes either the initial lighting and shadow composite image or a high-resolution lighting and shadow composite image. For example, the high-resolution lighting and shadow composite image is obtained by processing the initial lighting and shadow composite image through high-resolution or other methods.

[0156] It should be noted that, in some embodiments of this application, the step S150, which involves inputting the initial synthesized image, the shadow generation region mask, and the structure guidance signal into the generator network for iterative denoising to generate the initial light and shadow synthesized image, includes: performing controlled iterative denoising in the latent space. For each denoising step, the generator network is configured to: estimate the current prediction latent variable based on the structure guidance signal; obtain a noisy latent variable matching the current noise level obtained by deterministic denoising of the initial synthesized image; perform weighted fusion of the prediction latent variable and the noisy latent variable based on the shadow generation region mask to obtain the current fused latent variable (for example, within the region defined by the shadow generation region mask, the value of the prediction latent variable is used; outside the region defined by the shadow generation region mask, the value of the noisy latent variable is used), as the input for the next denoising step; iteratively execute until denoising is completed, and decode the final fused latent variable to obtain the initial light and shadow synthesized image.

[0157] For example, in some embodiments of this application, the portrait synthesis method further includes: obtaining a detail restoration region mask, wherein the detail restoration region mask is used to identify regions where original portrait features need to be preserved; estimating the current prediction latent variable based on the structure guidance signal includes: predicting the prediction latent variable based on the structure guidance signal and a spatial mask, wherein the spatial mask is constructed based on the shadow generation region mask and the detail restoration region mask, and is used to limit feature interactions between the region defined by the shadow generation region mask and the region defined by the detail restoration region mask during the computation of the generation network.

[0158] In other words, in some embodiments of this application, the region restriction effect of the shadow generation region mask is achieved by performing mask weighted iteration in the latent space of the diffusion model (the latent space is the high-dimensional feature space encoded by the variational autoencoder); in each denoising stage, the system uses the shadow generation region mask Mshadow to linearly fuse the predicted latent variable and the preset background latent variable (e.g., the latent variable obtained by deterministic noise addition to the initial synthetic image), and introduces a spatial mask in the attention mechanism to limit the feature diffusion range, thereby achieving the goal of inducing the generation of physical shadows only within the region preset by the shadow generation mask while preserving the original environmental texture.

[0159] For example, in some embodiments of this application, each denoising step of the diffusion model In this system, the operation is not performed directly in pixel space, but rather in latent space, where the following logic is executed:

[0160] 1) Noise Prediction and Denoising: The generative network model predicts the noise at the current moment and calculates... Potential characteristics of time

[0161] 2) Mask-driven feature fusion: The system uses shadows to generate region masks. (After mapping to the latent space scale) the predicted features are weighted and fused.

[0162] A. In The indicated shadow area: fully retains the features generated by the diffusion model combined with ControlNet and shadow LoRA prediction, i.e., includes the new shadow texture area.

[0163] B. In the non-shaded area The system will force the features of this region to be constrained to the initial composite image. Latent space features after adding noise .

[0164] This operation is performed at the feature level, rather than at the pixel space, during the sampling process of the diffusion model.

[0165] The mathematical expression of the diffusion model in this application embodiment is as follows:

[0166]

[0167] in, Generate a region mask for the shadow. To generate the latent variables for prediction by the network, It is a noisy latent variable obtained from the initial synthesized image.

[0168] It should be noted that, in some embodiments, when a target type image needs to be output, the portrait synthesis method further includes: segmenting the light and shadow fusion image into multiple local blocks with overlapping regions; performing detail enhancement processing on each local block separately; and then seamlessly stitching the processed local blocks together. In some embodiments of this application, after the block processing, the stitched image is then subjected to high-frequency texture feature recovery using a detail recovery region mask. For example, in some embodiments of this application, when a target type image needs to be output (e.g., the target type image is a high-definition composite image with a resolution greater than a set threshold), the portrait synthesis method further includes: segmenting the initial light and shadow composite image into multiple local blocks with overlapping regions; performing detail enhancement and resolution enhancement processing on each local block separately; and then seamlessly stitching the processed local blocks together to obtain the high-definition light and shadow composite image; and, according to the weight of the detail recovery region mask, superimposing the high-frequency texture features onto the corresponding region of the high-definition light and shadow composite image to obtain the target light and shadow composite image.

[0169] The following is combined Figure 3 This application provides an exemplary embodiment of a portrait synthesis method. The figure illustrates the complete processing flow from input data to the final high-definition result image (i.e., the target light and shadow composite image), including the following stages: input data preparation, preliminary synthesis and channel merging, depth estimation, adaptive masking, diffusion model generation to generate high-definition magnification and detail restoration, and finally outputting the portrait composite image.

[0170] Figure 3 The input data layer is used to input background image data (as an example of a target background image), original captured photo data (as an example of an original portrait image), and subject person mask data (as an example of a portrait mask).

[0171] It should be noted that all types of image data involved in this application, including original portrait images, target background images, portrait masks, and output results at each stage, are digital images in pixels. Color images are represented using RGB three channels, while grayscale images are single-channel. The spatial resolution of all images is characterized by width × height (W × H), and pixel values ​​are stored as multi-bit (e.g., 8-bit) integers, ranging from 0 to 255, where 0 represents the lowest brightness (black) and 255 represents the highest brightness (white).

[0172] In other words, all image data involved in this application are represented in digital image form. The size of the image is denoted as width × height (W × H), and the unit is pixels; the color image consists of three color channels: red (R), green (G), and blue (B), while the grayscale image is a single channel; the pixel value of each channel is stored as a multi-bit integer, with a value range of 0 to 255, where 0 represents the lowest brightness and 255 represents the highest brightness.

[0173] Some embodiments provided in this application Figure 3 Human portrait synthesis methods include:

[0174] Step 501 involves performing channel merging and stitching on the original captured photo data and the subject mask data to generate the PNG cutout result image data. This process involves fusing color information with transparency information. The specific operations and the meaning of the result data are as follows:

[0175] Foreground image data obtained by channel mosaicking (i.e. Figure 3 The process (PNG image data cutout result) includes:

[0176] The first step is to prepare the input data.

[0177] The original captured photo data is a standard color photograph containing a human subject. Its data format is W×H×3, meaning that each pixel is composed of the values ​​of three channels: red (R), green (G), and blue (B), which together determine the pixel color of each pixel in the image.

[0178] The subject mask data is a grayscale image used to identify which pixels in the original photograph belong to the foreground (e.g., a portrait) and need to be retained. Its data format is W×H×1, meaning each pixel has only one channel value. In this mask, white (values ​​close to 255) typically represent the portrait, and black (values ​​close to 0) represent the background.

[0179] The second step is the merge operation:

[0180] Align the two images, which are exactly the same size (i.e., W and H are the same), one pixel at a time.

[0181] For each pixel, the R, G, and B channel color values ​​are extracted from the original photograph. Simultaneously, the grayscale value of that pixel is extracted from the subject's mask; this grayscale value is used as the alpha (transparency) channel value. Finally, the extracted R, G, and B values ​​of each pixel are recombine with the alpha value to form a new four-channel pixel data set.

[0182] The four-channel data of all pixels are combined to obtain a new image file, namely the PNG image data of the cutout result. Because the PNG format natively supports the alpha channel, it is the format for storing portrait images with transparent backgrounds.

[0183] It should be noted that, Figure 3The PNG cutout result image data is a digital image, and its specific meanings are as follows: W,H (width × height) represent the pixel size of the image. For example, 1920×1080 means that the image is 1920 pixels wide and 1080 pixels high, which is consistent with the original input photo; 4 (number of channels) indicates that the color depth of the image is composed of 4 channels, which is different from ordinary color photos (3 channels). The extra channel is specifically used to store transparency information. That is to say, the PNG cutout result image of this application embodiment includes the specific content of the 4 channels RGBA. The R (Red) channel data is used to store the brightness information of red in the image, the G (Green) channel data is used to store the brightness information of green in the image, the B (Blue) channel data is used to store the brightness information of blue in the image, and the A (Alpha, transparency channel) channel data is used to store the transparency information of each pixel in the image. The higher the Alpha value (closer to 255), the more opaque the pixel is; the lower the Alpha value (closer to 0), the more transparent the pixel is. The range of values ​​from 0 to 255 represents the range of pixel values ​​in an image. Different values ​​are used to represent the brightness or transparency level of each channel. For example, 0 represents the minimum intensity of the channel (e.g., colorless for RGB and completely transparent for Alpha), and 255 represents the maximum intensity of the channel (solid color for RGB and completely opaque for Alpha).

[0184] The PNG cutout result obtained through channel mosaicking in the embodiments of this application is an RGBA four-channel image with the original background removed, portrait details preserved, and transparent borders included. This image can be used as material to be overlaid onto any new target background (e.g., Figure 3 (The background image data) above.

[0185] Unlike related technologies, in this embodiment, the green screen desaturation mask αsmooth is used as an Alpha channel and merged with the RGB channels of the original portrait image to generate an RGBA image with a transparency channel. Therefore, the Alpha channel data of the RGBA image in this embodiment is the pixel value of the corresponding pixel in the green screen desaturation mask. This embodiment uses the above channel merging operation to solidify the green screen desaturation mask αsmooth as an Alpha channel into the image data, generating an RGBA foreground layer with precise transparency information. This Alpha channel data fully preserves the semi-transparent weight information obtained from double threshold mapping and Gaussian smoothing: the main subject is preserved in the completely opaque area (α=255), the background is removed in the completely transparent area (α=0), and a natural transition between the foreground and background is achieved in the semi-transparent area (0<α<255). This design allows the foreground layer to be used as a standardized four-channel image material for subsequent processing, independent of the original green screen environment. It provides a complete and quality-controllable input base for depth estimation, adaptive mask generation, and diffusion model shadow generation, while ensuring that the original details of semi-transparent objects such as hair and gauze are fully preserved in the initial compositing stage.

[0186] Step 502: The foreground and background are pieced together according to the transparency channel to obtain a collage composite image (as an example of the initial composite image).

[0187] For example, using the alpha channel value from the PNG cutout result as weight, the colors of the foreground pixels and the background pixels are weighted and summed to obtain the initial composite pixels. After performing the above operation on all pixels, the system generates a new collage composite image (i.e., the initial composite image). This image is an RGB three-channel color image, in which the portrait is seamlessly placed on a new background (i.e., background image data, as an example of the target background image) with a natural transition of semi-transparent edges.

[0188] In other words, the embodiments of this application, by performing the above steps 501 and 502, can achieve the following: parsing the original image and mask data, performing green screen overprinting and alpha channel encapsulation, establishing an RGBA layer, and completing the initial tiling. This process includes:

[0189] S11: As Figure 3 As shown, the system receives raw captured photo data through a standardized data interface. and pre-generated foreground mask data (i.e., the main subject mask data, also known as the portrait mask) Perform data alignment verification to ensure that the original image and the mask completely overlap in spatial resolution.

[0190] S12: Perform a channel merging operation on the original photo data and the person mask data, Mapped to an opacity channel, and with The RGB channels are tensor-stitched to generate an initial matting result image with four-channel features. Then, Mhuman is superimposed as the transparency channel onto the RGB channels of the original image to construct a four-channel layer tensor Irgba, achieving a structured encapsulation of color information and edge transparency information.

[0191] S13: Perform green screen spill elimination and edge consistency optimization to generate an optimized four-channel foreground layer. (Compare...) The tonal distribution of edge pixels in the HSV space is automatically identified to detect green overtone areas affected by green screen reflections. A preset desaturation operator is used to perform color compensation on the affected pixels, eliminating the green fringing at the portrait boundaries. Simultaneously, sub-pixel-level smoothing is applied to the alpha channel to preserve the semi-transparent features of high-frequency areas such as hair strands. The processing logic is as follows:

[0192]

[0193] in, This represents a composite function for color bleeding removal and edge repair. This is the optimized, finely cut-out layer.

[0194] The logic for color overflow removal in this embodiment is as follows: When shooting a transparent gauze dress in front of a green screen, but the semi-transparent dress cannot be accurately extracted using image cutout, the overflowing area needs to be semi-transparent to allow the background content to show through. Therefore, this embodiment obtains the pixel area in the color space near the green reference value through chroma keying, and uses this pixel area to generate a grayscale mask between 0-255, allowing the foreground and background to blend in a semi-transparent state. However, some color overflow will still exist (because only the mask controls the blending degree of foreground and background, without removing the green in the portrait). These small amounts of residual color will be removed during shadow generation (the diffusion model will fine-tune the hue while generating shadows to make the blending more natural). Therefore, the operator here is used to obtain a semi-transparent mask of the green area to guide image compositing.

[0195] This step in the embodiment of the application performs a chroma keying algorithm, calculating based on the difference between pixel color and a preset background chroma space, to achieve preliminary separation of the foreground object from the background environment. Specifically, for any pixel in the original photograph... Its color vector is The system calculates the vector and the target background reference color. Color distance between To achieve a smooth transition at the edges, the system introduces a dual-threshold adjustment mechanism, adjusting the distance... Mapped to a preliminary transparency mask Its expression is as follows:

[0196]

[0197] in, and These are the preset soft edge start threshold and stop threshold ( ). and Dynamically calculated based on the background average chromaticity (mean plus or minus two variances), this is used to define the weight distribution of pixels between the foreground subject and the background region. Then... Smoothing process is performed to obtain the green screen desaturation mask. in Represents convolution operation. For variance The Gaussian convolution kernel. Finally, based on... Eliminate common color bleeding and green screen residue issues in green screen footage. For example, in some embodiments of this application, the green screen desaturation mask is directly fused into the portrait mask Mhuman to make the transparent clothing area of ​​the person in the original captured image semi-transparent again.

[0198] In some embodiments of this application, (Used to identify areas with transparent clothing in Mhunman), directly replace Mhunman, that is, turn the transparent clothing areas in the character area back into semi-transparent areas.

[0199] S14: Perform adaptive tiling of the foreground layer and background image. Based on the preset layout, the optimized four-channel cutout layer... Overlay to background image Above, and based on the pixel weights in the alpha channel. Linear blending of layers generates an initial composite image. The calculation logic is as follows:

[0200] Formula 2

[0201] This step involves spatially collaging the foreground and background, and using mathematical calculations to achieve physical-level edge blending, thus addressing the harshness often found in green screen compositing.

[0202] It should be noted that reading The Alpha channel A(p) = αsmooth, and Alpha blending (Formula 2) is performed to obtain the initial synthesized image.

[0203] S503, obtain a depth map based on the collage composite image (i.e., an example of the initial composite image).

[0204] For example, in some embodiments of this application, S503 uses DepthAnythingV2 as a monocular depth estimation model to recover depth information from the initial synthesized image and obtain a depth map. That is, embodiments of this application extract three-dimensional spatial geometric information from the initial synthesized image to generate a depth map, providing physical constraints for subsequent shadow generation.

[0205] It should be noted that step 503 includes:

[0206] S21, to synthesize the initial image The input is fed into a pre-trained depth estimation model (DepthAnything2, i.e., the monocular depth perception version of the DepthAnything v2 algorithm, which can obtain a grayscale image of depth information from 0 to 255, i.e., a depth map, based on the input RGB image). In other words, some embodiments of this application utilize the monocular depth perception capability of the DepthAnything2 model to perform global continuous depth estimation on all objects in the scene, generating a feature tensor reflecting the distance relationships within the scene. This feature can capture the spatial contact relationship between the human figure and the background ground and facades, providing a basis for the logical localization of shadows.

[0207] In some embodiments of this application, the depth estimation model is configured to receive an input initial synthetic image and mine the relative distance information of each pixel in the scene of the image to obtain a depth map.

[0208] S22: Perform numerical mapping and scaling transformation on the feature tensor to generate a standardized spatial depth map. (i.e., depth map). Based on a preset scene scale, the system converts the relative depth values ​​output by the model into a spatially meaningful gradient distribution. This distribution indicates the physical distance from each pixel in the synthesized scene to the virtual camera, and its mathematical mapping relationship is as follows:

[0209]

[0210] in, The nonlinear mapping function representing the deep model, These are pixel coordinates. This depth map serves as a rigid spatial constraint to ensure that the subsequently generated lighting and shadow features closely match the environmental structure in terms of perspective logic.

[0211] S23: Through analysis The gradient abrupt change region in the image is automatically identified, and the contact boundary between the portrait and the background plane (i.e., the potential shadow projection area) is automatically identified. This signal is then passed to the subsequent adaptive mask extraction module.

[0212] It should be noted that in some embodiments of this application, the depth map is a grayscale image ranging from 0 to 255. The pixel values ​​in this grayscale image are not absolute depth values, but rather represent a spatial relationship of image depth. 0 represents infinity, and 255 represents the lens space. The content in the image is represented by a value along the depth Z-axis within this range, resulting in a grayscale transition image where near-white and far-black pixels appear as grayscale values. In some embodiments of this application, this depth map serves as the depth condition input map for ControlNet (as an example of a control module) to guide shadow generation. Simultaneously, during adaptive mask extraction, the depth space of the portrait is determined, and then the depth map is used to extract the foreground region mask of the portrait space.

[0213] Step 504: Input the collage composite image. Figure 3 The adaptive mask extraction module obtains shadow generation region mask data (i.e., data corresponding to the shadow generation region mask) and detail recovery region mask data. The shadow generation region mask data and the data output from the control network module are then input into the main network (i.e., the generator network) of the diffusion model and the fine-tuned LORA (as an example of a fine-tuning module) iterative sampling and denoising module to obtain a preliminary composite image (i.e., a light and shadow fusion image).

[0214] It should be noted that in some embodiments of this application, LoRA (Low-Rank Adaptation) technique is used when fine-tuning the diffusion model for shadow generation. LoRA is a parameter-efficient fine-tuning method that does not modify the original weights of the pre-trained model (i.e., the diffusion model), but introduces low-rank bypasses to specific layers of the model, learning task-specific incremental information by updating only the bypass parameters.

[0215] The LoRA implementation principle is as follows: For an original weight matrix W0 in the pre-trained model, it is completely frozen during fine-tuning and no longer updated. Simultaneously, two low-rank matrices A and B are introduced for W0, and their product BA is used as a bypass. During forward propagation, the output of the original path and the output of the bypass are superimposed with a certain scaling factor to obtain the final output. Here, the rank r controls the capacity of the bypass, and the scaling factor alpha adjusts the contribution of the bypass to the original model. In practice, alpha is usually set to be equal to r to simplify the setup.

[0216] In some embodiments of this application, the SDXL model is used as the base diffusion model for fine-tuning the shadow generation task to obtain a diffusion model (which includes a control module and a generation network). The SDXL model contains a dual text encoder and a UNet denoising network, and LoRA fine-tuning is applied to the attention layer of the UNet.

[0217] The fine-tuning process aligns with the training objective of the original diffusion model: the model receives noisy latent variables as input and learns to predict the added noise, essentially teaching UNet to reconstruct the original image from the noisy map. The loss function calculates the error between the model's predicted noise and the actual added noise. During fine-tuning, only the A and B matrix parameters in the LoRA module are updated; the original weights of the SDXL model remain frozen.

[0218] To train the LoRA module of this application embodiment, it is necessary to construct sample pairs containing realistic shadows. For example, a set of scenes can be generated using a 3D rendering engine, each scene containing an object (such as a portrait or a general object) and its shadows cast under different lighting conditions. For each scene, two images are rendered: one is the complete scene (containing the object and realistic shadows), and the other is a version containing only the object without shadows. Using the image without shadows as input and the image with shadows as the target, a training pair can be formed. In practice, data can also be collected from real-world shooting: under fixed lighting conditions, background photos with and without object occlusion are taken respectively, shadow areas are obtained through difference, and samples with realistic shadows are synthesized. During training, these image pairs are encoded into the latent space, noise is added according to the noise addition process of the diffusion model, and then the trained diffusion model is allowed to learn to predict the noise, thereby implicitly learning to generate a shadowed output from a shadowless input.

[0219] Through the above fine-tuning, the LoRA module has learned the ability to generate shadows under specific lighting conditions. This allows the basic diffusion model to generate physically accurate contact shadows and ambient light in a specified area (i.e., the area defined by the shadow generation mask) during inference.

[0220] During the inference phase, LoRA is used in two ways: First, in real-time overlay, the LoRA weights are kept separate from the original diffusion model weights. During each forward propagation, the calculation results of the LoRA bypass are dynamically overlaid onto the original UNet weight output with a specified scaling factor (lora_strength), allowing for flexible adjustment of shadow generation intensity during inference. Second, in a fusion and merging method, the LoRA bypass is pre-added to the original weights to generate a new fused weight file (i.e., the weight file corresponding to the diffusion model), eliminating additional overhead during inference. Some embodiments of this application employ the real-time overlay method, controlling the shadow generation effect by adjusting the lora_strength parameter, providing users with flexible adjustment capabilities.

[0221] The embodiments of this application introduce an adaptive mask extraction and shadow generation module by executing step 504, which is used to realize the light and shadow fusion between the portrait and the background environment. By automatically identifying the area affected by light interference and generating a guiding mask, the light and shadow logic is reshaped within a specific range using a diffusion model, thereby eliminating the floating feeling common in synthetic images and enhancing the overall visual realism.

[0222] As an example, step 504 includes:

[0223] S31: Enable adaptive mask extraction algorithm for multi-dimensional analysis of the composite scene. Based on the main character mask... (Right now Figure 1 The system automatically calculates and outputs a set of masks for key redraw regions, including at least the shadow generation region mask, based on the portrait mask and depth map. And detailed region masks for key features of the portrait (such as skin, clothing, and user-specified areas). This allows for precise spatial limitation of the generated range.

[0224] The following example illustrates how to obtain the shadow generation region mask. and detail recovery area mask The process.

[0225] It should be noted that the shadow generation region mask in this application embodiment is used to determine a physically reasonable projection area and to constrain its spatial distribution in combination with depth information to avoid the shadow from having a floating feeling.

[0226] The process of obtaining the mask set for the key redraw region includes:

[0227] The first step is depth constraint extraction.

[0228] First, the DepthAnythingV2 depth estimation model is used to predict the depth of the input human image to obtain a depth map. Calculate the human face mask. (This human face mask is used to represent the region in the depth map whose depth is in front of the human face, with the human face plane as the boundary.) The average depth within the covered region is used as the threshold. The depth constraint mask is obtained based on this threshold. This is used to ensure that shadows are generated only in physical space that is deeper than the background of the character.

[0229] In other words, the first step includes the following steps:

[0230] Step 1: Depth Map Generation

[0231] The initial synthesized image is input into the pre-trained depth estimation model DepthAnythingV2. Through a monocular depth perception algorithm, a depth map of the same size as the input image is output, where each pixel value reflects the relative scene depth at that point.

[0232] Step 2: Depth Threshold Calculation

[0233] Read the portrait mask Mr, locate all pixel positions (i.e., the foreground subject region) where the pixel value is greater than a preset threshold (e.g., 0.1), and obtain the foreground subject region; extract the corresponding depth values ​​of these positions in the depth map depth, and calculate the arithmetic mean of these depth values ​​as the depth threshold τ. This threshold quantifies the average depth level of the portrait.

[0234] Step 3: Depth Constraint Mask Generation

[0235] The depth map is binarized: for each pixel location, if its depth value satisfies depth≥τ, then the pixel is assigned a value of 1; otherwise, it is assigned a value of 0. The resulting binary image is the depth constraint mask D, used to indicate spatial regions in the scene with a depth no less than that of the human figure that may be suitable for shadow casting.

[0236] The second step is geometric projection calculation: Based on the human face mask (a mask reflecting the person's location), the minimum bounding rectangle is obtained (this bounding rectangle roughly outlines the area where the person is located). Geometric shadow areas are generated on the top and bottom sides of the rectangle according to a preset scaling factor. After dilation and blurring of these areas, the extended shadow mask is obtained. .

[0237] The third step is logical fusion and hidden surface removal: the union of the extended shadow mask and the depth constraint mask is taken to obtain the shadow prediction region mask. .

[0238] Fourth, determine the shadow generation region mask using the following formula based on the shadow prediction region mask:

[0239]

[0240] Shadow generation area The shadow prediction region mask S is the union of the expanded human figure region and the region to be retained, and then the region to be retained is subtracted.

[0241] It should be noted that the shadow generation area mask This is the output of the entire adaptive mask extraction module. Sexp is a projection area calculated based on the preset position of the shadow under Mhuman's feet. In other words, some embodiments of this application use preset hyperparameters (assuming a specific area calculated based on Mhuman's feet or knees) as a potential shadow-generating region. This region is then combined with a depth-constrained region calculated from the depth map to obtain a merged region, which is S. The physical meaning of the shadow prediction region mask S is the potential shadow-generating region. Shadow generation region mask The physical meaning of is all areas that need to be redrawn by the diffusion model, including at least a portion of the shadow generation area.

[0242] In some embodiments of this application, the formula for calculating the detail recovery region mask is as follows:

[0243]

[0244] In other words, the detail recovery mask is the inverse of the shadow generation region mask.

[0245] It should be noted that, in order to better represent the features of the original captured image in the detail restoration area, some embodiments of this application require the use of a detail restoration area mask (i.e., M). prot The high-frequency texture features are transferred to the corresponding area of ​​the light and shadow composite image to obtain the ultimate light and shadow composite image (i.e. the target light and shadow composite image).

[0246] To obtain the detail recovery region mask or the shadow generation region mask, a protection region mask is also needed. In some embodiments of this application, the protection region mask extraction method employs the following dual-track identification method, which includes:

[0247] 1) The first path uses the SegFormer semantic segmentation model to perform pixel-level semantic segmentation on the initial synthesized image to obtain a category map. And extract the segmentation mask corresponding to the face category. .

[0248] 2) The second path uses the YOLO object detection model to locate the face and hand detection boxes in the initial synthesized image, and then segments the detection box regions based on the SAM instance segmentation model to obtain the face instance segmentation mask and the hand instance segmentation mask, denoted as... .

[0249] The main protected area for portrait photography is .

[0250] 3) Combine the face segmentation mask with the instance segmentation mask. Take the union of the results and apply dynamic dilation and Gaussian smoothing to obtain the protected region mask. .

[0251] In other words, the methods for obtaining the protection region mask in some embodiments of this application include:

[0252] The first step is to perform semantic segmentation on the original portrait image and extract the first feature mask corresponding to the category of interest (e.g., face category, skin or hair).

[0253] The second step involves performing target detection and instance segmentation on the original portrait image, and extracting the second feature mask corresponding to the category of interest (e.g., face and / or hand).

[0254] The third step is to take the union of the first feature mask and the second feature mask, and then perform dilation and smoothing to obtain the protection region mask.

[0255] It is not difficult to understand that the protection region mask in the embodiments of this application... The acquisition employs a dual-track recognition approach to ensure robustness, specifically including: a semantic segmentation path that provides pixel-level face region recognition with comprehensive coverage; a detection and instance segmentation path that accurately locates faces and hands, particularly suitable for handling occlusion and multi-angle situations; multi-feature fusion that simultaneously protects faces and hands, covering key areas most prone to identity distortion in portrait synthesis; and adaptive post-processing, namely dynamic dilation and Gaussian smoothing, ensuring sufficient coverage of the protected area and a natural transition to the surrounding environment.

[0256] In other words, some embodiments of this application provide a protection region mask. The method for obtaining the protected region mask is as follows: the protected region mask is used to identify the human face area or background element area that needs to be preserved. For example, in some embodiments of this application, the protected region mask protects the face and hand areas through a fusion strategy of semantic segmentation and instance segmentation.

[0257] S505, combine the shadow generation region mask data (which is a grayscale image) and the depth map (from...) Figure 3The image output by DepthAnything2) is input to the diffusion model (which includes...). Figure 3 The control module and the generation network are used to input the depth map into the control module and the shadow generation region mask into the generation network to obtain an initial light and shadow composite image. In some embodiments of this application, this initial light and shadow composite image is obtained through... Figure 3 The fine-tuning model (used to adjust the model that generates shadow effects, for example, fine-tuning LORA) is obtained by adjusting the diffuse model to generate a composite image of light and shadow.

[0258] As an example, the S505 includes:

[0259] S32: Construct a conditional inference environment based on multimodal guiding signals. Load the pre-trained Diffusion Model weights (as an example of a generative network) and its accompanying LoRA shadow generation fine-tuning model (as an example of a fine-tuning model). Then, the generated depth map... The input control module serves as the structural control signal for the diffusion model. The diffusion model is configured to incorporate the text description of the current scene into the shadow generation region mask. Initialize the noise distribution within the indicated area to ultimately generate projection features consistent with ambient lighting logic.

[0260] S33: Perform controlled generative sampling and lighting reshaping to obtain an initial composite lighting image. During iterative denoising inference, the diffusion model performs pixel sampling within the area defined by the shadow generation region mask Mshadow, using a control module (e.g., ControlNet) to forcibly anchor the spatial structure of the original scene, synthesizing shadows with natural attenuation characteristics; simultaneously, a spatial mask is introduced into the attention mechanism to limit the feature interaction between the shadow generation region mask and the detail recovery region mask. The transparency and distribution range of the shadow follow the weight distribution of the adaptive mask, generating a local lighting field. Compared with the initial composite image Dynamic overlay is performed according to the following logic:

[0261]

[0262] in, This represents the initial composite image after the lighting and shadow reshaping is completed. This step is used to ensure the natural blending of shadows and background textures, and to achieve deep coupling between the foreground and background in the lighting and shadow dimension.

[0263] It should be noted that the shadow generation region mask in this embodiment of the application has a region restriction effect during the image generation process of the diffusion model. This region restriction effect is achieved by performing mask-weighted iterations within the latent space of the diffusion model (the latent space is the high-dimensional feature space encoded by a variational autoencoder); in each denoising stage, the shadow generation region mask is used. The predicted latent variables and preset background latent variables (e.g., noisy latent variables) are linearly fused, and a spatial mask (i.e., the result obtained by encoding the depth map by the control module) is introduced into the attention mechanism to limit the feature diffusion range, thereby achieving the goal of guiding the generation of physical shadows only within a preset area while preserving the original environmental texture. In other words, in the embodiments of this application, the region restriction effect of the shadow generation region mask on the diffusion model is achieved by performing mask-weighted iterations within the latent space of the diffusion model. Specifically, in each denoising step, based on the shadow generation region mask... The latent variables predicted by the diffusion model are linearly fused with the noisy latent variables corresponding to the initial synthetic image, so as to generate light and shadow features only in the area indicated by the mask corresponding to the shadow generation area mask while preserving the original content of the non-shadow area.

[0264] Unlike related technologies, some embodiments of this application perform noise reduction at each step of the diffusion model. In this process, instead of directly regenerating the entire image in pixel space, the image is regenerated in latent space for the region defined by the shadow generation mask. This process includes:

[0265] 1) Noise Prediction and Denoising: The diffusion model predicts the noise at the current moment and calculates " "Potential characteristics of time"

[0266] 2) Mask-driven feature fusion: generating region masks using shadows. (After mapping to the latent space scale) the predicted features are weighted and fused.

[0267] A. Mask in the shadow generation area Indicated shadow area: Fully preserves the features (i.e., predicted latent variables) generated by the diffusion model combined with ControlNet and shadow LoRA predictions, which contain new shadow texture regions.

[0268] B. In the non-shaded area : Force the features of this region to be constrained as the initial composite image Latent space features after adding noise (i.e., noisy latent variables), the physical meaning of the non-shadow area is the background preservation area and the portrait area without shadows.

[0269] The above operations are performed at the feature level, rather than at the pixel space, during the sampling process of the diffusion model.

[0270] The mathematical expression is:

[0271]

[0272] in, It represents element-wise multiplication, which is the multiplication of corresponding elements of two tensors with the same dimension.

[0273] It should be noted that the generator network, control module, and fine-tuning module described in the embodiments of this application are not limited to a specific implementation. The generator network can adopt a U-Net architecture (such as SDXL) or a Transformer architecture (such as DiT, PixArt-α); the control module can adopt ControlNet, or it can adopt IP-Adapter, training free editing methods, or other conditional injection architectures; the fine-tuning module can adopt LoRA, or it can adopt DreamBooth, Adapter, or other parameter fine-tuning techniques. The core inventive point of this application does not lie in the specific implementation of a particular model.

[0274] For example, in some embodiments of this application, Stable Diffusion XL is used as the generator network, ControlNet as the control module, and pre-trained shadow LoRA is loaded as the fine-tuning module. The generator network is responsible for iterative denoising in the latent space; ControlNet encodes the depth map into spatial guidance features and injects them into the layers of the base generator; shadow LoRA is attached to the base network, adjusting its behavior to be better at generating realistic shadows. The three work together as follows: at each denoising step, the generator network receives the current latent variables and spatial guidance features, and estimates the predicted latent variables. Simultaneously, the latent variables are predicted based on the shadow generation region mask Mshadow. Noise-adding latent variables of the initial synthesized image Weighted fusion enables region control; at the attention layer, a spatial mask is introduced based on the shadow generation region mask Mshadow and the detail recovery region mask Mprot to limit the feature interaction between the shadow region and the protected region.

[0275] Unlike related technologies, the embodiments of this application do not use ControlNet to generate the entire image and then use LoRA to adjust the style. Instead, the image space is divided into a generation region (i.e., the region defined by the shadow generation region mask) and a protection region (e.g., the region defined by the detail recovery region mask) according to the shadow generation region mask Mshadow. In the generation region, the control module (e.g., ControlNet) and the fine-tuning model (e.g., shadow LoRA) work together to generate reasonable shadows and lighting, while in the protection region, the original features are forcibly preserved.

[0276] S506, input the initial light and shadow composite image into the high-definition refinement module to obtain a high-definition light and shadow composite image.

[0277] In some embodiments of this application, the optional high-definition upscaling strategies of the high-definition refinement module include: using a GAN model for super-resolution upscaling, or using a diffusion model again for generative high-definition upscaling.

[0278] The embodiments of this application can use this step to upscale a finite-sized image output by the diffusion model to a target high-definition resolution (such as 4K / 8K) through a block-by-block upscaling strategy, while maintaining image quality.

[0279] S507 inputs the high-definition light and shadow composite image into the detail restoration module to obtain the target light and shadow composite image.

[0280] In some embodiments of this application, S507 includes:

[0281] The frequency domain decomposition step involves separating high-frequency details from low-frequency textures in the initial synthesized image using Gaussian blur.

[0282] The feature transfer step, based on the detail recovery region mask Mprot, compensates for the high-frequency details of the initial synthetic portrait image back to the synthetic image.

[0283] It's easy to understand that this step allows for the lossless restoration of the original portrait's biometric details while preserving the generated lighting and color tones. By executing S507, the high-frequency details of the original portrait are losslessly compensated back into the synthesized image, ensuring the authenticity of the portrait's features.

[0284] The technical effects brought about by adopting the technical solutions of the above embodiments of this application are briefly analyzed below:

[0285] In order to address the technical problem that in green screen compositing results, the edges of the portrait (especially high-frequency detail areas such as hair and transparent clothing) are often accompanied by obvious green halo residue or a harsh cutting effect; and that when the mask is excessively shrunk in order to eliminate color overflow, key high-frequency details such as hair are mistakenly cut off, this application solves the contradiction between eliminating green screen color overflow and preserving portrait details by adopting a phased processing strategy of first coarse processing, then decoupling and then compensation. First, in the preprocessing stage to obtain the initial synthesized image, the green screen overflow area is identified in the HSV color space and color compensation and alpha channel optimization are performed to initially eliminate obvious green edges, providing a clean starting point for subsequent processing. Then, the initial synthesized image is decomposed in the frequency domain, and the low-frequency components representing global color are separated by low-pass filtering. Then, the high-frequency texture features containing microscopic details such as hair strands and skin pores are extracted and saved by subtracting the low-frequency components from the initial synthesized image. On this basis, the diffusion model uses the depth map as a spatial constraint to perform generative redrawing of low-frequency information in the background area defined by the shadow generation mask, achieving uniform ambient lighting without causing high-frequency loss. Finally, based on the detail recovery region mask, the pre-saved high-frequency texture features are non-linearly superimposed back into the synthesized image according to weights, and histogram matching and edge sharpening are used for consistency calibration. Thus, while obtaining realistic lighting effects, the biometric details of the portrait are restored without loss, achieving the dual goals of eliminating background pollution and preserving original details.

[0286] To address the lack of spatial awareness caused by light and shadow discontinuities due to the absence of depth priors in related technologies, this application introduces depth perception and spatial constraint mechanisms to elevate shadow generation from traditional two-dimensional planar tiling to precise control guided by three-dimensional space. First, monocular depth estimation is performed on the preliminary synthesized image to reconstruct the scene's three-dimensional geometry and generate a depth map. Based on the depth map and a human face mask, a depth threshold is calculated, generating a depth constraint mask to exclude physically unreasonable areas such as the air, ensuring that shadows are projected only onto a background plane with a depth not lower than that of the human face. Simultaneously, by analyzing abrupt changes in depth gradients to identify the contact boundary between the foreground and background, a spatial constraint operator containing the shadow projection direction and attenuation law is constructed in conjunction with preset lighting parameters. Based on this, projection calculations and dilation are performed based on the foreground's geometric features. By combining blurring with the generation of an extended shadow mask with a natural decay pattern, and then intersecting this extended shadow mask with a depth constraint mask, a shadow generation mask that satisfies both geometric possibility and physical plausibility is obtained. Finally, the depth map is used as input to ControlNet to inject spatial constraints, and the shadow generation mask is used as a regional constraint. During the iterative denoising process of the diffusion model, the generation of shadow areas and the preservation of non-shadow areas are precisely controlled through latent space weighted fusion. A spatial mask is introduced into the attention mechanism to limit the feature interaction between shadow areas and portrait areas. Finally, within the area defined by the shadow generation mask, contact shadows and ambient light and shadow with natural decay characteristics are synthesized, fundamentally eliminating the common problems of shadow floating and harsh edges in synthesized images, and achieving a spatially realistic light and shadow fusion effect.

[0287] To address the issues of identity feature shift and detail degradation caused by generative reconstruction, some embodiments of this application construct a four-layer progressive control mechanism to systematically resolve the problems of non-target region modification and feature shift caused by the full-image rewriting characteristics of the diffusion model: The first layer is a weighted fusion of latent space regions, using the shadow generation mask Mshadow as the weight. In each denoising step, the latent variables predicted by the diffusion model and the noisy latent variables of the initial synthesized image are forcibly weighted and mixed to ensure Mshadow The first layer employs a defined mechanism: The shadow area generates features while the non-shadow area retains its original features, automatically protecting key areas of the portrait covered by the detail recovery region mask. The second layer uses structural guidance constraints, constructing spatial constraint signals by inputting depth maps into ControlNet, geometrically guiding the model to generate shadow shapes within the shadow area that conform to 3D physical logic. The third layer involves feature interaction adjustment, introducing a spatial mask based on the shadow generation region mask and the detail protection region mask into the attention mechanism to restrict feature flow between shadow and non-shadow areas, preventing feature borrowing and contamination. The fourth layer is final detail compensation; after the generation process is complete, high-frequency texture features of the original portrait are forcibly compensated back into the synthesized image based on the detail recovery region mask, achieving lossless detail recovery. These four layers—from forced result correction to geometric guidance, feature interaction blocking, and final detail compensation—constitute a complete protection system from process to result, from macro to micro, enabling this application to fully utilize the powerful generation capabilities of the diffusion model while precisely achieving differentiated control over shadow area generation and non-shadow area preservation. Please refer to [reference needed]. Figure 4 , Figure 4 The present application illustrates a human portrait synthesis apparatus according to an embodiment of the present application. It should be understood that this apparatus is similar to the one described above. Figure 2 Corresponding to the method embodiments, it can execute the various steps involved in the above method embodiments. The specific functions of the device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in the memory or embedded in the device's operating system in the form of software or firmware. The human portrait synthesis device includes: an initial synthesized image tiling processing module 310, a depth map acquisition module 320, a shadow generation region mask acquisition module 330, a spatial guidance feature acquisition module 340, and a light and shadow fusion image acquisition module 350.

[0288] The initial composite image tiling processing module is configured to acquire an initial composite image, wherein the initial composite image is obtained by fusing the foreground of the original portrait image with the target background image.

[0289] The depth map acquisition module is configured to perform depth estimation on the initial synthesized image to obtain a depth map, wherein the pixel values ​​in the depth map are used to characterize the depth information of each pixel point on the initial synthesized image.

[0290] The shadow generation region mask acquisition module is configured to acquire a shadow generation region mask based on the depth map, wherein the shadow generation region mask is used to limit the pixel area where light and shadow are generated during the diffusion model iterative denoising process.

[0291] The spatial guidance feature acquisition module is configured to encode the depth map into a structural guidance signal, wherein the structural guidance signal is used to control the geometry of the generated light and shadow.

[0292] The light and shadow fusion image acquisition module is configured to input the initial synthesized image, the shadow generation region mask, and the spatial guidance features into the generation network for iterative denoising to generate the initial light and shadow fusion image.

[0293] It should be noted that, in the embodiments of this application, during the iterative denoising process, the latent variables predicted by the generating network and the latent variables obtained by deterministic noise addition processing of the initial synthesized image are weighted and fused according to the shadow generation region mask, so that light and shadow are generated within the area defined by the shadow generation region mask with the structure guidance signal as a constraint, and at least some of the original features of the initial synthesized image are retained in the area outside the area defined by the shadow generation region mask.

[0294] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0295] Some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement any of the embodiments included in the above-described human portrait synthesis method.

[0296] Some embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the human image synthesis method as described in any of the above embodiments.

[0297] like Figure 5 As shown, some embodiments of this application provide an electronic device 400, which includes a memory 410 and a processor 420. The memory 410 is used to store a computer program, and the processor 420 can read the computer program from the memory 410 via a bus 430 and execute it to implement any of the embodiments included in the above-described human portrait synthesis method.

[0298] Processor 420 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 420 may be a microprocessor.

[0299] Memory 410 can be used to store instructions executed by processor 420 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 420 of the embodiments of this disclosure can be used to execute the instructions in memory 410 to implement… Figure 2 The method shown. Memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0300] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0301] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0302] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0303] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0304] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0305] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for synthesizing human images, characterized in that, The human image synthesis method includes: An initial synthesized image is obtained by fusing the foreground of the original portrait image with the background image of the target. A depth map is obtained by performing depth estimation on the initial synthesized image, wherein the pixel values ​​in the depth map are used to characterize the depth information of each pixel in the initial synthesized image; A shadow generation region mask is obtained based on the depth map, wherein the shadow generation region mask is used to limit the pixel region where light and shadow are generated during the iterative denoising process of the diffusion model; The depth map is encoded into a structure guidance signal, wherein the structure guidance signal is used to control the geometry of the generated light and shadow; The initial synthesized image, the shadow generation region mask, and the structure guidance signal are input into the generation network for iterative denoising to generate the initial light and shadow synthesized image. In the iterative denoising process, the latent variables predicted by the generating network and the latent variables obtained by adding noise to the initial synthesized image are weighted and fused according to the shadow generation region mask, so that light and shadow are generated within the area defined by the shadow generation region mask with the structure guidance signal as a constraint, and at least some of the original features of the initial synthesized image are retained in the area outside the area defined by the shadow generation region mask.

2. The human image synthesis method as described in claim 1, characterized in that, The generation of the shadow generation region mask based on the depth map includes: A depth constraint mask is determined based on the depth map, wherein the depth constraint mask is used to indicate a spatial region with a depth not lower than the depth level of the human figure. An extended shadow mask is determined based on the geometric prior of the human figure, wherein the extended shadow mask is used to indicate shadow candidate regions with natural attenuation characteristics in the scene, and the geometric prior includes the position and / or size of the human figure; The shadow prediction region mask is obtained by solving the union of the spatial regions defined by the depth constraint mask and the extended shadow mask. The shadow generation region mask is obtained based on the shadow prediction region mask.

3. The human portrait synthesis method as described in claim 2, characterized in that, The step of obtaining the shadow generation region mask based on the shadow prediction region mask includes: The initial shadow generation area is obtained by taking the union of the shadow area defined by the shadow prediction region mask and the portrait expansion region defined by the portrait expansion region mask. The portrait expansion region mask is obtained by morphological expansion of the portrait mask and is used to cover the semi-transparent area of ​​the portrait edge. The target shadow generation area is obtained by subtracting the protected region defined by the protected region mask from the initial shadow generation area, wherein the protected region mask is taken from the portrait mask or obtained by semantic segmentation of the initial synthesized image, and the shadow generation area mask is used to define the target shadow generation area.

4. The human portrait synthesis method as described in claim 3, characterized in that, The human image synthesis method also includes: Semantic segmentation is performed on the original portrait image to extract a first feature mask corresponding to the target category, wherein the target category includes at least one of face, skin and clothing; The original portrait image is subjected to target detection and instance segmentation, and a second feature mask corresponding to the target category is extracted; The protection region mask is obtained by taking the union of the first feature mask and the second feature mask, and then performing dilation and smoothing processing.

5. The human portrait synthesis method as described in claim 2, characterized in that, Determining the depth constraint mask based on the depth map includes: The foreground subject region is determined based on the portrait mask, wherein the foreground subject region is used to characterize the set of pixels in the portrait mask corresponding to the portrait whose confidence level is higher than a preset confidence threshold; Extract the depth value corresponding to the foreground subject region from the depth map, and calculate the average of all extracted depth values ​​to obtain the depth threshold; The depth constraint mask is obtained by binarizing the depth map according to the depth threshold.

6. The human portrait synthesis method as described in claim 2, characterized in that, The step of binarizing the depth map according to the depth threshold to obtain the depth constraint mask includes: For each pixel in the depth map, determine whether the depth value of the corresponding pixel is greater than or equal to the depth threshold. If the determination result is yes, then the corresponding pixel is marked with the first value; if the determination result is no, then the corresponding pixel is marked with the second value. After all pixels are marked, a binary image is obtained, which is then used as the depth constraint mask.

7. The human portrait synthesis method as described in claim 2, characterized in that, The process of determining the extended shadow mask based on the geometric prior of the human image includes: The human image region is determined based on the human image mask, wherein the human image region is used to characterize the overall spatial range of the human image; A preliminary shadow area is generated based on the geometric features of the portrait area and a preset scaling factor; The initial shadow region is sequentially subjected to dilation and blurring processes to obtain the extended shadow mask.

8. The method for synthesizing human images as described in any one of claims 1-6, characterized in that, The human image synthesis method also includes: Obtain the detail recovery region mask, wherein the detail recovery mask is the complement of the shadow generation region mask; The initial synthesized image is subjected to low-pass filtering to obtain low-frequency components; High-frequency texture features are extracted based on the initial synthesized image and the low-frequency component; The high-frequency texture features are superimposed onto the image to be synthesized based on the mask weight of the detail recovery region mask to obtain the target image to be synthesized, wherein the image to be synthesized includes the initial image to be synthesized.

9. The human portrait synthesis method as described in claim 4, characterized in that, The step of inputting the initial synthesized image, the shadow generation region mask, and the structure guidance signal into the generation network for iterative denoising to generate the initial light and shadow synthesized image includes: Controlled iterative denoising is performed in the latent space, and for each denoising step, the generator network is configured as follows: Based on the structural guidance signal, estimate the current latent variables for prediction; Obtain the noise latent variable that matches the current noise level from the initial synthesized image obtained through deterministic noise addition processing; Based on the shadow generation region mask, the predicted latent variable and the noisy latent variable are weighted and fused to obtain the current fused latent variable, which is used as the input for the next denoising step. The process is iterated until denoising is complete, and the final fused latent variable decoding yields the initial light and shadow composite image.

10. The human portrait synthesis method as described in claim 9, characterized in that, The step of weightedly fusing the predicted latent variable and the noisy latent variable according to the shadow generation region mask to obtain the current fused latent variable shadow generation region mask includes: Within the area defined by the shadow generation region mask, the value of the predicted latent variable is adopted; The value of the noisy latent variable is used in the region outside the area defined by the shadow generation region mask.

11. The method for synthesizing human images as described in any one of claims 9-10, characterized in that, The human image synthesis method also includes: Obtain the detail restoration region mask, wherein the detail restoration region mask is used to identify the region where the original portrait features need to be preserved; The step of estimating the current latent variable based on the structure guidance signal includes: predicting the latent variable based on the structure guidance signal and a spatial mask, wherein the spatial mask is constructed based on the shadow generation region mask and the detail recovery region mask, and is used to limit the feature interaction between the region defined by the shadow generation region mask and the region defined by the detail recovery region mask during the computation of the generator network.

12. The human portrait synthesis method as described in claim 1, characterized in that, The process of obtaining the initial synthesized image includes: Pixels from the green screen area are sampled from the original portrait image, and the average chromaticity of the sampled pixels is calculated as the background reference color. For each pixel in the original portrait image, calculate the color distance between the color vector of the corresponding pixel and the background reference color; The color distance is mapped to a preliminary transparency mask based on a start threshold, a stop threshold, and the following principles: if the color distance is less than or equal to the start threshold, the corresponding pixel is marked as completely transparent; if the color distance is greater than or equal to the stop threshold, it is marked as completely opaque; if the color distance is between the start threshold and the stop threshold, it is marked as semi-transparent by linear interpolation; wherein the start threshold and the stop threshold are determined based on the mean and variance of the chromaticity distribution on the original portrait image, and the start threshold is less than the stop threshold; The initial transparency mask is smoothed to obtain a green screen desaturation mask; The foreground content is extracted from the original portrait image based on the portrait mask, and the foreground content is merged with the green screen desaturation mask to generate a foreground layer with a transparency channel. The foreground layer is fused with the target background image to obtain the initial composite image.

13. The human portrait synthesis method as described in claim 8, characterized in that, When a target type image needs to be output, the portrait synthesis method further includes: The initial composite image is segmented into multiple local blocks with overlapping regions; Each local block is individually enhanced in detail and its resolution is improved. The processed local blocks are then seamlessly stitched together to obtain a high-definition composite image. Based on the weight of the detail recovery region mask, the high-frequency texture features are superimposed onto the corresponding region of the high-definition light and shadow composite image to obtain the target light and shadow composite image.

14. A human image synthesis device, characterized in that, The human image synthesis device includes: The initial composite image tiling processing module is configured to acquire an initial composite image, wherein the initial composite image is obtained by fusing the foreground of the original portrait image with the target background image; The depth map acquisition module is configured to perform depth estimation on the initial synthesized image to obtain a depth map, wherein the pixel values ​​in the depth map are used to characterize the depth information of each pixel point on the initial synthesized image; The shadow generation region mask acquisition module is configured to acquire a shadow generation region mask based on the depth map, wherein the shadow generation region mask is used to limit the pixel area where light and shadow are generated during the diffusion model iterative denoising process; The spatial guidance feature acquisition module is configured to encode the depth map into a structure guidance signal, wherein the structure guidance signal is used to control the geometry of the generated light and shadow; The light and shadow fusion image acquisition module is configured to input the initial synthesized image, the shadow generation region mask and the spatial guidance feature into the generation network for iterative denoising to generate the initial light and shadow fusion image; In the iterative denoising process, the latent variables predicted by the generating network and the latent variables obtained by adding noise to the initial synthesized image are weighted and fused according to the shadow generation region mask, so that light and shadow are generated within the area defined by the shadow generation region mask with the structure guidance signal as a constraint, and at least some of the original features of the initial synthesized image are retained in the area outside the area defined by the shadow generation region mask.

15. A computer-readable storage medium having a computer program stored thereon, said computer program being executed by a processor to implement the human image synthesis method as described in any one of claims 1-13.

16. An electronic device comprising a memory and a processor, wherein, The memory is used to store a computer program, and the processor can read the computer program from the memory and execute it to implement the human image synthesis method as described in any one of claims 1-13.

17. A computer program product comprising a computer program that, when executed by a processor, implements the human portrait synthesis method as described in any one of claims 1-13.

Images