A portrait scene enhancement method, apparatus, medium and electronic device

Abstract

Embodiments of the present application provide a portrait scene enhancement method, device, medium and electronic equipment, the portrait scene enhancement method comprises: obtaining an original portrait picture and a main subject mask; obtaining a background region mask according to the main subject mask, wherein the background region mask is used to identify a background region; adjusting the background region code according to at least region selection prompt information, obtaining a redraw region mask; obtaining the spatial geometric structure information of the scene in the original portrait picture, wherein the geometric structure information includes a depth map and / or a line feature map; inputting the spatial geometric structure information, the redraw region mask and semantic guidance information into a diffusion model, controlling the diffusion model to perform iterative denoising within the spatial range defined by the redraw region mask, and generating an enhanced image. Embodiments of the present application can improve the display effect of the portrait scene enhanced image.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, embodiments of this application relate to a method, apparatus, medium, and electronic device for enhancing portrait scenes. Background Technology

[0002] Human portrait scene reconstruction technology has become a tool for digital art creation, e-commerce display, virtual reality (VR / AR) and film and television post-production. This technology aims to use deep learning to perform structural reconstruction, stylistic transformation or reshaping of the background environment of the person in the image while preserving the core features of the person (such as facial details, body posture, and clothing texture).

[0003] Current portrait scene enhancement technologies mainly fall into two categories: **Segmentation and synthesis-based background replacement methods:** These methods extract a person mask and then overlay the person onto a new background image. The problems include: the need for pre-prepared target background images, inability to generate personalized background content, and a noticeable texture-like appearance in the synthesized result, making it difficult to achieve substantial changes in scene style. **Diffusion-based background repainting methods:** Diffusion models, such as Stable DiffusionInpainting, are widely used in image repainting tasks. However, existing diffusion models for scene enhancement suffer from several issues: **Personal features are easily eroded:** During denoising iterations in the latent space, due to the spatial diffusion of features, even with a protective mask, the model still modifies pixels related to the person when reconstructing the background, leading to lost hair, altered facial micro-expressions, or distorted finger structures, making it difficult to preserve the person's identity. Furthermore, when generating stylized backgrounds, diffusion models may alter the scene's perspective, causing ground tilt, horizon misalignment, and a spatial mismatch between the person and background, resulting in a floating effect.

[0004] In summary, existing portrait scene enhancement technologies mainly suffer from the following problems in application: human features are easily distorted or lost during the generation process; the perspective relationship and spatial structure of the generated background conflict significantly with the subject; and the control method for the background area is singular, failing to meet the differentiated needs of different users and thus failing to achieve fine-tuning of the generated background. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, medium, and electronic device for enhancing portrait scenes. This application's embodiments introduce both line feature maps and depth maps as dual geometric constraints to ensure that the new style background is aligned with the original scene in both two-dimensional contours and three-dimensional spatial levels, avoiding obvious conflicts between the perspective relationship and spatial structure of the generated background and the subject. A two-level masking mechanism is constructed: first, a static background baseline (i.e., a background region mask) is generated based on the subject's mask; then, based on region selection prompts (e.g., user-input prompts for redrawing regions or prompts for regions with partial feature retention), sub-regions that actually need to be redrawn are dynamically activated or activated differently, achieving precise background redrawing region selection control for different users and meeting differentiated needs. During the diffusion model iteration process, the extracted geometric constraint information of the original background (through depth maps and line feature maps) can better control the geometric realism of the target style background generated by the diffusion model. The redrawing region mask can better control the generation of the target style background in appropriate background areas, improving the realism and image effect of the target style background.

[0006] In a first aspect, embodiments of this application provide a portrait scene enhancement method, the method comprising: acquiring an original portrait image and a subject mask, wherein the subject mask is used to characterize the subject region of the portrait, the subject region including the region where the person is located and neighboring regions related to the person; obtaining a background region mask based on the subject mask, wherein the background region mask is used to identify the background region; adjusting the background region mask based at least on region selection prompt information to obtain a redraw region mask; acquiring spatial geometric structure information of the original scene in the original portrait image, wherein the spatial geometric structure information is obtained using a depth map and / or A line feature map is used to represent the two-dimensional geometric information of the scene, including object contours and / or edges. A depth map is used to represent the depth information of the original scene. The spatial geometric structure information, the redrawing region mask, and semantic guidance information are input into a diffusion model. The diffusion model is controlled to perform iterative denoising within the space defined by the redrawing region mask to generate an enhanced image. The spatial geometric structure information is used to constrain the diffusion model to generate a background that conforms to a preset geometric structure, and the semantic guidance information is used to guide the diffusion model to generate an enhanced scene with specified content and / or style.

[0007] The embodiments of this application introduce a two-level masking mechanism: a background region mask as a static semantic benchmark, and a redrawing region mask dynamically filtered and generated based on region selection prompts. This achieves semantically driven, precise control over the background redrawing region. For example, by using redrawing region selection prompts, only background regions related to user expectations can be activated for redrawing, avoiding unnecessary calculations and potential distortions. Alternatively, by using partial feature retention region selection prompts, a second feature retention region can be selected, preventing the region from being completely redrawn using new features obtained from the diffusion model. Simultaneously, by introducing depth maps and line feature maps as dual geometric constraints, the two-dimensional contours and three-dimensional spatial levels of the scene are jointly locked during the diffusion model iteration process. This ensures that the enhanced background remains consistent with the original scene in terms of object boundaries, perspective relationships, and depth levels, achieving targeted, precise semantically driven scene enhancement and improving the controllability and realism of the generated content.

[0008] In some embodiments, generating a background region mask based on the subject mask includes: using the complement of the subject mask as the background region mask; adjusting the background region mask at least according to region selection prompt information to obtain a redrawn region mask includes: in response to a redrawn region selection prompt word input by the user, identifying a specified background redrawn region indicated by the redrawn region selection prompt word in the original portrait image; representing the specified background redrawn region using a semantic activation mask; and performing a pixel-by-pixel logical AND operation on the semantic activation mask and the background region mask to obtain the redrawn region mask.

[0009] The embodiments of this application achieve accurate semantic control of the background region through a two-level process: basic background generation (i.e., background region mask) and semantic region refinement (i.e., input redrawing region prompts). First, a static candidate background baseline (i.e., background region mask) is established based on the complement of the main character mask, ensuring that all operations are strictly limited to the background area. Then, the background is semantically refined by using user-input region selection prompts (such as "only change the sky" or "only change the walls") to generate a redrawing background region mask that precisely matches the user's intent. This mechanism enables the system to activate only the semantic region specified by the redrawing region selection prompt as the background region for subsequent redrawing, while the remaining background regions remain unchanged. This avoids computational waste and potential distortion caused by indiscriminate redrawing, and achieves precise focusing from the entire background to specific semantic regions, providing a spatial basis for semantic alignment for subsequent generative redrawing.

[0010] In some embodiments, performing a pixel-by-pixel logical AND operation on the semantic activation mask and the background region mask to obtain the redrawn region includes: performing a pixel-by-pixel logical AND operation on the semantic activation mask and the background region mask to obtain a first background mask; and performing morphological correction on the first background mask to obtain the redrawn region mask, wherein the morphological correction includes performing a dilation operation.

[0011] The embodiments of this application introduce morphological correction when generating the redraw area mask. By performing an expansion operation on the relevant mask, the mask boundary is slightly expanded outward, thereby accurately covering the semi-transparent pixel areas such as hair strands and clothing fuzz on the edge of the character. This can be used to reduce the distortion of character features caused during the background redrawing process.

[0012] In some embodiments, generating a background region mask based on the subject mask includes: using the complement of the subject mask as the background region mask; adjusting the background region mask at least according to region selection prompts to obtain a redrawn region mask includes: determining a first feature-preserving region from the subject area of ​​the portrait, wherein the first feature-preserving region includes at least: pixel regions where the face and skin are located; determining a second feature-preserving region from the original portrait image in response to a partial feature-preserving region selection prompt input by the user, wherein the second feature-preserving region includes at least: pixel regions corresponding to clothing or handheld props; setting the pixel values ​​of the pixels corresponding to the first feature-preserving region on the background region mask to 0, and setting the grayscale values ​​of the pixels corresponding to the second feature-preserving region on the background region mask to values ​​greater than zero and less than 1 according to the region redrawing magnitude, to obtain an initial redrawn region mask; and performing morphological correction and / or blurring on the initial redrawn region mask to obtain the redrawn region mask.

[0013] The embodiments of this application achieve fine-grained control over the portrait scene enhancement process by constructing a hierarchical redraw weight map: First, regions containing sensitive features such as faces and skin are designated as mandatory protected areas (weight 0) to ensure zero distortion of the core identity features of the person during the diffusion model's redrawing of the image; second, partially adjustable regions such as clothing and handheld props are selected through partial feature retention regions, and a region redraw amplitude coefficient (adjustable from 0-1) is introduced, allowing users to flexibly control the degree to which these elements are redrawn (from complete retention to complete alteration), achieving precise adjustment of the redrawing region and redraw amplitude; the background region, excluding the second feature retention region, is completely redrawn (weight 1, where 1 is the original pixel value on the background region mask), providing ample space for stylized generation. Finally, the weight map is smoothed at the edges through erosion, dilation, and Gaussian blur to ensure a natural transition between the redrawn region and the retained region, avoiding harsh boundaries or a sense of discontinuity in the generated result.

[0014] In some embodiments, the semantic guidance information uses text prompts to describe the target style and content of the background to be generated; before inputting the spatial geometric structure information, the redrawing region mask, and the semantic guidance information into the diffusion model, the method further includes: loading LoRA weights pre-trained for the target style, and encoding the original portrait image into the latent space and adding noise according to a preset redrawing intensity coefficient to obtain initial latent variables, wherein the preset redrawing intensity coefficient is a fixed hyperparameter determined after pre-tuning for the target style, and the fixed hyperparameter is used to control the degree of modification of the original portrait image by the diffusion model during iterative denoising; inputting the spatial geometric structure information, the redrawing region mask, and the semantic guidance information into the diffusion model, and controlling the diffusion model to perform iterative denoising within the space defined by the redrawing region mask to generate an enhanced image, includes: inputting the depth map, the line feature map, the text prompts, and the redrawing region mask into the diffusion model, and during iterative denoising, updating the initial latent variables according to the weight values ​​of the redrawing region mask to obtain an initial enhanced image.

[0015] The embodiments of this application achieve a precise balance between overall style consistency and local region controllability through a three-layer collaborative mechanism of fixed redraw intensity coefficient, pre-trained LoRA weights, and weighted update of redraw region masks. The redraw intensity coefficient, as a fixed hyperparameter pre-tuned for the target style, controls the number of noise addition steps during latent variable initialization to ensure the degree of modification of the generated content relative to the original image, thus ensuring the style stability of the generated result. The LoRA weights pre-trained for a specific style serve as a plug-and-play style expert module, injecting precise style visual knowledge into the diffusion model, ensuring that the generated result strictly conforms to the target style requirements in terms of color, lighting, and texture. The redraw region mask performs pixel-level fine-grained control of the iterative denoising process through continuous weight values. For example, in some embodiments, the weight of the forced protection zone, such as facial skin, is 0 (completely unaffected by the diffusion generation result), the weight of the background region is 1 (completely generated by the diffusion model), and the weight of adjustable regions such as clothing and props is determined by the region redraw amplitude coefficient. The synergy of these three elements enables the diffusion model to generate an overall image according to the target style during the iterative denoising process, while strictly protecting the core features of the character and flexibly adjusting the redrawing degree of the specified area. Ultimately, this achieves high-quality scene enhancement with controllable style, feature protection, and flexible area handling.

[0016] In some embodiments, the depth map and the line feature map serve as geometric structure guides for the diffusion model, the text prompts serve as semantic content and style guides for the diffusion model, the redraw region mask is used to impose spatial constraints on the corresponding latent variables, the preset redraw intensity coefficient is used to control the noise reduction intensity of the diffusion model, and the magnitude of the redraw intensity coefficient is related to the type to which the target style belongs.

[0017] In some embodiments, obtaining the spatial geometric structure information of the scene in the original portrait image includes: inputting the original portrait image into a depth estimation model to obtain the depth map, wherein the depth estimation model includes at least one of DepthAnything, MiDaS, or DPT; and inputting the original portrait image into a line estimation model to obtain the line feature map, wherein the line feature estimation model includes at least one of Lineart, HED, or PIDNet.

[0018] In some embodiments, the diffusion model includes a control module, and the method further includes: concatenating the line feature map and the depth map in the channel dimension to generate a multi-channel control tensor; inputting the multi-channel control tensor into the control module of the diffusion model; extracting multi-scale features through the control module; and outputting a set of feature offsets corresponding to each intermediate layer of the main network of the diffusion model through a zero convolutional layer.

[0019] In some embodiments, the method further includes: selecting one of the following processing modes according to preset computing resources or quality requirements to obtain a high-definition enhanced image: performing pixel-level magnification on the initial enhanced image using a generative adversarial network model to obtain the high-definition enhanced image; or, performing block processing on the initial enhanced image using a diffusion magnification strategy based on tiling sampling, and performing a second diffusion iteration within each block to obtain a block magnified image; performing block stitching and edge smoothing processing on the block magnified image to obtain the high-definition enhanced image.

[0020] The embodiments of this application achieve a flexible balance between generation efficiency and image quality accuracy through a dual-mode high-definition refinement design: in fast mode, a GAN super-resolution model is used to achieve pixel-level magnification, meeting the real-time requirements of high-throughput scenarios; in quality mode, a diffusion magnification technique based on tiling sampling is used, which overcomes the limitations of video memory through block processing, and performs a secondary diffusion iteration within each block to compensate for texture details. Finally, weighted fusion of overlapping areas eliminates block boundary artifacts. This mechanism not only solves the problem of resolution limitation in direct output of diffusion models, but also meets the multi-level requirements from fast preview to commercial-grade output through differentiated processing paths, providing a high-quality base map with both high resolution and rich texture for subsequent detail restoration modules.

[0021] In some embodiments, the method further includes: obtaining a detail restoration region mask, wherein the detail restoration region mask includes a skin protection mask and a prop protection mask; performing frequency domain decomposition on the original portrait image to extract high-frequency texture features, wherein the high-frequency texture features include skin texture features and prop texture features in the original portrait image; and performing nonlinear superposition of the high-frequency texture features with the high-definition enhanced image according to the detail restoration region mask to obtain the target enhanced image.

[0022] The embodiments of this application restore the original texture of a person through a detail restoration module. First, frequency domain decomposition is used to extract high-frequency features containing microscopic details such as skin pores, hair texture, and clothing fibers from the original portrait image. Then, based on skin protection masks and prop protection masks, the pixel areas to be restored are located. Finally, non-linear superposition transfers the high-frequency features losslessly to the corresponding positions in the high-definition enhanced image. This mechanism effectively solves the common problems of redrawn and plastic-like appearances in generative redrawing and super-resolution upscaling, ensuring that the final enhancement result retains as much of the original optical shooting texture of the person as possible while obtaining a stylized background, achieving a unity of artistic enhancement and authenticity preservation.

[0023] In some embodiments, the portrait scene enhancement method further includes: detecting the amount of structural information of the background region in the line feature map and / or the depth map; when the amount of structural information of the background region is detected to be lower than a preset threshold, adaptively reducing the constraint strength of the spatial geometric structure information on the generation network.

[0024] The embodiments of this application introduce an adaptive constraint adjustment mechanism to achieve intelligent processing of backgrounds with varying complexity. For example, when a rich texture structure or spatial hierarchy is detected in the background region (where the structural information content of the background region is greater than a preset threshold), the strong constraint of the structural guidance signal is maintained to ensure that the new background follows the geometric skeleton of the original scene. When the background is blank or lacks effective structural information (where the structural information content of the background region is less than a preset threshold), the structural constraint strength of the depth map and line feature map on the diffusion model is automatically reduced, allowing the diffusion model to rely more on semantic guidance information (e.g., text prompts) for free creation. This mechanism enables the system to dynamically adjust the generation strategy according to the actual content of the input image, achieving a balance between accurate restoration and free expression. It avoids spatial perspective distortion in structural backgrounds while providing freedom for creative generation in blank backgrounds, thus improving the scene adaptability and robustness of the technical solution.

[0025] In some embodiments of this application, the diffusion model includes a control module and a generator network, wherein the generator network includes SDXL, Stable Diffusion 3, or a similar latent space diffusion model; the control module includes ControlNet or T2I-Adapter; and the adversarial network model includes at least one of Real-ESRGAN, SwinIR, or StableSR.

[0026] Secondly, some embodiments of this application provide a portrait scene enhancement device. The portrait synthesis device includes: an original data acquisition module configured to acquire an original portrait image and a subject mask, wherein the subject mask is used to characterize the main body region of the portrait, and the main body region includes the region where the person is located and the adjacent regions related to the person; a background region mask acquisition module configured to obtain a background region mask based on the subject mask; a redraw region mask acquisition module configured to adjust the background region mask at least according to region selection prompt information to obtain a redraw region mask; and a spatial geometric structure information acquisition module configured to acquire the spatial geometric structure information of the scene in the original portrait image, wherein the spatial geometric structure information is... The spatial geometric information is represented using depth maps and / or line feature maps. The line feature maps are used to represent the two-dimensional geometric information of the scene, and the two-dimensional geometric signals include object contours and / or edges. The depth maps are used to represent the depth information of the scene. The enhanced image processing module is configured to input the spatial geometric information, the redrawn region mask, and semantic guidance information into the diffusion model, and control the diffusion model to perform iterative denoising within the spatial range defined by the redrawn region mask to generate an enhanced image. The spatial geometric information is used to constrain the diffusion model to generate a background that conforms to a preset geometric structure, and the semantic guidance information is used to guide the diffusion model to generate an enhanced scene with specified content and / or style.

[0027] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the portrait scene enhancement method as described in any of the embodiments of the first aspect above.

[0028] Fourthly, some embodiments of this application provide an electronic device including a memory and a processor, wherein the memory is used to store a computer program, and the processor can read the computer program from the memory and execute it to implement the portrait scene enhancement method as described in any embodiment of the first aspect.

[0029] Fifthly, some embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the portrait scene enhancement method as described in any embodiment of the first aspect. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is an architecture diagram of the portrait scene enhancement system provided in the embodiments of this application.

[0032] Figure 2 This is one of the flowcharts for the portrait scene enhancement method in this application.

[0033] Figure 3 The second flowchart is a method for enhancing human portrait scenes provided in the embodiments of this application.

[0034] Figure 4 A block diagram illustrating the composition of the portrait scene enhancement device provided in this application embodiment.

[0035] Figure 5 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0036] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0037] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0038] Notes to relevant terms used in the embodiments of this application:

[0039] A subject mask is a mask corresponding to the original portrait image, used to identify the subject area of ​​the portrait image that needs to be protected. This mask divides the image pixels into an initial protected area and an initial background area, where the initial protected area corresponds to the subject and any attached objects, providing a protection boundary for subsequent processing.

[0040] A background region mask is a binary mask generated from a subject mask, used to identify candidate background regions in the original portrait image that are allowed for background reshaping. This mask serves as a static semantic benchmark, defining the candidate space range for background processing and providing foundational data for the subsequent dynamic generation of redraw region masks.

[0041] The redrawing region mask is a binary or grayscale mask generated after adjusting the background region mask based on the region selection prompt information. It is used to identify the target background region in the original portrait image that actually needs to be denoised iteratively by the diffusion model. This mask is a spatial hard constraint of the diffusion model, ensuring that the content generation process of the diffusion model is only carried out within the area specified by the redrawing region mask.

[0042] The detail restoration region mask refers to the binarized or grayscale mask generated based on the key region recognition results of the portrait. It is used to identify the regions in the original portrait image that need to restore the original high-frequency details, including at least the skin protection region and the prop protection region. It is used to guide the subsequent transfer of the original high-frequency texture features to the corresponding positions in the high-definition enhanced image.

[0043] The original portrait image refers to the original photograph data input into this system. This original photograph data is an RGB three-channel image containing the main subject and is the input data for the entire processing flow.

[0044] The initial enhanced image refers to the image generated and output by the diffusion model after iterative denoising within the spatial range defined by the redrawing region mask. The semantic-level reconstruction of the background has been completed, but the resolution may be limited.

[0045] High-resolution enhanced images refer to images output after performing resolution enhancement processing on the initial enhanced image. They can be obtained through fast mode (GAN super-resolution) or quality mode (tiling diffusion magnification), with the resolution increased to the target level and the background texture details enhanced.

[0046] Target-enhanced images refer to the final output image after performing detail restoration processing on high-definition enhanced images. While preserving the background texture of the high-definition enhanced image, the original skin texture, hair details, and texture of the props of the main character are restored through high-frequency feature transfer, which is the output result of the entire scene enhancement process.

[0047] A binary mask has pixel values ​​that take only two values: 0 or 1. A pixel value of 1 indicates that the pixel belongs to the area defined by the mask, while a pixel value of 0 indicates that the pixel does not belong to the area defined by the mask.

[0048] In some embodiments of this application, the redrawing region mask is a grayscale mask with pixel values ​​continuously distributed between 0 and 1. This mask is used to control the update degree of each pixel during the iterative denoising process of the diffusion model: a pixel value of 1 indicates that the region where the pixel is located is a fully redrawn region, completely replaced by the model-generated result after each denoising step; a pixel value of 0 indicates that the region where the pixel is located is a forced protected region, completely preserving the information corresponding to the original portrait image after each denoising step, and not participating in any updates; a pixel value between 0 and 1 indicates that the pixel is located at the transition boundary between the redrawn region and the protected region, and after denoising, the original information and the generated result are linearly mixed according to this weight ratio, thereby achieving a natural and smooth transition between the redrawn region and the preserved region, avoiding the sense of discontinuity or edge artifacts caused by overly rigid mask boundaries. This grayscale mask is obtained by sequentially performing morphological operations such as erosion, dilation, and Gaussian blur on the initial redrawing weight map to eliminate isolated noise points, fill holes, and feather edges.

[0049] The diffusion model includes a generator network and a control module. The generator network is the denoising network of the diffusion model, which can adopt a U-Net architecture (such as Stable Diffusion XL) or a Transformer architecture (such as DiT, PixArt-α) for iterative denoising in the latent space. The control module is used to encode external conditions (such as depth maps and line feature maps) into guiding features and inject them into the generator network. This control module can adopt ControlNet or its alternatives (such as IP-Adapter, training free editing methods, etc.). It should be noted that in the diffusion model of this application embodiment, the attention layer of its generator network is supplemented with a low-rank adaptive LoRA weight module obtained through fine-tuning. This LoRA module is connected in parallel with the original weight matrix in a bypass form. During the fine-tuning process, the original weights remain frozen, and only the low-rank matrix parameters of the LoRA module are updated. This LoRA module is obtained by fine-tuning training data containing specific style features and is used to guide the generator network to output the enhanced scene according to the user's desired target style during the iterative denoising process. The target style is described by text prompts and associated with the LoRA module. Different target styles correspond to different LoRA modules. Each module can be loaded independently or used in combination to achieve accurate control over the style of the generated scene.

[0050] Please refer to Figure 1 , Figure 1 The portrait scene enhancement system provided in some embodiments of this application includes: a depth estimation module 120, a line estimation module 110, an adaptive mask generation module 150, a high-frequency feature extraction module 160, a diffusion model 130, and a protected area feature replacement module 140.

[0051] Figure 1The input parameters include: the original portrait image 101 and the main figure mask 102.

[0052] For example, the original portrait image is the original photograph input into this system. The original photograph is an RGB three-channel image containing the main subject. This image will be used simultaneously for geometric feature extraction, mask generation, and subsequent iterative denoising processes.

[0053] For example, the main character mask 102 is a binary mask corresponding to the original portrait image. It is obtained through the matting model and is used to identify the position of all portrait pixels in the original portrait image, including areas such as face, skin, clothing, hair and personal props. This mask is the basis for the generation of subsequent related masks.

[0054] The region selection prompt 114 is the semantic description information of the user input adaptive mask generation module 150. The region selection prompt includes redraw region selection prompt or partial feature preservation region selection prompt. Through these prompts, the background region defined by the background region mask can be further adjusted to obtain the redraw region mask.

[0055] The text prompt word 104 is used to guide the diffusion model to generate a new background that conforms to the description style and content. For example, in some embodiments of this application, each text prompt word includes: style description (such as cyberpunk, ink painting, minimalism) and content elements (such as neon lights, beach, bookshelf), and each style corresponds to loading a fine-tuned LoRa model.

[0056] Figure 1 The core processing modules include:

[0057] An adaptive mask generation module 150 can generate a redraw area mask 106 and a detail restoration area mask 107 based on the main character mask 102 and the area selection prompt 114.

[0058] In some embodiments of this application, the redrawing region mask 106 is a grayscale image with a value range of 0-1, used to identify the degree to which each pixel in the image is redrawn. For example, in some embodiments of this application, the generation logic of the redrawing region mask is as follows: the complement of the main character mask is used as the background region mask; a mandatory protection region (face and skin) is determined from the main character mask; adjustable regions such as clothing and handheld props for prompt word recognition are selected based on the partially feature-preserved regions, and a region redrawing amplitude coefficient α or β (adjustable from 0-1) is assigned to the pixels in this region; the weight values ​​of the pixels corresponding to the mandatory protection region in the background region pixel mask are set to 0, and the weight values ​​of the pixels corresponding to the adjustable regions are set to values ​​greater than zero and less than 1 related to the redrawing amplitude coefficient, while the weight values ​​of the original pixels in other regions are retained to obtain an initial weight map; the generated initial weight map is subjected to erosion, dilation, and Gaussian blur processing to ensure that the mask edges transition smoothly and naturally to obtain the redrawing region mask.

[0059] In some embodiments of this application, the redraw region mask 106 is a binary image used to identify which background regions in the image need to be redrawn. For example, in some embodiments of this application, the generation logic of the redraw region mask is as follows: the complement of the main character mask is used as the background region mask; the background region to be redrawn is determined according to the redraw region selection prompt; the intersection of the background region mask and the mask of the background region to be redrawn is calculated, and the intersection result is subjected to erosion, dilation and Gaussian blur processing to obtain the redraw region mask. This mask is a binary mask, wherein the pixel weight of the region corresponding to the redraw region selection prompt is 1, and a complete redraw is subsequently performed through a diffusion model; the pixel weight of the remaining background regions is 0, and no background redraw is performed.

[0060] The detail restoration area mask 107 can be a binary mask used to identify the areas where high-frequency details in the original portrait image need to be restored, including at least the facial skin area and the area protected by props. This mask will guide the transfer of high-frequency features in the post-processing stage.

[0061] The diffusion model 130 includes a control module (ControlNet) 131 and a generative network 132.

[0062] The control module 131 receives line feature maps and depth maps as input, extracts multi-scale features through a pre-trained ControlNet structure, and outputs spatial guidance features corresponding to each intermediate layer of the generator network, which is part of the main network set included in the diffusion module. These feature offsets serve as structural guidance signals and are injected into the generator network during iterative denoising to constrain the geometric structure of the generated background, ensuring that the new background remains consistent with the original scene in terms of two-dimensional contour and three-dimensional depth.

[0063] The generative network 132 (e.g., a diffusion model U-Net) is the generative engine that receives the following inputs: spatially guided features (from ControlNet) to provide geometric constraints, textual prompts to provide semantic content and style guidance, and a redrawn region mask 106 to provide spatial update constraints. The generative network then generates an initial enhanced image 109 based on the inputs. For example, in some embodiments of this application, for LoRA weights (a plug-and-play add-on, not shown in the figure) pre-trained for the target style, during iterative denoising, the generative network updates the latent variables with weights based on the weight values ​​of the redrawn region mask: positions with weights of 0 retain the original information completely, positions with weights of 1 are entirely generated by the diffusion model, and positions with weights between 0 and 1 blend features from the original portrait image and features generated by the diffusion model according to their weight ratios, ultimately outputting the initial enhanced image 109.

[0064] The protected region feature replacement module 140 receives the initial enhanced image 109, the detail restoration region mask 107, and the high-frequency texture features 108 as input, and performs the following operations: performs frequency domain decomposition on the original portrait image to extract high-frequency texture features containing micro-details such as skin pores, hair texture, and clothing fibers; superimposes the high-frequency texture features with the initial enhanced image according to the detail restoration region mask 107 to restore the original texture of the face, skin, and prop areas; performs global color consistency calibration and edge sharpening on the fused image to finally output the target enhanced image 100.

[0065] Figure 1 The system output can include: an initial enhanced image 109, which is the output image after iterative denoising by the generative network. The semantic-level reconstruction and stylization of the background have been completed, but the resolution is limited, and the original texture of the human figure area may be lost during the generation process. Or, a target enhanced image 100, which retains the background texture of the initial enhanced image while restoring the original details of the human figure area through high-frequency feature transfer, achieving a unity between the stylized background and the original human figure.

[0066] The following is combined Figure 2 The present application provides exemplary methods for enhancing human portrait scenes through some embodiments.

[0067] like Figure 2 As shown in the embodiments of this application, the portrait scene enhancement method includes:

[0068] S110, obtain the original portrait image and the mask of the main figure.

[0069] It should be noted that, in the embodiments of this application, the main person mask is used to characterize the main image area, which includes the area where the person is located and the adjacent areas related to the person (e.g., the area where the person's clothes or props are located).

[0070] S120, obtain the background area mask based on the main character mask.

[0071] S130, adjust the background region code at least according to the region selection prompt information to obtain the redraw region mask.

[0072] S140, Obtain the spatial geometric structure information of the scene in the original portrait image.

[0073] It should be noted that the spatial geometric structure information in the embodiments of this application is characterized by depth maps and / or line feature maps. The line feature maps are used to characterize the two-dimensional geometric information of the original scene in the original portrait image. The two-dimensional geometric signals include object contours and / or edges. The depth maps are used to characterize the depth information of the original scene.

[0074] S150, the spatial geometric structure information, the redrawing region mask, and the semantic guidance information are input into the diffusion model, and the diffusion model is controlled to perform iterative denoising within the spatial range defined by the redrawing region mask to generate an enhanced image. The spatial geometric structure information is used to constrain the diffusion model to generate a background that conforms to a preset geometric structure, and the semantic guidance information is used to guide the diffusion model to generate an enhanced scene with specified content and style.

[0075] It should be noted that the embodiments of this application do not limit the execution order of the above steps. Those skilled in the art can adjust the execution order of the relevant steps according to actual needs.

[0076] It is easy to understand that the embodiments of this application achieve semantically driven precise control of background redrawing regions by introducing a background region mask as a static semantic benchmark and then dynamically filtering and generating redrawing region masks based on region selection information through a two-level mask mechanism. For example, by using redrawing region selection prompts, only background regions related to user expectations can be activated for redrawing, avoiding unnecessary calculations and potential distortions. Alternatively, by using partial feature retention region selection prompts, a second feature retention region can be selected, and the pixel values ​​of pixels in that region are obtained by fusing redrawing and some original features. At the same time, by introducing depth maps and line feature maps as dual geometric constraints, the two-dimensional contours and three-dimensional spatial levels of the scene are jointly locked during the diffusion model iteration process, ensuring that the enhanced background remains consistent with the original scene in terms of object boundaries, perspective relationships, and depth levels. This achieves targeted and precise semantically driven scene enhancement, improving the controllability and realism of the generated content.

[0077] The following example illustrates the process of obtaining the redraw region mask.

[0078] For example, in some embodiments of this application, the step of generating a background region mask based on the subject mask in step S120 includes: using the complement of the subject mask as the background region mask. The step of adjusting the background region mask at least based on region selection prompt information to obtain a redrawn region mask in step S130 includes: in response to a user-inputted redrawn region selection prompt, identifying a specified background redrawn region indicated by the region selection prompt in the original portrait image; representing the specified background redrawn region using a semantic activation mask; and performing a pixel-wise logical AND operation on the semantic activation mask and the background region mask at least to obtain the redrawn region mask. For example, in some embodiments of this application, the step of performing a pixel-wise logical AND operation on the semantic redrawn region mask and the background region mask to obtain the redrawn region includes: performing a pixel-wise logical AND operation on the semantic redrawn region mask and the background region mask to obtain a first background mask; and performing morphological correction on the first background mask to obtain the redrawn region mask, wherein the morphological correction includes performing a dilation operation. The embodiments of this application introduce morphological correction when generating the redraw area mask. By performing an expansion operation on the relevant mask, the mask boundary is slightly expanded outward, thereby accurately covering the semi-transparent pixel areas such as hair strands and clothing fuzz on the edge of the character. This can be used to reduce the distortion of character features caused during the background redrawing process.

[0079] It is easy to understand that the embodiments of this application achieve accurate semantic control of the background region through two-level processing: basic background generation and semantic region refinement. First, a static candidate background baseline is established based on the complement of the main character mask, ensuring that all operations are strictly limited to the background area. Then, the background is semantically refined by using the user-input redrawing region selection prompt (such as "only change the sky" or "only change the walls") to generate a redrawing background region mask that precisely matches the user's intent. This mechanism enables the system to activate only the semantic region specified by the region selection prompt as a candidate background region for subsequent redrawing, while the remaining background regions remain unchanged. This avoids computational waste and potential distortion caused by indiscriminate redrawing, and achieves precise focusing from the entire background to specific semantic regions, providing a spatial basis for semantic alignment for subsequent generative redrawing.

[0080] For example, in some embodiments of this application, S120, generating a background region mask based on the subject mask, includes: using the complement of the subject mask as the background region mask; S130, adjusting the background region mask at least according to region selection prompt information to obtain a redrawn region mask, includes: determining a first feature retention region from the subject area of ​​the portrait, wherein the first feature retention region includes at least: pixel regions where the face and skin are located; responding to a partial feature retention region selection prompt word input by the user, determining a second feature retention region from the original portrait image, wherein the second feature retention region includes at least: pixel regions corresponding to clothing or handheld props; setting the pixel values ​​of the pixels corresponding to the first feature retention region on the background region mask to 0, and setting the pixel values ​​of the pixels corresponding to the second feature retention region on the background region mask to a value greater than zero and less than 1 according to the region redraw amplitude coefficient, to obtain an initial redrawn region mask; and performing morphological correction and / or blurring processing on the initial redrawn region mask to obtain the redrawn region mask.

[0081] It is easy to understand that the embodiments of this application achieve accurate control over the portrait scene enhancement process by constructing a hierarchical redraw weight map: First, the face and skin areas are set as mandatory protected areas (weight 0) to ensure zero distortion of the core identity features of the person; second, adjustable areas such as clothing and handheld props are selected by selecting prompt words in the partially feature-preserved areas, and a region redraw amplitude coefficient (adjustable from 0-1) is introduced, allowing users to flexibly control the degree to which these elements are redrawn (from complete preservation to complete change), achieving precise adjustment of the redraw area and redraw amplitude; areas other than the first and second feature-preserved areas are set as completely redrawn (weight 1), providing ample space for stylized generation. Finally, the weight map is smoothed by erosion, dilation, and Gaussian blur to ensure a natural transition between the redrawn areas and the preserved areas, avoiding harsh boundaries or a sense of discontinuity in the generated results.

[0082] The following example illustrates the process of obtaining depth maps and line feature maps.

[0083] In some embodiments of this application, obtaining the spatial geometric structure information of the scene in the original portrait image includes: inputting the original portrait image into a depth estimation model to obtain the depth map, wherein the depth estimation model includes at least one of DepthAnything, MiDaS, or DPT; inputting the original portrait image into a line estimation model to obtain the line feature map, wherein the line feature estimation model includes at least one of Lineart, HED, or PIDNet.

[0084] The following exemplifies the architecture of the diffusion model and the data processing procedure for obtaining the enhanced image of the embodiment of this application using the diffusion model.

[0085] For example, in some embodiments of this application, the diffusion model includes a control module, and the method further includes: concatenating the line feature map and the depth map in the channel dimension to generate a multi-channel control tensor; inputting the multi-channel control tensor into the control module, extracting multi-scale features through the control module, and outputting a set of feature offsets corresponding to each intermediate layer of the main network of the diffusion model through a zero convolutional layer.

[0086] For example, in some embodiments of this application, the semantic guidance information uses text prompts to describe the target style and content of the background to be generated; before executing S150, which inputs the spatial geometric structure information, the redrawing region mask, and the semantic guidance information into the diffusion model, the method further includes: loading LoRA weights pre-trained for the target style, and encoding the original portrait image into the latent space and adding noise according to a preset redrawing intensity coefficient to obtain initial latent variables, wherein the preset redrawing intensity coefficient is a fixed hyperparameter determined after pre-tuning for the target style, and the fixed hyperparameter... The number is used to control the degree of modification of the original portrait image by the diffusion model during the iterative denoising process; correspondingly, S150 inputs the depth map, the line feature map, the redrawing region mask and the text prompt into the diffusion model, and controls the diffusion model to perform iterative denoising within the space defined by the redrawing region mask to generate an enhanced image, including: inputting the structure guidance signal, the text prompt and the redrawing region mask into the diffusion model, and during the iterative denoising process, updating the initial latent variables according to the weight value of the redrawing region mask to obtain an initial enhanced image.

[0087] It should be noted that, in some embodiments of this application, the spatial geometric structure information serves as the geometric structure guide for the diffusion model, the semantic guidance information serves as the semantic content guide for the diffusion model, the redrawing region mask is used to apply spatial constraints to the corresponding latent variables, the preset redrawing intensity coefficient is used to control the noise reduction intensity, and the magnitude of the redrawing intensity coefficient is style-related.

[0088] It is easy to understand that the embodiments of this application achieve a balance between overall style consistency and local region controllability through the collaborative design of a redraw intensity coefficient that matches the target style indicated by the text prompts, pre-trained LoRA weights, and a dynamically redrawn region mask. The redraw intensity coefficient, as a fixed hyperparameter pre-tuned for the target style, ensures stability in the overall modification range during the generation process, avoiding style shifts caused by parameter fluctuations. The pre-trained LoRA weights, as a plug-and-play style expert module, inject specific style visual knowledge into the diffusion model, ensuring that the generated results meet the target style requirements in terms of color, lighting, and texture. The redrawn region mask, on the other hand, finely controls the update degree of each pixel through continuous weight values. This allows the diffusion model to generate the entire image according to the target style during iterative denoising while strictly protecting the core features of the character, and flexibly adjusting the redrawing degree of areas such as clothing and props. Ultimately, this achieves a high-quality scene enhancement effect with controllable overall style, protected local features, and natural edge transitions.

[0089] LoRA is a parameter-efficient fine-tuning technique that learns incremental style-specific changes by introducing a small number of trainable parameters through low-rank decomposition without modifying the original weights of the pre-trained model. Specifically, for a fixed weight matrix in the model, its update is decomposed into the product of two low-rank matrices. During fine-tuning, the original weights (i.e., the weight coefficients of the original diffusion model) remain frozen, and only these two small matrices are updated. In this way, LoRA enables the base model to generate style-specific visual features, such as the various target styles required in this application, with minimal parameter count and training cost.

[0090] LoRA's training process is consistent with the basic diffusion model, aiming to predict and remove noise from noisy latent variables. Training requires collecting image datasets of specific styles, such as cyberpunk, ink painting, or minimalist Nordic portrait scenes, typically ranging from tens to hundreds of images. Each image needs to be labeled with a text description, including a fixed style description as well as descriptions of the image's elements and composition, so that the model can establish a correspondence between style semantics and visual representation. During training, only the matrix introduced by LoRA participates in gradient updates; the basic model parameters are completely frozen, and the loss function uses the commonly used mean squared error. Training hyperparameters such as learning rate, number of training epochs, and batch size can be adjusted according to the specific task.

[0091] In diffusion models like SDXL, the LoRA module is primarily inserted into the attention layers of UNet, including the various linear projection layers in self-attention and cross-attention. During inference, there are two weight merging methods: real-time stacking, which dynamically calculates the sum of the original weights and the LoRA weights, supporting online adjustment of style intensity; and fusion merging, which physically merges the LoRA weights into the original model, resulting in faster inference but sacrificing adjustment flexibility. In the real-time stacking method, users can control the degree of LoRA's influence on the generated results through scaling factors.

[0092] In the portrait scene enhancement method of this application, LoRA is used for style tuning, enabling the diffusion model to generate backgrounds according to the user's desired target style. After obtaining pre-trained LoRA weights for specific styles such as cyberpunk, sunset beach, or minimalist Nordic style through the above training process, these weights are loaded during the iterative denoising process of the diffusion model. Simultaneously, a preset redraw intensity coefficient controls the overall degree of modification, and both work together to ensure that the generated background matches the user's expectations in terms of style attributes. LoRA's plug-and-play nature allows it to be stored as an independent file and dynamically loaded as needed. Different styles correspond to different LoRA weights, facilitating user switching and combination.

[0093] The following example illustrates the process of magnifying the obtained initial enhanced image.

[0094] For example, in some embodiments of this application, the method further includes: selecting one of the following processing modes according to preset computing power resources or quality requirements to obtain a high-definition enhanced image: performing pixel-level magnification on the initial enhanced image through a generative adversarial network model to obtain the high-definition enhanced image; or, performing block processing on the initial enhanced image using a diffusion magnification strategy based on tiling sampling, and performing a second diffusion iteration within each block to obtain a block magnified image; performing block stitching and edge smoothing processing on the block magnified image to obtain the high-definition enhanced image.

[0095] The embodiments of this application achieve a flexible balance between generation efficiency and image quality accuracy through a dual-mode high-definition refinement design: in fast mode, a GAN super-resolution model is used to achieve pixel-level magnification, meeting the real-time requirements of high-throughput scenarios; in quality mode, a diffusion magnification technique based on tiling sampling is used, which overcomes the limitations of video memory through block processing, and performs a secondary diffusion iteration within each block to compensate for texture details. Finally, weighted fusion of overlapping areas eliminates block boundary artifacts. This mechanism not only solves the problem of resolution limitation in direct output of diffusion models, but also meets the multi-level requirements from fast preview to commercial-grade output through differentiated processing paths, providing a high-quality base map with both high resolution and rich texture for subsequent detail restoration modules.

[0096] The following example illustrates the process of restoring high-frequency details in high-definition enhanced images.

[0097] For example, in some embodiments of this application, the method further includes: obtaining a detail restoration region mask, wherein the detail restoration region mask includes a skin protection mask and a prop protection mask; performing frequency domain decomposition on the original portrait image to extract high-frequency texture features, wherein the high-frequency texture features include skin texture features and prop texture features in the original portrait image; and superimposing the high-frequency texture features with the high-definition enhanced image according to the detail restoration region mask to obtain the target enhanced image.

[0098] The embodiments of this application restore the original texture of a person through a detail restoration module. First, frequency domain decomposition is used to extract high-frequency features containing microscopic details such as skin pores, hair texture, and clothing fibers from the original portrait image. Then, key areas requiring restoration are located based on skin protection masks and prop masks. Finally, non-linear superposition transfers the high-frequency features losslessly to the corresponding positions in the high-definition enhanced image. This mechanism effectively solves the common problems of redrawn and plastic-like appearances in generative redrawing and super-resolution upscaling, ensuring that the final enhancement result retains as much of the original optical quality of the person as possible while obtaining a stylized background, achieving a unity of artistic enhancement and authenticity preservation.

[0099] It should be noted that, in some embodiments of this application, the portrait scene enhancement method further includes: detecting the amount of structural information in the background region of the line feature map and / or depth map; when the amount of structural information in the background region is detected to be lower than a preset threshold, adaptively reducing the constraint strength of the spatial geometric structure information on the generation network. It is easy to understand that the embodiments of this application achieve intelligent processing of backgrounds with different complexities by introducing an adaptive constraint adjustment mechanism. For example, when a rich texture structure or spatial hierarchy is detected in the background region, strong constraints on the structural guidance signal are maintained to ensure that the new background follows the geometric skeleton of the original scene; when the background is blank or lacks effective structural information, the structural constraint strength is automatically reduced, allowing the diffusion model to rely more on text prompts and other elements for free creation. This mechanism enables the system to dynamically adjust the generation strategy according to the actual content of the input image, achieving a balance between accurate restoration and free expression. It avoids spatial perspective distortion in structural backgrounds while providing freedom for creative generation in blank backgrounds, improving the scene adaptability and robustness of the technical solution.

[0100] In some embodiments of this application, the diffusion model includes a control module and a generator network, wherein the generator network includes SDXL, Stable Diffusion 3, or a similar latent space diffusion model; the control module includes ControlNet or T2I-Adapter; and the super-resolution model includes at least one of Real-ESRGAN, SwinIR, or StableSR.

[0101] The following is combined Figure 3 This application provides an exemplary embodiment of a portrait scene enhancement method, which includes:

[0102] S1: Multi-source mask generation and semantic benchmark alignment step. This step uses multiple semantic segmentation models to identify key regions to obtain the adaptive mask of this embodiment. The system extracts the main subject mask and the redrawing region mask for key background regions using a segmentation algorithm. The system performs tensor alignment of the multi-source masks to ensure clear boundaries between the generated region and the protected subject region, providing a spatial benchmark for subsequent accurate local redrawing by generative AI.

[0103] In some embodiments of this application, step S1, the human image perception and preprocessing includes:

[0104] S11: Obtain the original portrait image to be processed via a standardized interface. and the corresponding main character mask The main subject mask is a single-channel grayscale image, with a grayscale value of 255 representing the main subject and the area close to it, and a grayscale value of 0 representing the initial background.

[0105] S12: Regarding the original human portrait image Perform color space verification and pixel normalization to map pixel values ​​to... For continuous intervals, generate standardized input tensors.

[0106] For example, in some embodiments of this application, the subject mask is a single-channel grayscale image. After binarization, the grayscale value of 255 can represent the pixel area where the subject is located, and the grayscale value of 0 can represent the background area. It is easy to understand that binarization is performed to represent the specific location of the person in the image.

[0107] S2: The Physical Space Feature Perception and Multidimensional Structure Extraction Block is responsible for capturing the geometric structure and texture features of the original scene in the original portrait image. It obtains line feature maps that reflect the contour distribution of the image by deploying a line extraction model (Lineart), and estimates the pixel-level depth map of the image using a depth perception model (DepthAnything2) to lock the spatial perspective relationship of the original scene, ensuring that the reconstructed scene maintains logical consistency with the subject in terms of physical structure.

[0108] In some embodiments of this application, step S2, specifically includes the multidimensional structure prior estimation:

[0109] S21: Enable the line estimation model to extract line features from the original portrait image to lock the spatial topology of background details and ensure the geometric consistency of background objects during the enhancement process.

[0110] S22: Enable the depth estimation model to obtain a continuous depth map of the original scene. By perceiving the spatial layout prior of the scene's near and far relationships, it provides a spatial layout prior for the enhancement algorithm.

[0111] S23: Align the line features and depth features at the feature level to construct a multi-dimensional spatial constraint operator to prevent geometric distortion during background reset.

[0112] S3: Local reshaping and lighting reconstruction based on the generative diffusion model inputs the multi-source mask generated by S1 and the structural control signal generated by S2 into the diffusion model (SDXL). Style shift control is performed through a specific Lora model (i.e., a Lora model trained for this specific scene style), and the depth and lines are constrained by the ControlNet module.

[0113] It should be noted that a specific LoRa model refers to a LoRa model trained specifically for a particular scene and style. The difference between this and other LoRa models lies in the training dataset and parameter settings; that is, the dataset is used to control the LoRa's ability to generate images in a specific style. For example, in some embodiments of this application, the training process requires collecting a dataset of portraits in a specific style. This dataset consists of approximately 50-100 images obtained by designers through retouching based on a theme style, meeting the corresponding style's quality requirements. Each image is then labeled with prompts, including descriptions of the fixed style and descriptions of the image's content elements and composition. The training process uses a set of LoRa training parameters based on the SDXL base model, including settings for LoRa's low-rank dimension, total number of training epochs, learning rate, and other hyperparameters.

[0114] The following example illustrates the annotation content for an image:

[0115] Fixed style descriptions, such as "cyberpunk style," "ink painting style," and "minimalist Nordic style," are used to activate the corresponding LoRA style during inference.

[0116] Content element descriptions: such as "skyscrapers," "neon lights," "rainy night," and "bamboo forest," are used to define the specific objects that should appear in the image.

[0117] Composition description: such as "wide-angle lens", "overhead angle", "close-up", used to define the visual presentation of the image.

[0118] It is easy to understand that the embodiments of this application train the LoRA model using training data obtained through such annotation, enabling it to learn the mapping relationship between text description and visual presentation under a specific style, thus providing stylized generation capabilities for the subsequent inference stage.

[0119] The ControlNet module provides deterministic spatial priors for the diffusion model. In the scene enhancement of this application embodiment, stronger conditional constraints are provided on the generation process through the collaboration of dual channels (depth map and line feature map).

[0120] Step 1: Preprocessing stage

[0121] The system performs asymmetric feature extraction on the original human image:

[0122] 1. Depth Branch: The DepthAnything2 model is used to estimate the depth of the original portrait image, generating a distance field that reflects the distance relationship between objects. This yields a depth map.

[0123] 2. Line Branching: Extract the outline lines of the original portrait image using the Lineart operator. Preserve the geometric edges of buildings and objects in the background to obtain a line feature map.

[0124] Step 2: Control Feature Encoding

[0125] Input: Preprocessed data (Depth map) and (Line feature diagram).

[0126] Operation: Two corresponding ControlNet pre-trained models are used to process the depth map and line feature map, respectively. The features obtained through the ControlNet network are then injected into the main diffusion model of SDXL through weight parameters, which plays a role in the noise reduction process of the diffusion model.

[0127] In some embodiments of this application, the output of ControlNet is used to achieve: geometric position locking, where the output feature vector of ControlNet provides conditional control in the U-Net of the diffusion model to fix the position of the generated content; lighting logic guidance, where the diffusion model maintains spatial consistency between the generated image and the original photo based on depth information; and line feature maps to ensure pixel-level alignment of the edges of the redrawn people and objects, ensuring no pixel displacement with the original photo.

[0128] It is easy to understand that the embodiments of this application only perform denoising iteration on the reshaped area defined by the redrawing area mask, and achieve structural reconstruction and lighting reshaping of the background environment while protecting the integrity of the character features (is this achieved through the character subject mask?).

[0129] For example, in some embodiments of this application, step S3 specifically includes:

[0130] S31: Load the corresponding diffusion model base weights, local fine-tuning weights (i.e., Lora model weights corresponding to this style) and control module according to the preset style.

[0131] For example, in an embodiment of this application, if the preset style is cyber style, then the embodiment of this application needs to train the LoRA model in advance using cyber style images (to obtain local fine-tuning weights) to obtain local fine-tuning weights. Then, when it is necessary to generate this style for the original portrait image, the local fine-tuning weights and the basic weights of the diffusion model can be directly loaded for generation.

[0132] S32: Based on adaptive mask calculation logic, combined with the main character mask. Generate a redraw region mask with the region selection prompt information. .

[0133] S33: Mask of the redrawn area Iterative sampling and noise reduction are performed within the unprotected area of ​​the constraint (i.e., the area where the weight value or pixel value of a pixel in the redraw area mask is non-zero) to achieve the reshaping of the background environment's light and shadow and the enhancement of its texture.

[0134] S4: Cross-scale super-resolution enhancement yields high-definition enhanced images. This step addresses the resolution limitations of the generative model's output by performing cross-scale enhancement using Tile magnification techniques or super-resolution networks. By slicing the redrawn region and introducing noise for redraw enhancement, the system not only improves the physical resolution of the image but also achieves secondary detail optimization of the background texture, ensuring that the final reconstructed scene possesses commercial-grade high-definition accuracy.

[0135] For example, in some embodiments of this application, step S4 specifically includes:

[0136] S41: Obtain the initial enhanced image and select the refinement mode according to the preset computing power resource parameters: the fast mode uses a generative adversarial network to enlarge pixels, and the quality mode uses a diffusion amplification technique based on tiling sampling to compensate for details.

[0137] S42: Perform block-based parallel processing on the initial enhanced image, and while maintaining global lighting harmony, increase the image resolution to 4K or higher to generate an ultra-high-definition enhanced image.

[0138] S5: High and low frequency detail restoration and global color alignment to obtain the target enhanced image.

[0139] For example, in some embodiments of this application, the detail recovery and feature protection described in step S5 specifically includes:

[0140] S51: Use semantic region segmentation technology to identify and extract the key region mask of the portrait from the original portrait image (i.e., the detail recovery region has a mask). For example, the detail recovery region mask includes a skin protection mask. and complex personal protective equipment mask .

[0141] S52: Based on the skin protection mask and complex personal protective equipment mask

[0142] The set dynamically weights and fuses the high-definition enhanced image output from S4 with the original high-frequency features retained in S11.

[0143] S53: Adjust the blending coefficient according to the protection requirements of different semantic regions to restore skin texture, hair edges and details of personal props, and finally output commercial-grade enhanced result image.

[0144] The following sections will elaborate on S1, S2, S3, and S4.

[0145] In some embodiments of this application, step S1 may introduce a multi-source mask generation and semantic benchmark alignment module. This module performs pixel-level semantic parsing and region segmentation on the original portrait image by deploying multiple segmentation models in parallel, in order to obtain the spatial logical boundary between the protected foreground subject and the background region to be reshaped, that is, to obtain the redrawing region mask.

[0146] For example, this step includes the following sub-steps:

[0147] S11: The system receives the raw human portrait data. The system invokes a pre-trained portrait segmentation operator to perform semantic extraction. It generates a binarized mask of the main subject by recognizing anatomical features and visually salient objects carried in the original portrait image. The subject mask divides the image pixels into deterministic protected regions (weight 1) and unprotected regions (weight 0), and its generation process can be represented as follows:

[0148]

[0149] in, This represents a segmentation deep neural network optimized for human portraits.

[0150] S12: The system uses the main character mask. Automatically derive background region mask from complement relationship (i.e., background area mask) The system then refines the region by combining the user-inputted reshaping command (which carries region selection prompts). If the user specifies that only a specific background region (such as the ground, sky, or specific background props, as an example of a redraw region selection prompt) should be reshaped, the system will use a semantic segmentation model to perform secondary classification of the background and extract the accurate reshaping guide mask from the background region mask through logical AND operations.

[0151] S13: System mask for generated main character and Perform morphological corrections and tensor alignment checks. Through... Perform dilation operations with a small radius (e.g., 3-5 pixels). Obtain the redraw area mask Ensure that the redrawn area slightly covers the semi-transparent pixels at the edge of the character to prevent background breaks or white edges after generative redrawing:

[0152]

[0153] in, The structural element is circular. This step establishes a unified spatial coordinate mapping benchmark, providing strict spatial constraints for the subsequent accurate local redrawing of the diffusion model, ensuring that the reshaping process does not infringe on the core features of the main character.

[0154] In some embodiments of this application, step S2 may introduce a physical space feature perception and multi-dimensional structure extraction module. This module is used to capture the geometric contours and depth topological features of the original scene, providing physical constraint priors for the subsequent generative reshaping of the diffusion model, ensuring that the reshaped new background maintains logical consistency with the subject in spatial perspective. For example, this step includes the following sub-steps:

[0155] S21: The system will process the original portrait image. The input is a pre-trained line estimation module, which uses the Lineart contour extraction operator to capture the brightness gradient and edge response in the original scene, generating a line feature map that reflects the physical structure of the environment. :

[0156]

[0157] This line feature map is used to determine the outlines of objects, building lines, and ground orientation in the original scene, ensuring that the reconstructed scene can continue the compositional logic and geometric framework of the original image and avoid perspective misalignment.

[0158] S22: The system synchronously calls the depth estimation module, which adopts a depth perception model based on DepthAnything2. This model uses monocular depth estimation technology to calculate the relative distance of each pixel in the original portrait image and generate a pixel-level depth map. This depth-sensing model is trained using large-scale synthetic data aligned with real data, exhibiting strong generalization ability and capable of outputting fine-grained depth gradient distributions.

[0159]

[0160] This depth map serves as a spatial reference for subsequent lighting and shadow simulations, ensuring that newly generated background light sources can produce natural occlusion and attenuation effects based on object distance.

[0161] S23: The system extracts the line feature map. With depth map Feature alignment and regularization are performed. The system scales the line feature map and depth map to the size required by the diffusion model and performs normalization mapping, transforming them into control signal tensors that can be directly read by the ControlNet module. This step provides a reliable physical space reference for the virtual-real fusion reshaping task through the cooperative locking of multi-dimensional structures.

[0162] In some embodiments of this application, step S3 may introduce a local reshaping and lighting reconstruction module based on a diffusion model. This module guides the injection of multi-dimensional control signals and redrawing region masks extracted in the previous stage into the diffusion model framework. While locking the main features of the character, it performs generative reconstruction of the background to achieve depth transformation of environmental colors and lighting. For example, this step S3 includes the following sub-steps:

[0163] S31: The system loads the pre-trained diffusion model (SDXL) and Lora weights fine-tuned for specific scene styles. The system then uses the redrawn region mask generated in step S1. As a hard constraint on the noise reduction area, and connected to the line feature map provided in step S2. With depth map As a conditional input tensor of ControlNet. This is achieved by establishing a conditional guided distribution. It provides control over the geometry and perspective relationships of new scenes. Specifically, in the iterative denoising diffusion process, each iteration of the diffusion model references feature injection from ControlNet for denoising. The two ControlNets essentially control the intensity of their respective impacts on denoising through weights.

[0164] S32: The system performs iterative local denoising based on the redrawn region mask. Guided by the input text prompt, the diffusion model redraws the region mask. Within a defined spatial range, the inverse diffusion operation is performed starting from Gaussian noise. During this process, the system adjusts the redrawing intensity coefficient. While maintaining the basic compositional logic of the original scene, a new background with specific style textures and lighting attributes is generated. This generation process can be expressed by the following formula:

[0165]

[0166] in, Noise predicted by deep neural networks It is a comprehensive control vector that includes text semantics, lines, and depth.

[0167] S33: The system performs consistency calibration for lighting and shadow reshaping. During the generation of a new scene, the model will adjust the depth map accordingly. The provided depth information automatically simulates the diffuse reflection and occlusion shadows of the new light source on the object's surface. By calculating the mutual attention weights between the new background and the edges of the protected subject, the system induces a diffusion model to generate edge halos that match the lighting logic of the new scene, thus initially resolving the incongruity of combining reality and virtuality during the generation stage.

[0168] In some embodiments of this application, step S4 may introduce a cross-scale super-resolution enhancement and detail restoration module. This module is used to overcome the inherent limitations of the diffusion model in output resolution (such as the 1024-level pixel limit), and utilizes block enhancement and noise compensation techniques to upscale the reconstructed image to commercial-grade high-definition resolution, while achieving in-depth refinement of background textures. For example, this step S4 includes the following sub-steps:

[0169] S41: The system uses a Tile slicing algorithm to divide the generated image output from step S3 into multiple sub-block tensors with overlapping regions. For each sub-block, the system calls a super-resolution convolutional neural network (such as Real-ESRGAN) or an amplification operator integrated into a diffusion framework to perform spatial scale expansion. Through this block-based computation mechanism, the system can effectively avoid the risk of memory overflow when processing ultra-large images, increasing the physical image size to the 4K or 8K level.

[0170] S42: The system performs local redraw enhancement (Tile Diffusion) to restore detailed textures. After image magnification, interpolation or simple upsampling can cause texture blurring; therefore, the system sets a lower noise reduction intensity coefficient. A second diffusion iteration is performed on the sub-blocks of the background region. By introducing a small amount of noise and combining it with the original high-definition semantic information, the system can compensate for the missing subtle textures (such as stone textures, plant veins, or halo grains) in the generated background. The detail restoration process can be described by the following formula:

[0171]

[0172] in, For super-resolution operators, This represents the amount of detail compensation generated through low-intensity redrawing.

[0173] S43: The system performs tensor reconstruction and stitching smoothing on all sub-blocks. A weighted average mixing operator is used to process overlapping edges between sub-blocks, eliminating physical seams that may arise from block enhancement. Finally, the system outputs a high-resolution reconstructed base map (i.e., a high-resolution enhanced map) with 4K / 8K resolution at the macroscopic scale and rich texture features at the microscopic scale.

[0174] In some embodiments of this application, step S5 may introduce a multi-level layer blending and global color alignment module. This module is used to physically stitch the high-definition reshaped background generated in step S4 with the original high-definition subject protected in step S1, and to eliminate visual discrepancies between the real and virtual layers through color space conversion technology. For example, this step S5 includes the following sub-steps:

[0175] S51: The system performs high-precision layer compositing based on the alpha channel. The subject mask generated in step S1, with edge optimization, is then used. The original high-definition portrait subject Overlay to the high-definition reshaped background generated by S4 Above. To eliminate hard cut marks at the edges, the system performs a Gaussian pyramid fusion operator at the mask boundaries to achieve a smooth sub-pixel transition of edge pixels.

[0176] S52: The system enables a global color alignment algorithm. Because the background light source distribution, color temperature, and saturation have changed after reshaping, the system needs to perform tone remapping on the original portrait layer. This is achieved by extracting the histogram statistical features (mean) of the portrait area in the reshaping background. and standard deviation A linear transformation is performed on the RGB channels of the portrait area to align its lighting and tonal values ​​with the new environment. The transformation logic can be described as follows:

[0177]

[0178] This step ensures that the light and shadow tones of the subject are highly consistent with the color space of the reconstructed scene, eliminating the sense of separation after compositing.

[0179] This formula allows you to calculate the pixel value of the portrait area in the new result by subtracting the mean from the pixel value of the portrait area in the collage result image, dividing by the standard deviation, and then multiplying by the standard deviation of the pixel value of the portrait area in the high-resolution result image plus the corresponding mean. Essentially, it pulls the average tone of the pasted portrait area back into the range of the generated result while preserving the texture.

[0180] S53: The system performs final global image refinement and noise reduction processing. This is done by analyzing the fused complete image. Minor sharpening compensation and statistical noise reduction are performed to eliminate the inconsistency in granularity introduced by different model processing paths. Ultimately, the system outputs a high-resolution image layer for reconstructing portrait scenes with 4K resolution, consistent lighting and shadow logic, and zero distortion of human features.

[0181] In other words, in some embodiments of this application, S5 includes the step of performing layer fusion and color alignment on the main subject of the person in the high-definition enhanced image and the original portrait image, the step including:

[0182] The first step is to obtain the subject mask after edge optimization. That is, to perform edge optimization processing on the subject mask to obtain the target mask.

[0183] The second step involves using the target person mask to overlay the main subject of the original portrait image onto the high-definition enhanced image to obtain a first fused image. Gaussian pyramid fusion is then performed on the area corresponding to the boundary of the target person mask in the initial fused image to obtain a second fused image, achieving a sub-pixel level smooth transition of edge pixels.

[0184] The third step is to extract the histogram statistical features of the human image region from the high-definition enhanced image, wherein the histogram statistical features of the human image region include the mean. and standard deviation .

[0185] Fourth step: Based on the mean and standard deviation, perform the following linear transformation on the RGB channels of the main subject in the second fused image to obtain the third fused image, aligning its lighting and tonal characteristics with the high-definition enhanced image. The transformation formula is:

[0186]

[0187] in,

[0188] The fifth step is to perform sharpening compensation and statistical noise reduction on the complete image of the third fused image to obtain the target enhancement image, which is a target enhancement image with consistent lighting and shadow logic and zero distortion of human features.

[0189] like Figure 4 As shown, some embodiments of this application provide a portrait scene enhancement device. It should be understood that this device is similar to the one described above. Figure 2 or Figure 3 Corresponding to the method embodiments, it can execute the various steps involved in the above method embodiments. The specific functions of the device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in the memory or embedded in the device's operating system in the form of software or firmware. This portrait scene enhancement device includes: a raw data acquisition module 310, a background area mask acquisition module 320, a redrawing area mask acquisition module 330, a spatial geometric structure information acquisition module 340, and an enhanced image processing module 350.

[0190] The raw data acquisition module is configured to acquire the raw portrait image and the main person mask, wherein the main person mask is a binary mask, which is used to represent the main body region of the portrait, and the main body region of the portrait includes the area where the person is located and the neighboring areas related to the person.

[0191] The background region mask acquisition module is configured to obtain a background region mask based on the main character mask, wherein the background region mask is used to identify the background region.

[0192] The redrawing region mask acquisition module is configured to adjust the background region code based at least on the region selection prompt information to obtain a redrawing region mask, wherein the redrawing region mask is used to represent the target background redrawing region determined based on the semantic guidance information.

[0193] The spatial geometric structure information acquisition module is configured to acquire the spatial geometric structure information of the scene in the original portrait image, wherein the spatial geometric structure information is represented by a depth map and / or a line feature map, the line feature map is used to represent the two-dimensional geometric information of the scene, the two-dimensional geometric signal includes object contours and / or edges, and the depth map is used to represent the depth information of the scene.

[0194] An enhanced image processing module is configured to input the spatial geometric structure information, the redrawn region mask, and the semantic guidance information into a diffusion model, and control the diffusion model to perform iterative denoising within the spatial range defined by the redrawn region mask to generate an enhanced image. The spatial geometric structure information is used to constrain the diffusion model to generate a background that conforms to a preset geometric structure, and the semantic guidance information is used to guide the diffusion model to generate an enhanced scene with specified content and style.

[0195] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0196] Some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the portrait scene enhancement method as described in any of the embodiments of the first aspect above.

[0197] like Figure 5 As shown, some embodiments of this application provide an electronic device 400, which includes, for example, a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420. When the processor 420 reads the program through a bus 430 and executes the computer program, it can implement the portrait scene enhancement method as described in the above embodiments.

[0198] Processor 420 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 420 may be a microprocessor.

[0199] Memory 410 can be used to store instructions executed by processor 420 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 420 of the embodiments of this disclosure can be used to execute the instructions in memory 410 to implement… Figure 2 or Figure 3 The method shown. Memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0201] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0202] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0204] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0205] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for enhancing portrait scenes, characterized in that, The portrait scene enhancement method includes: Obtain the original portrait image and the main person mask, wherein the main person mask is used to represent the main body region of the portrait, and the main body region of the portrait includes the area where the person is located and the adjacent areas related to the person; A background region mask is obtained based on the main character mask, wherein the background region mask is used to identify the background region; The background region code is adjusted at least according to the region selection prompt information to obtain the redraw region mask; The spatial geometric structure information of the original scene in the original portrait image is obtained, wherein the spatial geometric structure information is represented by a depth map and / or a line feature map, the line feature map is used to represent the two-dimensional geometric information of the scene, the two-dimensional geometric signal includes object contours and / or edges, and the depth map is used to represent the depth information of the original scene; The spatial geometric structure information, the redrawing region mask, and the semantic guidance information are input into the diffusion model. The diffusion model is then controlled to perform iterative denoising within the spatial range defined by the redrawing region mask to generate an enhanced image. The spatial geometric structure information is used to constrain the diffusion model to generate a background that conforms to a preset geometric structure, and the semantic guidance information is used to guide the diffusion model to generate an enhanced scene with specified content and / or style.

2. The portrait scene enhancement method as described in claim 1, characterized in that, The step of generating a background region mask based on the main character mask includes: Use the complement of the main character mask as the background area mask; The step of adjusting the background region mask based at least on the region selection prompt information to obtain the redrawn region mask includes: In response to a user-inputted prompt for selecting a redrawing region, the specified background redrawing region indicated by the prompt is identified in the original portrait image. The specified background redrawing area is represented by a semantic activation mask; The redrawing region mask is obtained by performing a pixel-by-pixel logical AND operation between the semantic activation mask and the background region mask.

3. The portrait scene enhancement method as described in claim 2, characterized in that, The step of performing a pixel-by-pixel logical AND operation on the semantic activation mask and the background region mask to obtain the redrawn region includes: A first background mask is obtained by performing a pixel-by-pixel logical AND operation between the semantic activation mask and the background region mask; The redrawing region mask is obtained by performing morphological correction on the first background mask, wherein the morphological correction includes performing a dilation operation.

4. The portrait scene enhancement method as described in claim 1, characterized in that, The step of generating a background region mask based on the main character mask includes: Use the complement of the main character mask as the background area mask; The step of adjusting the background region mask based at least on the region selection prompt information to obtain the redrawn region mask includes: A first feature preservation region is determined from the portrait subject area, wherein the first feature preservation region includes at least the pixel region where the face and skin are located; In response to a user-input prompt for selecting a partial feature retention region, a second feature retention region is determined from the original portrait image, wherein the second feature retention region includes at least a pixel region corresponding to clothing or handheld props; The pixel values ​​of the pixels corresponding to the first feature retention area on the background area mask are set to 0, and the pixel values ​​of the pixels corresponding to the second feature retention area on the background area mask are set to values ​​greater than zero and less than 1 according to the area redraw amplitude coefficient, so as to obtain the initial redraw area mask. The initial redrawing region mask is morphologically modified and / or blurred to obtain the redrawing region mask.

5. The portrait scene enhancement method as described in any one of claims 1-4, characterized in that, The semantic guidance information uses text prompts to describe the target style and content of the background to be generated; Before inputting the spatial geometric structure information, the redrawn region mask, and the semantic guidance information into the diffusion model, the method further includes: Load LoRA weights pre-trained for the target style, encode the original portrait image into the latent space according to the preset redraw intensity coefficient, and add noise to obtain initial latent variables. The preset redraw intensity coefficient is a fixed hyperparameter determined after pre-tuning for the target style. The fixed hyperparameter is used to control the degree of modification of the original portrait image by the diffusion model during the iterative denoising process. The step of inputting the spatial geometric structure information, the redrawn region mask, and semantic guidance information into the diffusion model, and controlling the diffusion model to perform iterative denoising within the spatial range defined by the redrawn region mask to generate an enhanced image includes: The depth map, the line feature map, the text prompts, and the redrawn region mask are input into the diffusion model. During the iterative denoising process, the initial latent variables are updated with weights based on the weight values ​​of the redrawn region mask to obtain the initial enhanced image.

6. The portrait scene enhancement method as described in claim 5, characterized in that, The depth map and the line feature map serve as the geometric structure guide for the diffusion model, the text prompts serve as the semantic content and style guide for the diffusion model, the redraw region mask is used to apply spatial constraints to the corresponding latent variables, the preset redraw intensity coefficient is used to control the noise reduction intensity of the diffusion model, and the magnitude of the redraw intensity coefficient is set in a one-to-one correspondence with the type to which the target style belongs.

7. The portrait scene enhancement method as described in claim 1, characterized in that, The step of obtaining the spatial geometric structure information of the scene in the original portrait image includes: The original portrait image is input into the depth estimation model to obtain the depth map; The original human portrait is input into the line estimation model to obtain the line feature map.

8. The portrait scene enhancement method as described in claim 5, characterized in that, The method further includes: Select one of the following processing modes based on preset computing resources or quality requirements to obtain a high-definition enhanced image: The high-definition enhanced image is obtained by pixel-level upscaling of the initial enhanced image using a generative adversarial network model; or, The initial enhanced image is divided into blocks using a tiling sampling-based diffusion amplification strategy, and a second diffusion iteration is performed within each block to obtain a block-enlarged image. The block-enlarged images are then stitched together and edge-smoothing is performed to obtain the high-definition enhanced image.

9. The portrait scene enhancement method as described in claim 8, characterized in that, The method further includes: Obtain the detail recovery region mask, wherein the detail recovery region mask includes a skin protection mask and a personal accessory protection mask; The original portrait image is decomposed in the frequency domain to extract high-frequency texture features, wherein the high-frequency texture features include skin texture features and accessory texture features in the original portrait image; Based on the detailed recovery region mask, the high-frequency texture features are superimposed on the high-definition enhanced image to obtain the target enhanced image.

10. The portrait scene enhancement method according to claim 1, characterized in that, The portrait scene enhancement method further includes: Detect the amount of structural information of the background region in the line feature map and / or depth map; When the amount of structural information in the background region is detected to be lower than a preset threshold, the constraint strength of the spatial geometric structure information on the diffusion model is adaptively reduced.

11. A portrait scene enhancement device, characterized in that, The human image synthesis device includes: The raw data acquisition module is configured to acquire the raw portrait image and the main person mask, wherein the main person mask is used to represent the main body region of the portrait, and the main body region of the portrait includes the area where the person is located and the adjacent areas related to the person; The background region mask acquisition module is configured to obtain a background region mask based on the main character mask, wherein the background region mask is used to identify the background region; The redrawing region mask acquisition module is configured to adjust the background region code based at least on the region selection prompt information to obtain the redrawing region mask; The spatial geometric structure information acquisition module is configured to acquire the spatial geometric structure information of the scene in the original portrait image, wherein the spatial geometric structure information is represented by a depth map and / or a line feature map, the line feature map is used to represent the two-dimensional geometric information of the scene, the two-dimensional geometric signal includes object contours and / or edges, and the depth map is used to represent the depth information of the scene; An enhanced image processing module is configured to input the spatial geometric structure information, the redrawn region mask, and semantic guidance information into a diffusion model, and control the diffusion model to perform iterative denoising within the spatial range defined by the redrawn region mask to generate an enhanced image. The spatial geometric structure information is used to constrain the diffusion model to generate a background that conforms to a preset geometric structure, and the semantic guidance information is used to guide the diffusion model to generate an enhanced scene with specified content and / or style.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the portrait scene enhancement method as described in any one of claims 1-10.

13. An electronic device comprising a memory and a processor, wherein, The memory is used to store computer programs, and the processor can read the computer programs from the memory and execute them to implement the portrait scene enhancement method as described in any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the portrait scene enhancement method as described in any one of claims 1-10.

Images