Self-adaptive shadow generation method based on planar projection guidance and depth perception diffusion

Abstract

The invention discloses an adaptive shadow generation method based on planar projection guidance and depth perception diffusion, and belongs to the technical field of computer vision and image synthesis. The method comprises the following steps: in a first stage, generating a hard shadow mask of a foreground object under a virtual light source through physical projection calculation, and providing geometric position and shape priori; in the second stage, multi-modal conditions such as a hard shadow mask, a background depth image and a foreground and background fusion image and noise latent variables are uniformly coded into a Token sequence, the Token sequence is input into a Diffusion Transform model, detail rendering is carried out through a self-attention mechanism, and a composite image with a realistic shadow is output. According to the method, the problems of shadow geometric distortion, inconsistent illumination, insufficient texture fitting and the like in the prior art are solved through a mixed frame combining physical guidance and neural rendering.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image synthesis technology, and in particular to an adaptive shadow generation method based on planar projection guidance and depth-aware diffusion. Background Technology

[0002] In applications such as image editing, augmented reality (AR), and e-commerce advertising compositing, seamlessly integrating foreground objects (such as models and products) into background images is a core requirement. Among these, shadow generation is a key factor determining the realism of the composite; existing technologies mainly suffer from the following shortcomings: (1) Oversimplification of geometry: Some existing technologies (e.g., generating shadows by predicting the bounding box or ellipse parameters) ignore the complex outline of the object itself. For example, when a person makes a waving gesture, the shadow generated based on the bounding box cannot reflect the projection of the arm, resulting in "the shadow does not match the action". (2) It is difficult to guarantee the consistency of illumination: Pure end-to-end generative models (such as directly generating shadows using GAN) often rely on data memory to infer the position of the shadow, lack explicit modeling of the direction of the real light source in the background, and are prone to physical conflicts such as "the background shadow is to the left and the generated shadow is to the right". (3) Insufficient texture and fit: Although the shadows generated by traditional graphics rendering are accurately positioned, they are hard shadows and cannot simulate the "penumbra" effect that becomes blurred with distance in the real world. They are also difficult to handle the texture blending of complex ground such as grass and carpet. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, the technical problem to be solved by this invention is to propose an adaptive shadow generation method based on planar projection guidance and depth-sensing diffusion. This method aims to solve the problem that a selected image cannot generate a realistic shadow when placed in an arbitrary background during image editing. Specifically, it addresses the problems of lost projection shape details, inconsistent projection direction with ambient light, and harsh shadow texture in existing shadow generation technologies. This invention provides a high-quality shadow generation method that conforms to the laws of optical physics and can adapt to background texture.

[0004] To achieve this objective, the present invention adopts the following technical solution: This invention provides an adaptive shadow generation method based on planar projection guidance and depth-aware diffusion, comprising the following steps: S00: Obtain the background image, the foreground object image, and the corresponding foreground mask; extract scene parameters from the background image to obtain a background depth map and a background illumination vector; the steps for extracting scene parameters from the background image include: S01: Depth estimation, using a monocular depth estimation network to extract the background depth map of the background image; S02: Illumination parameter estimation. The illumination estimation module analyzes the brightness distribution of the background image and outputs the background illumination vector. :

[0005] in, The azimuth angle of the light source. The elevation angle of the light source.

[0006] S10: Based on the background illumination vector and the foreground mask, a hard shadow guide mask conforming to physical perspective is constructed through geometric projection; the step of constructing the hard shadow guide mask based on the background illumination vector and the foreground mask includes: S11: Anchor point positioning, detecting the center of the bottommost pixel region in the foreground mask, and defining it as the projection root node. ; S12: Construct the shearing projection matrix M based on the background illumination vector. Construct a clipping projection matrix M to plot the pixels in the foreground mask. Mapped to ground projection points The mapping relationship is as follows:

[0007]

[0008] S13: Transform the foreground mask using the shear projection matrix M to generate a hard shadow guide mask that preserves the outline of the foreground object.

[0009] S20: The hard shadow guide mask, the background depth map, and the foreground-background fusion map obtained by stitching the foreground object image and the background image are used as multimodal conditions; S30: Construct a joint input sequence containing the noise latent variable and the multimodal conditions, and input it into the DiT diffusion converter model; the steps for constructing the joint input sequence are as follows: S31: Multi-source feature map blockization, dividing the hard shadow guide mask, the background depth map, the foreground and background fusion map, and the noise latent variables to be denoised into image blocks respectively; S32: Token embedding and encoding, which maps each image block to a one-dimensional visual token through a linear projection layer, and adds modality type encoding and position encoding to tokens from different modalities; S33: Sequence concatenation. The tokens corresponding to the hard shadow guide mask, the background depth map, and the foreground / background fusion map are used as conditional sequences and concatenated with the target sequence corresponding to the noise latent variable along the sequence length dimension to form the joint input sequence. S40: Denoising is performed using the DiT model to generate a synthetic image that incorporates adaptive shadows; the steps of denoising using the DiT model include: S41: Full sequence self-attention interaction, the DiT model uses a self-attention mechanism to enable the token corresponding to the noise latent variable to interact with the token corresponding to the multimodal condition in order to query the geometric position, ground height, distance information and ground texture information of the generated shadow; S42: Selective decoding and reconstruction, the DiT model outputs the processed sequence, which is decoded and restored into a shadow image with adaptive penumbra effect and ground texture fusion.

[0010] In step S41, the denoising generation process of the DiT model introduces a control mechanism based on classifier-guided CFG, including: receiving at least one shadow attribute control parameter input by the user, the shadow attribute control parameter including shadow intensity, edge blur, or hue; inputting the shadow attribute control parameter as an additional condition into the DiT model, and controlling the performance of the shadow attribute in the output image through the CFG mechanism.

[0011] When the shadow falls on a non-planar surface, when constructing a hard shadow guiding mask, the non-planar geometric structure of the ground in the background image is identified based on the background depth map; based on the identified non-planar geometric structure, the shear projection matrix M is adaptively adjusted so that the generated hard shadow guiding mask adapts to the slope or curved terrain.

[0012] When there are multiple light sources, the illumination estimation module is further used to estimate the illumination vectors of the multiple light sources. The step of constructing the hard shadow guide mask includes generating corresponding hard shadow guide masks based on the illumination vectors of the multiple light sources and fusing them to simulate the shadow superposition effect under multiple light sources.

[0013] To further expand the application scope of the image processing solution in this case, the above solution can also be used for shadow generation in video sequences. When the shadow generation method is executed on each frame of the video, a temporal consistency constraint is further introduced, including: using the optical flow information between adjacent frames to distort the shadow image generated in the previous frame and using it as an auxiliary condition or initialization reference for the current frame generation process, so as to ensure the temporal continuity and stability of the shadow.

[0014] The beneficial effects of this invention are as follows: (1) Improve the physical accuracy and visual realism of shadow generation. This case uses a two-stage hybrid framework of physical guidance and neural rendering. First, a hard shadow mask that conforms to perspective is generated by the geometric projection layer, and then the diffusion model is used for detail rendering. The shadow direction of this solution is calculated by a clear lighting vector, which strictly ensures that "the shadow follows the light" and eliminates physical inconsistencies. At the same time, the shear transformation makes the generated shadow perfectly inherit the outline features of the foreground object (such as the projection of the person's fingers and hair). This approach ensures the optical validity of the shadows while achieving a highly realistic shadow effect with soft edges and blended textures, effectively solving problems such as excessive geometric simplification, inconsistent lighting, and insufficient texture fit in traditional methods. (2) Enhance the adaptability of complex scenes. Introduce depth maps as input for scene geometry conditions, enabling the model to perceive non-planar structures such as ground undulations, slopes, and steps, thereby generating adaptive shadows that fit the terrain. At the same time, it supports multi-light source estimation and shadow fusion, which can simulate the complex shadow effects of multiple light sources superimposed in the real world, significantly improving its applicability in complex scenes such as outdoor and architectural environments. (3) Improve generation efficiency and stability. By providing strong geometric priors through physical projection, the burden of the diffusion model in correcting geometric errors is greatly reduced, allowing it to focus more on optimizing details such as texture and penumbra, thereby improving the overall generation speed and effect stability. Compared with methods that rely entirely on ray tracing, this scheme has higher computational efficiency while maintaining physical accuracy. (4) Supports flexible attribute adjustment and temporal consistency. Introduces the classifier free guidance (CFG) mechanism, allowing users to adjust attributes such as shadow intensity, edge blur, and hue through parameters, thereby enhancing the controllability of the generated results. In video shadow generation, temporal constraints are applied in combination with optical flow information to effectively avoid inter-frame flicker and jitter, ensuring the continuity and stability of shadows in the time dimension. (5) Unify the expression of multimodal information and enhance the model's perception ability. Convert hard shadow mask, depth map, foreground and background fusion map and other multi-source information into token sequence and splice it with noise latent variables into DiT model. Through self-attention mechanism, cross-modal information interaction is realized, so that the model can consider multiple dimensions such as geometry, structure and texture at the same time, thereby realizing the full adaptation of shadow in terms of position, shape, ambiguity and color. The generated shadow is no longer a simple black layer, but can generate optical interaction with the material of the background ground (such as water reflection and brick seam). Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an adaptive shadow generation method based on planar projection guidance and depth-aware diffusion provided in a specific embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0017] To address issues such as oversimplification of geometry, lack of lighting consistency, and insufficient texture and fit in current shadow generation technologies—simply put, the inability to generate realistic shadows when a selected image is placed against an arbitrary background—this invention provides an adaptive shadow generation method based on planar projection guidance and depth-aware diffusion. The core concept of this approach is a two-stage hybrid framework combining physical guidance and neural rendering: a skeleton-first, texture-later generation. The first stage is the physical guidance layer, which uses physical calculations to obtain a hard shadow mask to provide the geometric skeleton. The second stage is the neural rendering layer, which transforms multimodal information such as the hard shadow mask, depth map, and foreground / background blending map into a token sequence, inputting it into a DiffusionTransformer for detail generation. The two stages are combined using a self-attention mechanism, along with depth geometry information and projection guidance, to generate adaptive shadows, ultimately reconstructing the output. This approach, which decomposes shadow generation into a "geometric projection layer" and a "texture rendering layer," first generates a physically accurate hard shadow mask through planar projection transformation, then uses it as a spatial prior input to the diffusion model for refinement. This "first localization, then rendering" framework ensures both optical plausibility and visual realism.

[0018] Example 1: An adaptive shadow generation method based on planar projection guidance and depth-aware diffusion. The following describes in detail the first and second stages of the hybrid architecture of physical projection guidance and multimodal conditional diffusion generation: (1) In the first stage, by decoupling the background lighting, the accurate "hard projection mask" of the foreground object under the virtual light source is calculated using planar homography transformation (Homography). The specific steps include: S00: Obtain the background image, foreground object image, and corresponding foreground mask; extract scene parameters from the background image to obtain a background depth map and background illumination vector; this step mainly involves analyzing the background image to extract a depth map representing the scene's geometric structure and an illumination vector representing the direction of the main light source. Specifically, it involves inputting the foreground object image, foreground mask, and background image, and then extracting scene parameters from the background image through the following steps: S01: Depth estimation, using a monocular depth estimation network (such as MiDaS, DPT, etc.) to extract the background depth map of the background image; used for subsequent understanding of ground structure, that is, encoding the layout, undulation and occlusion relationship of the scene. For shadow generation, the most critical role of the depth map is to define the geometry of the ground (or receiving surface), and the shadow needs to bend, break or stretch according to this shape. S02: Illumination parameter estimation. The illumination estimation module analyzes the brightness distribution of the background image to determine the light-dark boundary lines and shadow projection directions of existing objects in the image. This allows for the deduction of the light source, and the output of the background illumination vector. :

[0019] in, The azimuth angle of the light source (determines the direction of the shadow), that is, the direction of the light source on the horizontal plane (0~360° or -180°~180°), which affects the direction of the shadow's extension on the ground. Changing θ will cause the shadow to rotate around the object. The elevation angle of the light source (which determines the shape of the shadow) is the angle of elevation of the light source relative to the horizon (0° is the horizon, 90° is the zenith). The larger the size, the denser and shorter the shadow, and the smaller the area. The smaller the size, the longer and sparser the shadow, the larger the area, and the more likely there will be strong perspective stretching; Based on this, depth estimation includes environmental features such as ground normal direction, scene geometry undulation, and occlusion relationships, while lighting estimation includes features such as shadow extension direction, shadow length (or coverage area), and coverage region. This ensures that the shadows generated subsequently follow the same lighting rules as all other objects in the scene, thus solving the problem of "lighting consistency." This decoupling allows subsequent shadow generation to consider both directional correctness and morphological rationality separately, which also facilitates subsequent graphics rendering. While ensuring physical correctness, it provides clear, structured, and easy-to-understand prior conditions for the subsequent diffusion model.

[0020] S10: Based on the background illumination vector and the foreground mask, a hard shadow guide mask conforming to physical perspective is constructed through geometric projection. This step mainly involves constructing a physically consistent initial projection guide map, providing strong geometric guidance, and ensuring the physical position, direction, and shape of the shadow. Specifically, based on the foreground object's mask and illumination vector, each foreground pixel in the mask is considered as a point above the assumed ground plane. A ray is emitted along the opposite direction of the illumination, and the intersection of this ray with the ground is calculated. The 2D positions of all intersection points are gathered to form a new region located on the ground, which is the hard shadow mask. Its shape is the orthogonal projection of the foreground object under parallel light. This initially defines the position and main shape when generating shadows. Of course, in reality, the light source can only be approximated as parallel light when it reaches a relatively long distance. However, this step aims to provide a physically reasonable and detailed initial estimate. Thus, the steps of constructing a hard shadow guide mask based on the background illumination vector and the foreground mask include: S11: Anchor point positioning, detecting the center of the bottommost pixel region in the foreground mask, and defining it as the projection root node. That is, assuming the object is upright on the ground, the root node of the projection represents the main contact point between the object and the ground. S12: Construct the shear projection matrix M. Preferably, in this example, the ground is assumed to be a local plane, based on the background illumination vector. Construct a clipping projection matrix M to plot the pixels in the foreground mask. Mapped to ground projection points The mapping relationship is as follows:

[0021]

[0022] S13: The foreground mask is transformed using the shear projection matrix M to generate a hard shadow guide mask that preserves the outline of the foreground object. This yields the initial projection. This mask fully preserves the action outline of the object (such as fingers or clothing corners), rather than a simple rectangle or ellipse. It provides important guidance for the position, shape, and texture details of the shadow when the diffusion model generates shadows.

[0023] (2) In the second stage, the hard shadow guide mask, background depth map, foreground-background fusion map and noise latent variables to be denoised are respectively divided into patches and mapped to tokens, and concatenated into a joint input sequence in the sequence dimension. The joint sequence is input into the Diffusion Transformer model, and adaptive shadows are generated by combining depth geometric information and projection guidance using the self-attention mechanism. Finally, the output is reconstructed, which includes the following steps: S20: The hard shadow guide mask, the background depth map, and the foreground-background fusion map obtained by stitching the foreground object image and the background image are combined. The foreground-background fusion map is usually an initial composite image where the foreground object (with an alpha channel) is pasted onto the background, serving as a multimodal condition. The hard shadow guide mask is a physical constraint layer, representing the geometric position and main shape of the shadow. The background depth map is a geometric environment layer, representing the environmental features of shadow generation, such as the shadow needing to conform and deform when the ground has unevenness. The foreground-background fusion map is a visual appearance layer, representing the foreground appearance (color and material of the object casting the shadow), background texture and color (i.e., the environmental features of shadow generation, such as the material where the shadow falls, the surrounding hue, etc.), and spatial relationship (the object and the background are already in the correct foreground and background positions). These three elements are combined to guide the subsequent model to generate a shadow in the correct position, with the correct shape, and in harmony with the background geometry and texture.

[0024] In step S30, a joint input sequence containing the noise latent variable and the multimodal conditions is further constructed. Specifically, the construction steps of the joint input sequence are as follows: S31: Multi-source feature map patching: The hard shadow guide mask, the background depth map, the foreground-background fusion map, and the noise latent variable to be denoised are divided into image blocks. This step mainly involves constructing a full feature sequence including geometry, texture, and lighting guidance, obtaining the following four sets of feature maps, and then uniformly scaling them to the latent spatial resolution before dividing them into fixed-size blocks: (a) Latent variables of Gaussian noise to be generated; (b) The hard shadow guide mask generated in step S10 is used to provide the basic position skeleton of the shadow; (c) The background depth map extracted in step S00 is used to provide three-dimensional terrain and distance information of the scene; (d) Foreground / background blending map, used to provide foreground, environment texture, and color context; S32: Token embedding and encoding, which maps each image block to a one-dimensional visual token through a linear projection layer, and adds modality type encoding and position encoding to tokens from different modalities; S33: Sequence concatenation, the tokens corresponding to the hard shadow guide mask, the background depth map, and the foreground-background fusion map are used as conditional sequences and concatenated with the target sequence corresponding to the noise latent variable in the sequence length dimension to form the joint input sequence; After steps S32 and S33, the four sets of image patches are mapped into one-dimensional visual token vectors through a linear projection layer. To distinguish information from different modalities, a corresponding modality type encoding is added to each token, and a position encoding is added to preserve the spatial structure. Preferably, the hard shadow guide mask token, background depth map token, and foreground / background fusion map token are used as conditional sequences, and the noise latent token is used as the target sequence. These are concatenated along the sequence length dimension to form a joint input sequence. In summary, firstly, an encoder (such as the encoder of a VAE) is used to transform the target image (i.e., the image with shadows that is ultimately desired) into a low-dimensional latent variable; then, random noise is added to this latent variable to obtain the noise latent variable, i.e., the noise latent variable in (a) above; at the same time, the multimodal conditions (hard shadow guide mask, depth map, fusion map) in S20 are converted into a set of conditional labels through an independent conditional encoder (usually a CNN or a smaller Transformer), and concatenated with the noise latent variable or through cross-attention to form a long sequence (i.e., the joint input sequence), which is then input into the diffusion transformer DiT. The model, in this way, allows each layer of the DiT model to simultaneously perceive the current image state corrupted by noise, as well as all explicit physical and visual conditions; then, in step S40, denoising is generated using the DiT model to restore a clear image. Preferably, the step of denoising is generated using the DiT model includes: S41: Full-sequence self-attention interaction. The DiT model uses a self-attention mechanism to enable the token corresponding to the noise latent variable to interact with the token corresponding to the multimodal condition, in order to query the geometric position, ground height, distance information, and ground texture information of the generated shadow. Specifically, the Diffusion Transformer (DiT) model is used to process the joint input sequence. The joint input sequence is input into the Transformer block of the DiT network. Through the self-attention mechanism, the noise token representing the "shadow generation region" can be directly queried and generated, including: Hard shadow guide mask token, which determines "whether it should turn black"; The depth map token determines "the height of the ground" and "how far away from the root of the object". The depth map is input into the diffusion model as one of the conditions, so that the model can perceive "the distance between the shadow and the root of the object" and "the change of ground height", thereby automatically adjusting the blur of the shadow edge to achieve a soft shadow effect close to real optics. Foreground / background blending token determines the image content and "what the covered ground texture is (such as grass or tiles)"; S42: Selective decoding and reconstruction. The DiT model outputs the processed sequence, extracts the corresponding target sequence part, maps it back to two-dimensional space, and restores it to a synthetic image with real shadows through the decoder, that is, decodes and restores it to a shadow image with adaptive penumbra effect and ground texture fusion; further outputs a synthetic image fused with adaptive shadows.

[0025] In summary, this approach unifies and tokenizes three types of heterogeneous visual information—hard shadow guiding masks, depth maps, and foreground / background fusion maps—and concatenates them with noise latent variables to form a joint input sequence for the DiT model. Through a self-attention mechanism, the model simultaneously considers geometric position (from the hard shadow guiding mask), scene 3D structure (from the depth map), and texture (from the foreground / background fusion map), thus achieving adaptive shadow generation across multiple dimensions, including shape, position, texture, and blur. Specifically, when forming shadows, due to the mandatory conditions of the hard shadow guiding mask, the DiT model prioritizes ensuring that the generated content covers the area to constrain the shape. Through training, the DiT model incorporates the sharp boundaries of the hard shadow guiding mask based on the hints from the depth map. The distance between the object and the ground is diffused into a softly transitioning penumbra to soften the shadow edges. Furthermore, by referencing the color and texture of the ground in the foreground and background fusion image, the DiT model renders color shifts, brightness reductions, and subtle texture blending in the shadow area. At the same time, using the depth map, the shadow is deepened at the contact point near the foreground object (contact shadow) and lightened at the distance. Finally, the entire image is fine-tuned to ensure that the newly generated shadow is seamlessly integrated with the original background in terms of color balance, contrast, and noise level, eliminating traces of compositing. After multiple iterations of noise reduction, a clean latent variable is obtained, which is then converted back to pixel space by a decoder (such as the decoder of VAE), resulting in the final high-quality composite image.

[0026] Example 2: In Example 1, the main approach was based on the assumption that the ground from which the shadow is generated is planar. That is, a shear projection matrix M is used to project the foreground mask onto the ground. In real-world scenarios, the ground may have slopes, undulations, or other non-planar geometric structures. Therefore, a background depth map is used for further supplementation and processing. Building upon this, to further adapt to complex terrain conditions and ensure the visual realism and physical consistency of the synthesized image, when constructing the hard shadow guide mask, the non-planar geometric structures of the ground in the background image are identified based on the background depth map. Based on the identified non-planar geometric structures, the shear projection matrix M is adaptively adjusted to adapt the generated hard shadow guide mask to slopes or curved terrain. Specifically, a depth-aware adaptive adjustment module is further introduced. Using the depth information (background depth map) extracted from the background image, the three-dimensional geometric structure of the ground is modeled and analyzed, thereby deriving the correction amount for the parameters of the original shear projection matrix M. This ultimately achieves the transformation of the projection process from an assumed plane to a perceived terrain. The entire adaptive process includes the following steps: S101: Based on the input background depth map, analyze the terrain of the expected projection area of ​​the foreground object. This step first determines the projection root region, i.e., based on the foreground mask, locates the main area where the object contacts the ground (i.e., the local area where the projection root node is located). Next, reconstruct the local 3D terrain; that is, within the root region and its surrounding areas, using the pixel-level depth values ​​provided by the depth map, combined with camera intrinsic parameters (such as focal length), backproject the 2D pixel coordinates to 3D space to form a local 3D point cloud. Finally, fit the ground geometry model and perform geometric analysis on the local 3D point cloud. Preferably, a plane fitting algorithm (such as the RANSAC algorithm) is used to estimate the approximate plane equation of the ground in this area. This equation is derived from the normal vector. In addition to defining the planar height parameters, for more complex terrain, a discrete height field can be established or a piecewise planar model can be used for description. The output of this step is a structured parameter characterizing the local ground geometry, such as the ground normal vector in this embodiment. ; S102: Based on the terrain geometry parameters obtained in step S101, and combined with the known background illumination vector... (defined as azimuth angle) and elevation angle The parameters used to adjust the original shear projection matrix M are calculated. First, the adjustment relationship needs to be established. The original shear projection matrix M, under the assumption of a planar ground, linearly maps the foreground pixels along the opposite direction of illumination. When the ground is not planar, this linear mapping relationship needs to be adjusted according to the ground tilt angle. That is, the projection offset should not only be determined by the vertical position difference of the pixels and the illumination angle, but also consider the geometric relationship of the actual intersection point of the light rays and the ground. Then, the adjustment parameters can be calculated. Preferably, the illumination direction vector is calculated. With local ground normal vector The geometric relationship between them is explained by introducing the scale adjustment parameter s and the orientation correction angle. The scale adjustment parameter 's' is used to correct the length of the shadow. When the ground is a slope and the direction of the light is parallel to the slope, the shadow will be lengthened; conversely, it will be shortened. The scale adjustment parameter 's' can be derived from the light direction vector. With ground normal vector Calculation of dot product or other trigonometric function relationships; direction correction angle. Used to correct the direction of shadow extension, when the ground normal vector When a horizontal component exists, the direction of shadow extension will be deflected relative to the direction of projection onto the horizontal plane; the direction correction angle... From the ground normal vector Horizontal component and illumination direction vector The geometric relationship determines this; thus, the above parameters can be integrated into the original shear projection matrix M to obtain the adjusted adaptive mapping relationship:

[0027]

[0028] S103: Apply the adaptive projection mapping relationship obtained in step S102 to perform coordinate transformation on each valid pixel in the foreground mask. First, perform pixel-by-pixel projection, that is, traverse the foreground mask. For each pixel, use the adjusted formula to calculate its projection point coordinates on the background image plane. Then, perform mask rendering, such as setting all the calculated projection point coordinates together and rendering a new terrain-adaptive hard shadow guide mask. Since the projection mapping has been adjusted according to the ground geometry, this mask will accurately fall on the real ground contour represented by the background depth map. For example, it will show shortening and extension along the slope on a slope, and reasonable breaks at steps.

[0029] In summary, the hard shadow guide mask generated after the above steps closely matches the 3D geometry of the background scene, further resolving physical errors such as shadow position floating and penetration that may be caused by the planar assumption. This enhances the realism and accuracy of the synthesized image, enabling effective handling of complex backgrounds with rich non-planar terrain, such as outdoor natural scenes (mountains, hills) and built environments (stairs, ramps), greatly expanding the application scope. Simultaneously, it provides higher quality and more reliable initial geometric constraints for subsequent data-driven refinement modules based on diffusion models. Since the initial mask is close to reality, it reduces the burden on the generation model to correct major geometric errors, allowing it to focus more on optimizing details such as soft edges and texture blending of shadows, improving overall generation efficiency and effect stability. Furthermore, compared to precise ray tracing methods that rely entirely on time-consuming rendering, this approach primarily uses efficient matrix transformation operations after local geometric analysis and parameter adjustment, resulting in more efficient computation.

[0030] Example 3: Examples 1 and 2 are based on a single light source. However, in real-world scenarios, light sources are often not unique. Multiple light sources produce shadows of multiple directions and intensities. These shadows overlap and blend, forming a complex shadow effect with gradations and penumbra areas. Therefore, the illumination estimation module is further used to estimate the illumination vectors of multiple light sources. The step of constructing the hard shadow guide mask includes generating corresponding hard shadow guide masks based on the illumination vectors of the multiple light sources and fusing them to simulate the shadow overlay effect under multiple light sources. Specifically, in actual implementation, the following steps in the original process are adjusted: (1) In the original step S02, the illumination parameter calculation is configured to analyze the global brightness distribution, highlight area and existing shadow direction of the background image, and output a multi-source light source set, which contains multiple illumination vectors. ( ),in and Let be the azimuth and elevation angles of the i-th light source. The relative intensity or contribution weight of the i-th light source. Number of light sources; (2) When constructing the shearing projection matrix M in the original step S12, it is adjusted to construct multiple shearing projection matrices M in parallel, based on each illumination vector in the above multi-source set. Each of them independently constructs its corresponding shear projection matrix. The mapping relationship of each matrix is ​​the same as in the case of a single light source; (3) When generating the hard shadow guide mask in the original step S13, it is adjusted to utilize each projection matrix. The foreground mask is transformed to generate a hard shadow guide mask corresponding to the i-th light source. Then, multiple shadow masks are sorted according to the relative intensity of their light sources. Weighted overlay is performed to generate a blended hard shadow guide mask. Preferably, in conjunction with Example 2, the background depth map is used for each... After performing independent adaptive terrain adjustments, the mixture is then blended to ensure that each shadow component correctly fits the ground geometry, and the blended result remains physically accurate even on complex terrain. (4) The original step S20 is adjusted to use the fusion hard shadow guide mask, background depth map and foreground background fusion map as a multimodal condition and input it into the subsequent diffusion model (such as DiT). The task of the diffusion model is thus changed to: on the basis of existing physically correct multi-shadow superposition, perform penumbra generation, texture fusion and global coordination.

[0031] In summary, this embodiment provides more information-rich and more precise guidance for the subsequent diffusion model by using a fusion hard shadow guide mask under multiple light sources. It not only points out a shadow area but also implies the "shadow density" of different parts within the shadow area through a weight map. The direct effect is to explicitly model and fuse the projections of multiple physical light sources. The generated shadows have richer layers and more natural gradations. The shadows are no longer single, homogeneous dark areas but exhibit different densities and hardness in different areas. This is consistent with the physical principle of light interacting in three-dimensional space, which greatly enhances the stereoscopic effect and spatial credibility of the synthesized image and improves the realism of shadows under complex lighting conditions.

[0032] Example 4: In step S41, this example uses the DiT model to learn and synthesize realistic textures, colors, and details from the data distribution. However, its drawback is that it is less accurate in dealing with spatial changes. In areas with blurred depth maps or complex regions, shadows may deviate from the correct physical projection position, or it is difficult to ensure accurate perspective shortening and bending of shadows on slopes. Therefore, in order to make the physical calculation of the shear projection matrix M and its efficient cooperation, the denoising generation process of the DiT model introduces a control mechanism based on classifier-guided CFG, including: receiving at least one shadow attribute control parameter input by the user, the shadow attribute control parameter including shadow intensity, edge blur, or hue; inputting the shadow attribute control parameter as an additional condition into the DiT model, and controlling the performance of the shadow attribute in the output image through the CFG mechanism; thus, the shadow has a natural penumbra, accurate texture blending, and perfect ambient light integration, outputting a higher quality shadow synthesis effect.

[0033] Example 5: The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion described in this case can also be used for shadow generation in video sequences, extending the image shadow generation method of this case to video sequences for further expansion. In a video, shadows need to be generated for each frame, but the continuity and stability of shadows between frames must be guaranteed to avoid flickering and jitter. Therefore, when performing the shadow generation method on each frame of the video, it is also necessary to further introduce temporal consistency constraints, including: using the optical flow information between adjacent frames to distort the shadow image generated in the previous frame and using it as an auxiliary condition or initialization reference for the current frame generation process to ensure the temporal continuity and stability of the shadows; thus, combining the diffusion model with the temporal constraints of optical flow guidance for video shadow generation solves the problem of temporal instability in the generation model.

[0034] This invention has been described through preferred embodiments. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. This invention is not limited to the specific embodiments disclosed herein; other embodiments falling within the scope of the claims are also within the protection scope of this invention.

Claims

1. An adaptive shadow generation method based on planar projection guidance and depth-aware diffusion, characterized in that, Includes the following steps: S00: Obtain the background image, foreground object image, and corresponding foreground mask; Scene parameters are extracted from the background image to obtain a background depth map and a background illumination vector; S10: Based on the background illumination vector and the foreground mask, construct a hard shadow guide mask that conforms to physical perspective through geometric projection; S20: The hard shadow guide mask, the background depth map, and the foreground-background fusion map obtained by stitching the foreground object image and the background image are used as multimodal conditions; S30: Construct a joint input sequence containing the noise latent variable and the multimodal conditions, and input it into the diffusion converter DiT model; S40: Denoising is performed using the DiT model to generate a synthetic image that incorporates adaptive shadows.

2. The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion according to claim 1, characterized in that: In step S00, the step of extracting scene parameters from the background image includes: S01: Depth estimation, using a monocular depth estimation network to extract the background depth map of the background image; S02: Illumination parameter estimation. The illumination estimation module analyzes the brightness distribution of the background image and outputs the background illumination vector. : in, The azimuth angle of the light source. The elevation angle of the light source.

3. The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion according to claim 2, characterized in that, In step S10, the step of constructing a hard shadow guide mask based on the background illumination vector and the foreground mask includes: S11: Anchor point positioning, detecting the center of the bottommost pixel region in the foreground mask, and defining it as the projection root node. ; S12: Construct the shearing projection matrix M based on the background illumination vector. Construct a clipping projection matrix M to plot the pixels in the foreground mask. Mapped to ground projection points The mapping relationship is as follows: S13: Transform the foreground mask using the shear projection matrix M to generate a hard shadow guide mask that preserves the outline of the foreground object.

4. The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion according to claim 3, characterized in that, In step S13, when constructing the hard shadow guide mask, the non-planar geometric structure of the ground in the background image is identified according to the background depth map; based on the identified non-planar geometric structure, the shear projection matrix M is adaptively adjusted so that the generated hard shadow guide mask adapts to the slope or curved terrain.

5. The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion according to claim 3, characterized in that, In step S13, the illumination estimation module is further used to estimate the illumination vectors of multiple light sources. The step of constructing the hard shadow guide mask includes generating corresponding hard shadow guide masks based on the illumination vectors of the multiple light sources and fusing them to simulate the shadow superposition effect under multiple light sources.

6. The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion according to claim 3, characterized in that, In step S30, the construction steps of the joint input sequence are as follows: S31: Multi-source feature map blockization, dividing the hard shadow guide mask, the background depth map, the foreground and background fusion map, and the noise latent variables to be denoised into image blocks respectively; S32: Token embedding and encoding, which maps each image block to a one-dimensional visual token through a linear projection layer, and adds modality type encoding and position encoding to tokens from different modalities; S33: Sequence concatenation, the tokens corresponding to the hard shadow guide mask, the background depth map, and the foreground-background fusion map are used as conditional sequences and concatenated with the target sequence corresponding to the noise latent variable in the sequence length dimension to form the joint input sequence.

7. The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion according to claim 6, characterized in that, In step S40, the step of generating denoised data using the DiT model includes: S41: Full sequence self-attention interaction, the DiT model uses a self-attention mechanism to enable the token corresponding to the noise latent variable to interact with the token corresponding to the multimodal condition in order to query the geometric position, ground height, distance information and ground texture information of the generated shadow; S42: Selective decoding and reconstruction, the DiT model outputs the processed sequence, which is decoded and restored into a shadow image with adaptive penumbra effect and ground texture fusion.

8. The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion according to claim 7, characterized in that, In step S41, the denoising generation process of the DiT model introduces a control mechanism based on classifier-guided CFG, including: receiving at least one shadow attribute control parameter input by the user, the shadow attribute control parameter including shadow intensity, edge blur, or hue; inputting the shadow attribute control parameter as an additional condition into the DiT model, and controlling the performance of the shadow attribute in the output image through the CFG mechanism.

9. The adaptive shadow generation method based on planar projection guidance and depth-aware diffusion according to claim 1, characterized in that, It is also used for shadow generation in video sequences. When the shadow generation method is executed on each frame of the video, a temporal consistency constraint is further introduced, including: using the optical flow information between adjacent frames to distort the shadow image generated in the previous frame and use it as an auxiliary condition or initialization reference for the current frame generation process to ensure the temporal continuity and stability of the shadow.

Images