A 3D perception-based single synthetic image shadow generation method

Abstract

The application discloses a single image shadow generation method based on 3D perception, which comprises three-plane representation construction and pixel alignment volume rendering, uses two feature extractors to process different input combinations, uses a local channel-space cross-attention module to fuse the two features obtained to generate a unified three-plane feature representation, defines a 3D coordinate space for each image pixel, and projects an orthogonal ray along a depth axis to sample 3D points, the sampling points are projected onto the three-plane representation, and the sampling features are retrieved through a bilinear interpolation method; the sampling features are sent to a two-stage MLP: the first stage predicts shadow-related outputs and intermediate features, and the second stage predicts color-related outputs and volume density, and based on the volume density and an Alpha synthesis rule, foreground shadows and a final shadow image are generated.

Description

Technical Field

[0001] This invention relates to image synthesis technology, specifically to a method for generating shadows in a single synthesized image based on 3D perception. Background Technology

[0002] Image compositing aims to merge the foreground of one image with a background image to generate a composite image, with wide applications in virtual reality, artistic creation, e-commerce, and other fields. Simply pasting the foreground onto the background often leads to visual inconsistencies, including lighting incompatibility between the foreground and background, and a lack of foreground shadows / reflections. We focus on the shadow problem; if the inserted foreground object does not have a reasonable shadow on the background, this can severely degrade the realism and quality of the composite image. Shadow generation can improve the realism of images and scenes. Existing shadow generation methods can be divided into two main categories: rendering-based methods and image-to-image transformation methods.

[0003] Render-based methods typically require explicit knowledge of the lighting, reflections, material properties, or scene geometry of foreground and background objects to generate shadows for inserted virtual objects using rendering techniques. However, this information is often unavailable in real-world applications. Early approaches, such as Kee et al., used 3D scene reconstruction or linear programming to generate shadows, but required user interaction to ensure accuracy. Sheng, in SSN, used 2D object masks and ambient light maps to generate controllable shadows and combined this with an ambient occlusion prediction module to generate realistic shadows that could be interactively refined. Subsequent work used geometric representations to control shadow direction and shape, and in PixHt-Lab, mapped pixel height data to 3D space for geometry reconstruction. These methods sometimes require predicting 3D information from 2D images, which is quite challenging in complex scenes. In the absence of user interaction, Gardner attempted to recover explicit lighting conditions and scene geometry from a single image, but inaccurate estimations could lead to unsatisfactory results.

[0004] Image-to-image transformation methods: These methods learn a mapping from an input image without foreground shadows to an output image with foreground shadows, without requiring explicit knowledge of lighting, reflection, etc. The mainstream methods are broadly of two types: those based on generative adversarial networks (GANs) and those based on diffusion models.

[0005] Generative Adversarial Network (GAN)-based methods ensure the accuracy and realism of generated shadows through joint supervision of reconstruction and adversarial losses. ShadowGan utilizes a local discriminator to fine-tune details and a global discriminator to fine-tune the global perspective. It jointly uses global and local conditional discriminators to enhance the realism of the generated shadows. MASK-ShadowGan uses generated shadow templates to guide shadow generation through cycle consistency constraints. AR-ShadowGan uses an attention mechanism to simulate the mapping between virtual shadows and the real environment. SGRNet first predicts the foreground shadow template through the interaction of foreground and background information, then predicts shadow parameters to fill the shadow region. However, these GAN-based methods typically require training the model from scratch on limited paired data, which limits generalization ability.

[0006] Diffusion-based methods: DMASNet first decomposes shadow template generation into cuboid prediction and shape prediction, then focuses on background shadow pixels to achieve accurate foreground shadow filling. SGDiffulation makes ControlNet suitable for shadow generation and incorporates intensity modulation to optimize shadow intensity. Despite introducing diffusion models, these methods still face the problem of poor shadow geometry.

[0007] A key reason for this limitation is the lack of 3D perception in traditional 2D image generation pipelines, which hinders their ability to capture spatial relationships and maintain geometric consistency. Without a proper understanding of the underlying scene structure, these methods often generate shadows that appear visually plausible but are geometrically inaccurate. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a method for generating shadows in a single composite image based on 3D perception.

[0009] Technical solution: The present invention provides a method for generating shadows in a single composite image based on 3D perception, comprising the following steps:

[0010] Step 1: Construct a dual-branch feature extraction module to extract the foreground object mask from the same image. Predicted foreground shading mask Composite images without foreground shadows and background object-shadow mask pair Input the dual-branch feature extraction module together, background object-shadow mask pair Abbreviated as The dual-branch feature extraction module includes a structure-aware branch. and appearance perception branch Structure-aware branch (e.g., Unet network) foreground object mask Predicted foreground shading mask and background object-shadow mask pairs The process is performed to obtain the structural feature map. Appearance perception branch (e.g., Unet network) for synthesized images and background object-shadow mask pair The process is performed to obtain the appearance feature map. ;

[0011] Step 2: Construct a Local Channel-Spatial Cross Attention Module (LCSAM) that combines the structural feature map and appearance feature map obtained in Step 1. Figure 1 The input is a Local Channel-Spatial Cross-Attention Module (LCSAM), which includes a local channel cross-attention mechanism and a local spatial cross-attention mechanism; the specific method is as follows:

[0012] Step 2.1, in the local channel cross-attention mechanism, first, the appearance feature map... and structural feature diagram Each channel is divided into G groups, and each group includes This grouping strategy, which uses multiple channels, helps reduce computational costs while supporting localized attention modeling. It applies a group-specific convolution to each group. Generate query vector Q c and from Generate key K c Sum V c Then Q c With K c The attention weights are obtained by performing dot product operations and softmax normalization sequentially. Then, attention weights are used. Pair value V c Weighted aggregation is performed to obtain the attention-weighted response, which is then reshaped into the original feature dimension to obtain the attention-enhanced feature y. c Finally, attention-enhancing features y c Compared with the original appearance feature map The summation yields the local channel enhancement feature X. m2 ;

[0013] Step 2.2: To enhance spatial adaptability, in the local spatial cross-attention mechanism, for the original appearance feature map X... r and local channel enhancement features X m2 First, the feature map is expanded by a window size w, dividing it into non-overlapping local windows. Then, a query Q is generated for each local window w through convolution.s Key K s Sum V s And calculate local attention weights Then, the obtained local attention weights are... Acting on the value characteristic V s After summarizing the outputs of all local windows, a full-resolution feature map is reconstructed through a folding operation, thus obtaining the local spatial enhancement feature y. s ;

[0014] Step 2.3: Apply the local channel enhancement feature X obtained in Step 2.1 m2 Compared with the local spatial enhancement feature y obtained in step 2.2 s Adding them together yields the final output feature X of the entire Local Channel-Space Cross-Attention Module (LCSAM). LCSAM ;

[0015] Step 3: Based on the synthesized image For each pixel, an orthogonal parallel sampling ray is generated, and each ray samples several sampling points to construct a virtual 3D space, obtaining the spatial coordinates of a set of sampling points. Then, based on the spatial coordinates, the feature X is... LCSAM Reshaped into three orthogonal two-dimensional planes F xy F xz and F yz Then, the three orthogonal two-dimensional planes are input into the pixel-aligned volume rendering module. Utilizing the feature space encoding capability of the three planes, geometric perception is injected into the two-dimensional shadows to achieve virtual 3D space sampling, obtaining the feature F of the three-plane sampling points. t (Subsequently used for shadow prediction and volume rendering);

[0016] Step 4: Generate an optimized foreground shadow mask And the final image with a shadow background Here, a progressive two-stage volume rendering strategy is used to sample the features F of three planes through a dual MLP network. t To perform progressive volume rendering, the specific method is as follows:

[0017] First, the features F of the three plane sampling points t Inputting the first MLP, it predicts the shadow mask probability g ∈ [0, 1] and the intermediate features F. s Then the intermediate feature F s Compared with the original three-plane sampling point features F t The concatenated features are then input into a second MLP to predict RGB color c and volume density σ. Next, the shared volume density σ is used to perform volume rendering on the RGB color value c and the shadow mask g, following standard α-composition rules. Finally, an optimized foreground shadow mask is generated by weighted summation of the point-by-point prediction results. and background images with shadows ;

[0018] Step 5: During training, a composite loss function is used to optimize the dual-branch feature extraction module, the Local Channel-Spatial Cross Attention (LCSAM) module, the pixel-aligned volume rendering module, and the dual MLP network. The composite loss function... Including shadow mask prediction loss Image reconstruction loss and perceived loss .

[0019] To effectively capture structural layout and semantic appearance from the input, the dual-branch feature extraction module in step 1 independently processes the combination of structure-aware and appearance-aware inputs. It fully leverages the complementarity of structural and appearance cues, separating geometric layout and occlusion relationships from visual texture and color information. The structure-aware branch... Extracting features such as object boundaries, spatial configuration, and potential shadow projection regions to construct feature maps. Appearance perception branch Capture appearance-related features, including color contrast, lighting consistency, and material properties, to obtain an appearance feature map. .

[0020] To effectively integrate the structural and appearance features extracted from the dual branches, step 2 uses the Local Channel-Spatial Cross Attention Module (LCSAM, Focused Local Region Modeling) to capture fine-grained cross-channel dependencies in both channel and spatial dimensions. This enables collaborative channel-aware fusion while maintaining computational efficiency, promoting more robust and semantically aligned feature fusion, thereby achieving more accurate and context-aware shadow prediction.

[0021] Attention weight The calculation formula is: ;

[0022] Local channel enhancement feature X m2 The calculation formula is: ;

[0023] Local attention weights The calculation formula is: ;

[0024] Final output feature X LCSAM The calculation formula is: .

[0025] Step 3 allows sampling from the three-plane feature space without camera information. This is achieved by constructing a virtual three-dimensional volume space through orthogonal projection of sampling rays from each pixel of the synthesized image plane. This ensures the sampling process aligns with the two-dimensional image structure, enabling the extraction of spatially consistent three-plane sampling point features F from the pre-computed three-plane representation even in the absence of explicit three-dimensional cues. t The specific method is as follows:

[0026] Given a synthetic image with a resolution of r × r First, a normalized two-dimensional coordinate grid is generated. Each pixel coordinate (x, y) ∈ U is normalized to the interval [-1, 1]; for each coordinate, an orthogonal sampling ray is defined with the starting point O = [x, y, -1] and the direction of the orthogonal sampling ray D = [0, 0, 1], and perpendicular to the plane of the synthesized image;

[0027] Then, along each ray, S = 32 points are uniformly sampled along the depth direction within the preset range [0, 2]. The depth value of the s-th sampling point is calculated as follows:

[0028] ; ;

[0029] The corresponding three-dimensional coordinates are given by the following formula: P x,y,z = O + (Z s D);

[0030] Each sampling point P is projected onto three orthogonal two-dimensional planes F. xy F xz and F yz superior;

[0031] Finally, the features of the three orthogonal two-dimensional planes are extracted at the projection position using bilinear interpolation, and then fused into a three-plane sampling point feature F. t .

[0032] Although step 3 above has constructed a 3D-aware feature sampling space through pixel-aligned sampling, using a single MLP to predict color and volume density would result in poor shadow modeling performance. Therefore, step 4 employs a dual MLP network to enhance spatial inference and improve rendering quality; its first MLP predicts the shadow mask probability g ∈ [0, 1] and the intermediate feature F s The formula is as follows:

[0033] ;

[0034] The formulas for the second MLP to predict RGB color c and volume density σ are as follows:

[0035] ;

[0036] The specific method for volume rendering using volume density σ on RGB color value c and shadow mask probability g is as follows:

[0037] For sampling point i, given its color value Shadow mask probability and depth value First, the absorption coefficient of each segment is calculated based on the Beer-Lambert law. , The α value, representing the amount of light absorbed at each point, is calculated using the following formula:

[0038] ;

[0039] in Indicates the distance between two adjacent sampling points;

[0040] The weight of each sampling point i mentioned above Its contribution to the final pixel color is determined by its opacity. The result is obtained by multiplying the cumulative transmittance prior to sampling point i, as shown in the following formula:

[0041] ;

[0042] in Set to 1×10 −10 This is used to prevent numerical instability and avoid multiplication by zero. This represents the absorption coefficient of all points preceding the i-th sampling point along the direction of the light ray;

[0043] Finally, an optimized foreground-shadow mask is generated by weighted summation of the prediction results point by point. and background images with shadows The formula is as follows:

[0044] ;

[0045] ;

[0046] A two-stage design is used here, with auxiliary mask supervision, to enable the model to better learn spatially coherent shadow structures.

[0047] To simultaneously optimize the structural accuracy of predicted shadow regions and the perceptual quality of the generated image, a composite loss function is used. Including shadow mask prediction loss Image reconstruction loss and perceptual loss function ;

[0048] Shadow mask prediction loss To ensure the optimized shadow mask With the real mask To maintain consistency, the L1 loss function is used to penalize pixel-level differences:

[0049] ;

[0050] Image reconstruction loss To improve the quality of the synthesized image, a multi-reconstruction loss function is introduced. First, the mean squared error loss is calculated by... (The sentence is incomplete and requires more context to translate accurately.) With real images Pixel-level differences between them ensure the accuracy of the generated results:

[0051] ;

[0052] Perceived loss By comparing differences in deep features, the model is encouraged to preserve the structural and semantic details of the image.

[0053] ;

[0054] in, This indicates the number of steps taken from the pre-trained VGG network. Features extracted from layers;

[0055] Finally, the combined loss function is obtained by weighted summation of the three losses mentioned above. The formula is as follows:

[0056] ;

[0057] Here, λ is a trade-off parameter used to balance the weights of the perception loss and other loss terms, and λ is set to 0.001.

[0058] Beneficial effects: This invention proposes a novel shadow generation framework based on three-plane representation, introducing pseudo-3D perception into 2D image tasks. By constructing a structured three-plane feature space and introducing a volume rendering mechanism, our method significantly improves the spatial consistency and realism of generated shadows.

[0059] The dual-branch feature extractor of this invention, combined with a local channel-space cross-attention module, can effectively fuse semantic and geometric information from RGB images and mask inputs. Extensive experiments on the DESOBAv2 dataset demonstrate that this invention outperforms existing baselines in both visual fidelity and geometric consistency, validating the effectiveness and generalization ability of the proposed method.

[0060] The overall network model of this invention is relatively compact, which not only reduces storage and overhead but also makes it more suitable for deployment in resource-constrained environments. This invention achieves a good balance between accuracy, speed, and model complexity. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the overall network structure and process of the present invention;

[0062] Figure 2 This is a schematic diagram of the Local Channel-Spatial Cross Attention Module (LCSAM) in the embodiment.

[0063] Figure 3 This is a comparative diagram of the present invention and existing solutions. Detailed Implementation

[0064] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0065] like Figure 1 As shown, the single-frame composite image shadow generation method based on 3D perception of the present invention includes the following steps:

[0066] Step 1: Construct a dual-branch feature extraction module to extract the foreground object mask from the same image. Predicted foreground shading mask Composite images without foreground shadows and background object-shadow mask pair Input the dual-branch feature extraction module together, background object-shadow mask pair Abbreviated as The dual-branch feature extraction module includes a structure-aware branch. and appearance perception branch Structure-aware branch Foreground object mask Predicted foreground shading mask and background object-shadow mask pairs The process is performed to obtain the structural feature map. Appearance perception branch For synthetic images and background object-shadow mask pair The process is performed to obtain the appearance feature map. ;

[0067] Step 2: Construct a Local Channel-Spatial Cross Attention Module (LCSAM) that combines the structural feature map and appearance feature map obtained in Step 1. Figure 1The input is a Local Channel-Spatial Cross-Attention Module (LCSAM), which includes a local channel cross-attention mechanism and a local spatial cross-attention mechanism; the specific method is as follows:

[0068] Step 2.1, in the local channel cross-attention mechanism, first, the appearance feature map... and structural feature diagram Each channel is divided into G groups, and each group includes Each channel has a set of specific convolutions applied to it. Generate query vector Q c and from Generate key K c Sum V c Then Q c With K c The attention weights are obtained by performing dot product operations and softmax normalization sequentially. Then, attention weights are used. Pair value V c Weighted aggregation is performed to obtain the attention-weighted response, which is then reshaped into the original feature dimension to obtain the attention-enhanced feature y. c Finally, attention-enhancing features y c Compared with the original appearance feature map The summation yields the local channel enhancement feature X. m2 ;

[0069] Step 2.2, in the local spatial cross-attention mechanism, for the original appearance feature map X... r and local channel enhancement features X m2 First, the feature map is expanded by a window size w, dividing it into non-overlapping local windows. Then, a query Q is generated for each local window w through convolution. s Key K s Sum V s And calculate local attention weights Then, the obtained local attention weights are... Acting on the value characteristic V s After summarizing the outputs of all local windows, a full-resolution feature map is reconstructed through a folding operation, thus obtaining the local spatial enhancement feature y. s ;

[0070] Step 2.3: Apply the local channel enhancement feature X obtained in Step 2.1 m2 Compared with the local spatial enhancement feature y obtained in step 2.2 s Adding them together yields the final output feature X of the entire Local Channel-Space Cross-Attention Module (LCSAM). LCSAM ;

[0071] Step 3: Based on the synthesized image For each pixel, an orthogonal parallel sampling ray is generated, and each ray samples several sampling points to construct a virtual 3D space, obtaining the spatial coordinates of a set of sampling points. Then, based on the spatial coordinates, the feature X is... LCSAM Reshaped into three orthogonal two-dimensional planes F xy F xz and F yz Then, the three orthogonal two-dimensional planes are input into the pixel-aligned volume rendering module. Utilizing the feature space encoding capability of the three planes, geometric perception is injected into the two-dimensional shadows to achieve virtual 3D space sampling, obtaining the feature F of the three-plane sampling points. t (Subsequently used for shadow prediction and volume rendering);

[0072] Step 4: Based on a progressive two-stage volume rendering strategy, use a dual MLP network to sample the features F of the three planes. t To perform progressive volume rendering, the specific method is as follows:

[0073] First, the features F of the three plane sampling points t Inputting the first MLP, it predicts the shadow mask probability g ∈ [0, 1] and the intermediate features F. s Then the intermediate feature F s Compared with the original three-plane sampling point features F t The concatenated features are then input into a second MLP to predict RGB color c and volume density σ. Next, the shared volume density σ is used to perform volume rendering on the RGB color value c and the shadow mask probability g, following standard α-synthesis rules. Finally, an optimized foreground shadow mask is generated by weighted summation of the point-by-point prediction results. and background images with shadows ;

[0074] Step 5: During training, a composite loss function is used to optimize the dual-branch feature extraction module, the Local Channel-Spatial Cross Attention (LCSAM) module, the pixel-aligned volume rendering module, and the dual MLP network. Including shadow mask prediction loss Image reconstruction loss and perceived loss .

[0075] like Figure 2 As shown, step 2 of this embodiment uses the Local Channel-Spatial Cross-Attention Module (LCSAM) to capture fine-grained cross-channel dependencies in both channel and spatial dimensions, with attention weights... The calculation formula is: ;

[0076] Local channel enhancement feature Xm2 The calculation formula is: ;

[0077] Local attention weights The calculation formula is: ;

[0078] Final output feature X LCSAM The calculation formula is: .

[0079] In step 3 of this embodiment, a virtual three-dimensional volume space is constructed by orthogonally projecting sampling rays from each pixel of the synthesized image plane. This ensures that the sampling process is aligned with the two-dimensional image structure, so that even without explicit three-dimensional cues, spatially consistent three-plane sampling point features F can still be extracted from the pre-computed three-plane representation. t The specific method is as follows:

[0080] Given a synthetic image with a resolution of r × r First, a normalized two-dimensional coordinate grid is generated. Each pixel coordinate (x, y) ∈ U is normalized to the interval [-1, 1]; for each coordinate, an orthogonal sampling ray is defined with the starting point O = [x, y, -1] and the direction of the orthogonal sampling ray D = [0, 0, 1], and perpendicular to the plane of the synthesized image;

[0081] Then, along each ray, S = 32 points are uniformly sampled along the depth direction within the preset range [0, 2]. The depth value of the s-th sampling point is calculated as follows:

[0082] ; ;

[0083] The corresponding three-dimensional coordinates are given by the following formula: P x,y,z = O + (Z s D);

[0084] Each sampling point P is projected onto three orthogonal two-dimensional planes F. xy F xz and F yz superior;

[0085] Finally, the features of the three orthogonal two-dimensional planes are extracted at the projection position using bilinear interpolation, and then fused into a three-plane sampling point feature F. t .

[0086] Step 4: In the dual MLP network, the first MLP predicts the shadow mask probability g ∈ [0, 1] and the intermediate feature F. s The formula is as follows:

[0087] ;

[0088] The formulas for the second MLP to predict RGB color c and volume density σ are as follows:

[0089] ;

[0090] The specific method for volume rendering using volume density σ on RGB color value c and shadow mask probability g is as follows:

[0091] For sampling point i, given its color value Shadow mask probability and depth value First, the absorption coefficient of each segment is calculated based on the Beer-Lambert law. , The α value, representing the amount of light absorbed at each point, is calculated using the following formula:

[0092] ;

[0093] in Indicates the distance between two adjacent sampling points;

[0094] The weight of each sampling point i mentioned above Due to its opacity The result is obtained by multiplying the cumulative transmittance prior to sampling point i, as shown in the following formula:

[0095] ;

[0096] in Set to 1×10 −10 , This represents the absorption coefficient of all points preceding the i-th sampling point along the direction of the light ray;

[0097] Finally, an optimized foreground-shadow mask is generated by weighted summation of the prediction results point by point. and background images with shadows The formula is as follows:

[0098] ;

[0099] .

[0100] To optimize the network, step 5 combines the loss function. Including shadow mask prediction loss Image reconstruction loss and perceptual loss function ;

[0101] Shadow mask prediction loss The L1 loss function is used to penalize pixel-level differences, and the formula is as follows:

[0102] ;

[0103] Image reconstruction loss The generated image is represented using mean squared error loss. With real images The pixel-level difference between them is expressed by the formula:

[0104] ;

[0105] Perceived loss The calculation formula is as follows:

[0106] ;

[0107] in, This indicates the number of steps taken from the pre-trained VGG network. Features extracted from layers;

[0108] Finally, the combined loss function is obtained by weighted summation of the three losses mentioned above. The formula is as follows:

[0109] .

[0110] To verify the effectiveness of the present invention, this embodiment compares with existing solutions in image synthesis and shadow generation, including: ShadowGan, MASK-SG, AR-SG, SGRNet, and SGDision.

[0111] Table 1 summarizes the results of the quantitative comparison.

[0112]

[0113] As shown in Table 1, the present invention achieved optimal or near-optimal performance across all evaluation metrics. In terms of global bit error rate (GB) and local bit error rate (LB), the network model of the present invention achieved the best performance on images without BOS, while maintaining near-optimal results on images with BOS. This highlights the generalization ability of the present invention in handling both simple and complex scenes. Its superior performance stems from the pixel-aligned volume rendering module, which effectively incorporates 3D geometric cues in the scene layout and can synthesize shadows with more accurate shapes and spatial positions, even in the presence of occlusion or complex foregrounds.

[0114] from Figure 3 It can be seen that the shadows generated by this invention have more reasonable shapes and positions. In particular, as Figure 3As shown in the first and third rows, the shadows predicted by this invention are better aligned with the corresponding foreground objects, displaying more accurate positions and sharper boundaries. Furthermore, Figure 3 The fourth line shows that the interaction between the shadow and the ground is more realistic, maintaining consistency in direction and softness.

[0115] Table 2

[0116]

[0117] Table 2 shows that the network model of this invention contains only 8.68 MB of parameters, which contrasts sharply with the 1,457.80 MB required by SGDiffsion. This compactness not only reduces storage and memory overhead but also makes it more suitable for deployment in resource-constrained environments. This invention achieves a good balance between accuracy, speed, and model complexity.

[0118] This embodiment further conducts ablation studies to evaluate the contribution of each key component of the network model of the present invention: the dual-branch feature extraction module (DM), the local channel-space cross attention module (LCSAM), and the pixel-aligned volume rendering module (PR).

[0119] All the above ablation experiments were conducted on the BOS test split of the DESOBAv2 dataset, and the quantitative results are shown in Table 3.

[0120] Table 3

[0121]

[0122] The first row of Table 3 shows a single-branch feature extraction procedure and a two-stage pixel-aligned volume rendering module (PR). The PR module performed the worst across all settings and serves as the baseline. The second row introduces a dual-branch feature extraction module (DM), which significantly improved performance on the baseline, demonstrating its advantages. The third row adds a Local Channel-Spatial Cross-Attention (LCSAM) module to enhance feature fusion, further improving the results. The fourth row uses a LCSAM module and a one-stage pixel-aligned rendering strategy instead of the previously used two-stage rendering strategy. The fifth row of Table 3 introduces… Figure 1 The complete workflow integrates a dual-branch feature extraction module, a local channel-space cross attention module (LCSAM), and a two-stage rendering framework; this complete configuration achieves the best quantization performance, validating the effectiveness and synergy of the proposed method.

Claims

1. A method for generating shadows in a single composite image based on 3D perception, characterized in that, Includes the following steps: Step 1: Construct a dual-branch feature extraction module to extract the foreground object mask from the same image. Predicted foreground shading mask Composite images without foreground shadows and background object-shadow mask pair Input the dual-branch feature extraction module together, background object-shadow mask pair Abbreviated as The dual-branch feature extraction module includes a structure-aware branch. and appearance perception branch Structure-aware branch Foreground object mask Predicted foreground shading mask and background object-shadow mask pairs The process is performed to obtain the structural feature map. Appearance perception branch For synthetic images and background object-shadow mask pair The process is performed to obtain the appearance feature map. ; Step 2: Construct the Local Channel-Spatial Cross-Attention Module (LCSAM). Input the structural feature map and appearance feature map obtained in Step 1 into the LCSAM. The LCSAM includes a local channel cross-attention mechanism and a local spatial cross-attention mechanism. The specific method is as follows: Step 2.1, in the local channel cross-attention mechanism, for the appearance feature map... and structural feature diagram Each channel is divided into G groups, and each group includes Each channel, and each group, applies a convolution operation from... Generate query vector Q c and from Generate key K c Sum V c Then Q c With K c The attention weights are obtained by performing dot product operations and softmax normalization sequentially. Then, attention weights are used. Pair value V c Weighted aggregation is performed to obtain the attention-weighted response, which is then reshaped into the original feature dimension to obtain the attention-enhanced feature y. c Finally, attention-enhancing features y c Compared with the original appearance feature map The summation yields the local channel enhancement feature X. m2 ; Step 2.2, in the local spatial cross-attention mechanism, for the original appearance feature map X... r and local channel enhancement features X m2 First, the feature map is expanded by a window size w, dividing it into non-overlapping local windows. Then, a query Q is generated for each local window w through convolution. s Key K s Sum V s And calculate local attention weights Then, the obtained local attention weights are... Acting on value V s After summarizing the outputs of all local windows, a full-resolution feature map is reconstructed through a folding operation, thus obtaining the local spatial enhancement feature y. s ; Step 2.3: Apply the local channel enhancement feature X obtained in Step 2.1 m2 Compared with the local spatial enhancement feature y obtained in step 2.2 s Adding them together yields the final output feature X of the entire Local Channel-Space Cross-Attention Module (LCSAM). LCSAM ; Step 3: Based on the synthesized image For each pixel, an orthogonal parallel sampling ray is generated, and each ray samples several sampling points to construct a virtual 3D space, obtaining the spatial coordinates of a set of sampling points. Then, based on the spatial coordinates, the feature X is... LCSAM Reshaped into three orthogonal two-dimensional planes F xy F xz and F yz Then, the three orthogonal two-dimensional planes are input into the pixel-aligned volume rendering module. Utilizing the feature space encoding capability of the three planes, geometric perception is injected into the two-dimensional shadows to achieve virtual 3D space sampling, obtaining the feature F of the three-plane sampling points. t ; Step 4: Based on a progressive two-stage volume rendering strategy, use a dual MLP network to sample the features F of the three planes. t To perform progressive volume rendering, the specific method is as follows: First, the features F of the three plane sampling points t Inputting the first MLP, it predicts the shadow mask probability g ∈ [0, 1] and the intermediate features F. s Then the intermediate feature F s Compared with the original three-plane sampling point features F t The features are concatenated and then input into a second MLP to predict RGB color c and volume density σ. Next, the shared volume density σ is used to perform volume rendering on the RGB color value c and the shadow mask probability g, following standard α-synthesis rules. Finally, an optimized foreground shadow mask is generated by weighted summation of the point-by-point prediction results. and background images with shadows ; Step 5: During training, a composite loss function is used to optimize the dual-branch feature extraction module, the Local Channel-Spatial Cross Attention (LCSAM) module, the pixel-aligned volume rendering module, and the dual MLP network. The composite loss function... Including shadow mask prediction loss Image reconstruction loss and perceived loss .

2. The method for generating shadows in a single composite image based on 3D perception according to claim 1, characterized in that, The structure-aware branch of the dual-branch feature extraction module in step 1 The extracted features include object boundaries, spatial configuration, and latent shadow projection regions, as well as the appearance perception branch. The extracted features include color contrast, lighting consistency, and material properties.

3. The method for generating shadows in a single composite image based on 3D perception according to claim 1, characterized in that, Step 2 uses the Local Channel-Spatial Cross-Attention Module (LCSAM) to capture fine-grained cross-channel dependencies in both channel and spatial dimensions, with attention weights. The calculation formula is: ; Local channel enhancement feature X m2 The calculation formula is: ; Local attention weights The calculation formula is: ; Final output feature X LCSAM The calculation formula is: .

4. The method for generating shadows in a single composite image based on 3D perception according to claim 1, characterized in that, Step 3 constructs a virtual three-dimensional volume space by orthogonally projecting sampling rays from each pixel of the synthesized image plane. This ensures that the sampling process is aligned with the two-dimensional image structure, enabling the extraction of spatially consistent three-plane sampling point features F from the pre-computed three-plane representation even in the absence of explicit three-dimensional cues. t The specific method is as follows: Given a synthetic image with a resolution of r × r First, a normalized two-dimensional coordinate grid is generated. Each pixel coordinate (x, y) ∈ U is normalized to the interval [-1, 1]; for each coordinate, an orthogonal sampling ray is defined with the starting point O = [x, y, -1] and the direction D = [0, 0, 1], and is perpendicular to the plane of the synthesized image; Then, along each ray, S = 32 points are uniformly sampled along the depth direction within the preset range [0, 2]. The depth value of the s-th sampling point is calculated as follows: ； ； The corresponding three-dimensional coordinates are given by the following formula: P x,y,z = O + (Z s D); Each sampling point P is projected onto three orthogonal two-dimensional planes F. xy F xz and F yz superior; Finally, the features of the three orthogonal two-dimensional planes are extracted at the projection position using bilinear interpolation, and then fused into a three-plane sampling point feature F. t .

5. The method for generating shadows in a single composite image based on 3D perception according to claim 1, characterized in that, In step 4, the dual MLP network predicts the shadow mask probability g ∈ [0, 1] and the intermediate feature F. s The formula is as follows: ； The formulas for the second MLP to predict RGB color c and volume density σ are as follows: ； The specific method for volume rendering using volume density σ on RGB color value c and shadow mask probability g is as follows: For sampling point i, given its color value Shadow mask probability and depth value First, the absorption coefficient of each segment is calculated based on the Beer-Lambert law. , The α value, representing the amount of light absorbed at each point, is calculated using the following formula: ； in Indicates the distance between two adjacent sampling points; The weight of each sampling point i mentioned above Due to its absorption coefficient The result is obtained by multiplying the cumulative transmittance prior to sampling point i, as shown in the following formula: ； in Set to 1×10 −10 , This represents the absorption coefficient of all points preceding the i-th sampling point along the direction of the light ray; Finally, an optimized foreground-shadow mask is generated by weighted summation of the prediction results point by point. and background images with shadows The formula is as follows: ； 。 6. The method for generating shadows in a single composite image based on 3D perception according to claim 1, characterized in that, Step 5: Composite Loss Function Including shadow mask prediction loss Image reconstruction loss and perceptual loss function ; Shadow mask prediction loss The L1 loss function is used to penalize pixel-level differences, and the formula is as follows: ； Image reconstruction loss The generated shaded background image is represented using mean squared error loss. With real images The pixel-level difference between them is expressed by the formula: ； Perceived loss The calculation formula is as follows: ； in, This indicates the number of steps taken from the pre-trained VGG network. Features extracted from layers; Finally, the composite loss function is obtained by weighted summation of the three losses mentioned above. The formula is as follows: ； Here, λ is a trade-off parameter.

Images