Rendering perception remote sensing target light and shadow controllable generation method

Abstract

The invention provides a rendering perception remote sensing target light and shadow controllable generation method. The method comprises the following steps of 1, preparing training data; 2, constructing a virtual-real combination target generation network based on a diffusion model; step 3, constructing a shadow rendering network based on a Transform; and 4, generating a light and shadow controllable target. Outputting: generating a remote sensing image of a specified type of target on a given background, wherein the target has a shadow conforming to the light and shadow parameters; according to the generation method provided by the invention, a vivid remote sensing target and a controllable illumination shadow can be generated based on a target-free remote sensing background image. Experimental results fully show the effectiveness and generalization ability of the generation method provided by the invention, and the method has wide application prospects and application values.

Description

Technical Field

[0001] This invention relates to a method for controllable generation of light and shadow of remotely sensed targets based on rendering perception, belonging to the fields of computer graphics, computer vision, and remote sensing. Background Technology

[0002] Remote sensing imagery is increasingly used in national defense and civilian sectors such as aerospace defense, disaster monitoring, urban planning, and military reconnaissance. Meanwhile, deep learning technology is widely applied to various target perception and recognition tasks in remote sensing imagery, such as target detection and image segmentation. However, with the rapid development of various general-purpose foundational models, these data-driven tasks increasingly demand large-scale, high-quality datasets. Traditional data acquisition methods based on manual annotation are no longer sufficient due to their time-consuming and labor-intensive nature. Deep generative models offer a cost-effective alternative for acquiring large amounts of remote sensing image data.

[0003] Remote sensing target generation technology is an image processing technique based on deep generative models. It primarily uses deep neural networks to learn the latent distribution of target samples in a dataset, and then samples new target samples from this learned distribution. Controllable target generation technology further enhances the controllability of the targets, such as allowing users to specify the category, layout, and orientation of the generated targets, thus increasing the usability and diversity of the generated data. Currently, commonly used generation methods mainly include those based on generative adversarial networks (GANs) and those based on diffusion models.

[0004] Generative methods based on Generative Adversarial Networks (GANs) construct a generator and a discriminator that engage in adversarial competition and co-learn and evolve to generate realistic data. These methods often inject additional conditions into the generation process through semantic segmentation masks and feature modulation, guiding image generation and generating new targets at specified locations within the image. The advantage of GAN-based methods is that they do not require explicit modeling of the dataset distribution; they can automatically fit the distribution of real-world data through adversarial learning, generating high-quality data. The disadvantages are that the training process is unstable, and the adversarial competition between the generator and discriminator can easily lead to training failure. Furthermore, GANs may face the problem of mode collapse, where the generator only learns to generate a few types of samples and cannot cover the entire dataset distribution.

[0005] Generative methods based on diffusion models learn the latent distribution of data by simulating a progressive noise addition and denoising process, thereby generating new data through sampling. The diffusion model first progressively adds Gaussian noise to a clean image until it becomes a completely noisy image. Then, a neural network is trained to learn the image denoising process, thus achieving the progressive denoising from a purely noisy image to obtain a new clean image, thereby generating new data. The advantages of the diffusion model lie in its stable training process, clear optimization objectives, and the ability to generate detailed, high-resolution samples through multi-step fine-grained denoising. However, diffusion model algorithms that denoise in pixel space are relatively slow. Subsequent work, the latent diffusion model, introduces a latent space into the diffusion model, adding and denoising within the latent space, and also incorporates textual conditional guidance. This significantly improves the generation speed and controllability while maintaining image generation quality, becoming a milestone in generative models and finding widespread application in areas such as text-to-image generation, image inpainting, and cross-modal image super-resolution.

[0006] Shadow generation techniques are generally considered a branch of image reconciliation processing. This technique aims to add realistic shadows to foreground objects in composite images, thereby enhancing the realism of the composite image. In existing methods, most foreground objects are simply overlaid onto the background through copy-paste or rendering. For the former, depth-based generative models are typically used to further generate shadows for the textured object, while for the latter, graphics rendering engines are generally used in a computer graphics context to calculate shadow pixels.

[0007] Generative model-based target shadow generation methods learn shadow distribution features in a data-driven manner, rather than directly relying on physical rules or geometric calculations. These methods typically model shadow generation as an image translation task, using paired training data: synthetic images without shadows and images containing real shadows. Common methods utilize background shadow information from synthetic images to guide the prediction of foreground target shadows; others combine diffusion models, controlling the intensity of generated shadows through modulation information and using post-processing networks to correct color biases. While these methods improve the flexibility of shadow generation to some extent, they are heavily reliant on training data and still suffer from insufficient generalization ability in complex scenes.

[0008] Shadow rendering based on computer graphics refers to a common method for generating shadow effects in virtual 3D scenes using a graphics rendering engine, relying on accurate 3D models and lighting parameters. Existing target shadow generation methods based on rendering have been extensively studied, mainly including methods based on shadow volumes and methods based on shadow mapping. These methods can generate relatively realistic shadow effects when given accurate 3D target models. However, in practical applications, obtaining a large number of accurate 3D target models is quite difficult, and the rendering results often lack the complex conditions of real remote sensing scenes, resulting in significant visual differences from real remote sensing imagery, thus limiting the applicability of this type of method in remote sensing scenarios.

[0009] To address the aforementioned difficulties and challenges, this invention proposes a rendering-aware, controllable target lighting and shadow generation method. It aims to generate diverse and realistic targets based on a real remote sensing background through a two-stage framework of target generation and shadow rendering, and further achieve controllable target shadow generation. Users input a remote sensing background image and a target mask, specifying the desired target shadow direction and illumination intensity. This allows the generation of corresponding targets and shadows conforming to the lighting parameters on the background. This invention innovatively mimics the lighting and shadow rendering behavior of a rendering engine, solving the cross-domain problem between rendering and image transfer methods at both the data and model levels. The invention designs a shadow modulation module and an illumination intensity modulation module, encoding the user-input lighting and shadow parameters into the shadow generation model. This guides the synthesis of shadows in specific directions and the precise adjustment of global illumination intensity, achieving realistic and diverse target generation, highly controllable and high-fidelity target shadow generation and illumination intensity control, and extending the lighting and shadow rendering behavior of graphics rendering engines from three-dimensional model space to two-dimensional image space. Summary of the Invention

[0010] 1. Objective: The objective of this invention is to provide a method for controllable generation of lighting and shadows for remotely sensed targets using rendering perception, solving the problems of difficulty in generating virtual and real targets in remote sensing scenes and the inability to precisely control lighting and shadow conditions. First, this invention designs a method for generating virtual and real targets in typical remote sensing scenes based on a latent diffusion model. By introducing target-level image guidance conditions, virtual targets generated are superimposed at specified locations in the real scene, achieving the generation of virtual and real targets for remote sensing. Next, this invention constructs a controllable shadow generation network based on the Transformer architecture for remote sensing targets, designs a shadow rendering network simulating a high-precision graphics engine, learns relevant knowledge of lighting and shadow rendering behavior, and applies it to the virtual and real generated remote sensing targets, thus forming a two-stage controllable shadow generation framework for remote sensing targets. This invention not only has the ability to generate realistic virtual targets in real scenes, but also can generate physically consistent shadows for targets by manually inputting lighting parameters, while simultaneously controlling the light intensity received by the target, thus overcoming the problems of insufficient control over target lighting and shadows and lack of realistic physical characteristics in the current field of remote sensing image target generation.

[0011] 2. Technical Solution: This invention is achieved through the following technical solution:

[0012] This invention is a rendering-aware, controllable generation method for the lighting and shadow of remotely sensed targets, comprising four steps: training data preparation, constructing a virtual-real combined target generation network based on a diffusion model, constructing a shadow rendering network based on Transformer, and generating the lighting and shadow of the target. The specific steps of this method are as follows:

[0013] Step 1: Training Data Preparation

[0014] This invention proposes a dataset construction method for controllable generation of remote sensing target lighting and shadows to meet the training data requirements of a virtual-real combined target generation network and a shadow rendering network. The process is as follows: Figure 2As shown, for the virtual-real combined target generation network based on the diffusion model, the required training data consists of a series of real remote sensing target images with mask annotations. This invention integrates publicly available remote sensing target datasets and constructs typical remote sensing target image data and their mask annotations, including aircraft, ships, and vehicles, through manual and automatic annotation methods, for training the virtual-real combined target generation network. For the shadow rendering network, the required training data consists of pairs of shadowless and shadowed target images, the mask annotations of the targets themselves, and the lighting parameter information in the shadowed target images. This invention uses a graphics rendering engine to obtain target image data before and after adding lighting. It pre-integrates targetless remote sensing scenes from publicly available datasets as rendering backgrounds and integrates aircraft, ship, and vehicle models from publicly available 3D model datasets as rendering targets. Through the process of building a 3D scene, placing the background and targets, setting scene lighting, and setting the camera perspective, it obtains shadowless-shadowed target image pairs with lighting parameter labels and target masks through rendering, providing data support for the subsequent training of the shadow rendering network.

[0015] Step 2: Construct a virtual-real hybrid target generation network based on a diffusion model

[0016] The core model of the virtual-real hybrid target generation network constructed in this invention is the Latent Diffusion Model. The Latent Diffusion Model first encodes the input image into a low-dimensional representation in the latent space through an encoder, then gradually transforms the probability distribution of the target image into a standard normal distribution through a forward noise addition process, then gradually transforms the noisy image into the newly generated target image through an inverse diffusion process, and finally decodes the generated target image from the latent space to restore it to the pixel space through a decoder.

[0017] Assume the initial image distribution is equivalent to the true data distribution. Let x0 represent the original image, and x... T This represents the result obtained by adding T noise to the original image over T time steps. The forward process is represented by q(x). t |x t-1 This can be represented as ) that is, at each step of the forward pass, by feeding the image x t-1 Adding Gaussian noise to obtain x t The added Gaussian noise has a mean of . The variance is Σ t =β t I, where β t It is a noise adjustment parameter with a value between 0 and 1. The noise addition process can be represented as:

[0018]

[0019] Since the forward process of the diffusion model is a Markov process, from x0 to xT The posterior probability distribution can be expressed as:

[0020]

[0021] To obtain operable closed-form sampling at any time step, a reparameterization technique is introduced: a two-dimensional standard normal distribution variable is introduced. Then through the reparameter β t From x t-1 Get x t :

[0022]

[0023] Similarly, we can obtain from x t-2 Get x t-1 The process:

[0024]

[0025] Combining the two expressions above, we have:

[0026]

[0027] Define α t =1-β t By renormalizing the parameters, it can be transformed into a form containing only one random variable:

[0028]

[0029] Therefore, the state x at any given time can be obtained directly using parameter renormalization techniques. t :

[0030]

[0031] in Therefore, given the initial state x0 of the image and a pre-defined sequence of fixed values, In the case of the standard normal distribution, By sampling a standard noise ∈, the state at time t can be calculated directly in one step. The reverse process of the diffusion model is q(x). t-1 |x t This is a denoising process. If we can obtain the distribution of the inverse diffusion process, we can sample from a Gaussian distribution. The diffusion model reconstructs an "original" sample that follows the initial distribution of the data. The core problem of the diffusion model is how to learn this denoising process.

[0032] In the diffusion model, the state x at time t t It is known that the essence of noise reduction is actually based on x. tCome and learn x t-1 The probability distribution function can be used to implement this process by constructing a denoising neural network. The inverse diffusion process can be modeled as p θ (x t-1 |x t ), where θ is the model parameter of the denoising neural network. The constructed denoising neural network is optimizable. In the latent diffusion model, we use the U-net network model to predict noise.

[0033] Since we add Gaussian noise to the current image at each time step during the forward propagation process, the part filtered out by each denoising step in the back-diffusion process is also Gaussian noise. Theoretically, Gaussian noise can be determined by its mean and variance; therefore, the network model p for the denoising process... θ It can be written in the following form:

[0034]

[0035] Based on the Markov chain, we can obtain the expression for the inverse diffusion process:

[0036]

[0037] To predict the probability distribution of noise, we introduce the posterior diffusion conditional probability:

[0038]

[0039] Considering the posterior diffusion conditional probability, we use Bayes' theorem and substitute x0, x... into the derived values ​​above. t The relationship between them can be obtained as follows:

[0040]

[0041] After constructing the model, this invention uses self-supervised training to train the virtual-real combined target generation model, as follows: Figure 3 As shown. Specifically, based on the target-annotated mask in the training data, the target region in the input image is cropped out and then used as a conditional input into the latent diffusion model after a series of target augmentation operations; the background part of the input image is then stitched together with the target mask after being encoded at the input end, and after the inverse diffusion process, a new image with the original target as a reference is generated in the mask region of the image.

[0042] Step 3: Construct a Transformer-based shadow rendering network

[0043] This invention further constructs a shadow rendering network based on Transformer, the network architecture of which is as follows: Figure 5As shown, this invention is used to generate directionally controllable shadows from shadowless target images and to adjust the intensity of illumination. The shadow rendering network consists of an encoding stage and a decoding stage, with skip connections between the encoder and decoder at the same resolution. Both the encoder and decoder are built based on Transformer modules using a window self-attention mechanism, reducing computational costs while maintaining efficient feature extraction capabilities. In each Transformer module, the feature map sequentially passes through a conditional normalization layer, a window multi-head self-attention layer, a conditional normalization layer, and a feedforward layer for local enhancement to obtain the feature extraction result at the current scale. Then, it is upsampled or downsampled to the next scale to continue the above process. To achieve directionally controllable shadow generation and illumination adjustment, this invention designs shadow modulation and illumination modulation modules respectively. First, this invention divides the light and shadow conditions into two parts: direction and intensity. The illumination direction is controlled by the solar altitude angle and solar azimuth angle. The solar altitude angle β represents the angle between sunlight and the horizontal plane, and the solar azimuth angle... This represents the angle between the northward rotation and the horizontal projection of the sun. These two parameters are encoded by the shadow modulation module. The brightness of the scene and the intensity of the shadows are controlled by the illumination intensity I, which is encoded by the illumination modulation module.

[0044] Specifically, this invention designs a shadow modulation module based on a convolutional neural network. At the input of the shadow modulation module, the values ​​of the solar altitude angle and solar azimuth angle are linearly processed and normalized, then filled into a tensor with the same shape as the target mask. After being concatenated with the target mask, the tensor is input into the convolutional neural network. Different modulation information at different scales is output at different layers of the network, and these are added to the corresponding scale features in the decoder. Figure 6 This demonstrates the superposition process between the modulation information output by the shadow modulation module and the feature map output by the Transformer module.

[0045] On the other hand, this invention designs an illumination modulation module based on a conditional normalization layer. The illumination modulation module is used to control the illumination intensity of the target image, responsible for regulating the brightness of shadows and the reflectivity of the target. The conditional layer normalization operation designed in this invention can be expressed as:

[0046] γ LN ,β LN =MLP(I)

[0047]

[0048] Where, x i The current layer outputs features, μ represents the feature mean, and σ represents the feature mean. 2Let ε represent the feature variance, where ε is a small quantity. To encode the illumination intensity and control the generation process, this invention designs a multilayer perceptron to map the illumination intensity parameter I into a 2×C dimensional vector, which is then decomposed into γ. LN and β LN Two C-dimensional parameters are used as affine transformation parameters in the layer normalization of the above formula, thereby precisely adjusting the overall intensity of image features, where C is the number of feature channels in the window attention layer.

[0049] Step 4: Generating a target with controllable lighting and shadows

[0050] This invention utilizes the virtual-real combined target generation network and shadow rendering network constructed in steps two and three for training, and cascades the two networks after training to achieve target generation with controllable light and shadow.

[0051] In the virtual-real combined target generation stage, this invention proposes a virtual-real combined target generation network based on a diffusion model. Through the introduction of semantic masks and a self-supervised training process, virtual remote sensing targets are generated on a real remote sensing background. The input of this stage is a background image, a target mask, and a target reference image to be generated, and the output is a shadowless target image with virtual targets generated on the background.

[0052] In the shadow generation stage, this invention proposes a shadow rendering network. Through a shadow modulation module and an illumination modulation module, it achieves directional controllable shadow generation and illumination intensity adjustment on the targetless shadow image generated in the previous stage. This stage takes into account the solar azimuth angle, solar altitude angle, illumination intensity parameters, and the shadowless target image and its target mask generated in the virtual-real combined target generation stage. The output is a target image with generated shadows, where the shadow direction and illumination intensity match the input parameters.

[0053] 3. Advantages and effects:

[0054] This invention presents a rendering-aware, controllable light and shadow generation method for remote sensing targets. By designing a virtual-real hybrid target generation network and a light and shadow controllable shadow rendering network, it simulates the light and shadow rendering behavior of a high-precision graphics rendering engine, achieving virtual-real hybrid remote sensing target generation. The generated results cover various typical remote sensing scenes and targets, and can further render realistic and reasonable shadows for the generated targets based on user-input lighting direction and intensity parameters. The target images generated using this method maintain textural and spatial coherence and coordination between the background and the generated target, while possessing manually controllable shadow direction and lighting intensity. This significantly improves the realism and usability of the generated data, providing more realistic synthetic data for training remote sensing image perception and recognition tasks, and has significant application value. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall process proposed in this invention.

[0056] Figure 2 This is a schematic diagram of the self-supervised training process in the target generation stage that combines virtual and real elements.

[0057] Figure 3 This is a schematic diagram of the training process of the shadow rendering network using a rendering engine.

[0058] Figure 4 It is the process of compositing rendering data.

[0059] Figure 5 It is the process of superimposing the output information of the shadow modulation module.

[0060] Figure 6 It is the model framework for shadow rendering networks.

[0061] Figure 7 It is a target generation result with controllable lighting and shadows. Detailed Implementation

[0062] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0063] This invention is based on the PyTorch framework and implemented using the Python language. Specifically, this invention first prepares training data, then constructs and trains a virtual-real combined target generation network and a shadow rendering network, and finally uses the trained models to realize target shadows with controllable target generation direction and intensity in real remote sensing scenes.

[0064] The server hardware environment consists of a host equipped with an Intel(R) Xeon(R) Platinum 8336C CPU@2.30GHz and eight NVIDIA GeForce GTX 4090 graphics cards, each with a video memory capacity of 24.0GB.

[0065] The specific steps of this method are as follows:

[0066] Step 1: Training Data Preparation

[0067] This invention constructs datasets based on the training data characteristics of the virtual-real combined target generation model and the shadow rendering network, respectively.

[0068] For a virtual-real hybrid target generation network based on a diffusion model, this invention uses publicly available remote sensing target detection datasets DOTA and DIOR to acquire visible light remote sensing ground object target data from various sensors such as Jilin-1 and Gaofen-2. First, image slice data of aircraft, ship, and vehicle targets are constructed, and then target masks are labeled. For vehicle targets with low resolution, manual labeling is used, while for aircraft and ship targets, the open-source semantic segmentation large model SAM is used to achieve semi-automatic acquisition of target masks, thereby constructing the image mask pair data required for training the target generation model.

[0069] For shadow rendering networks, this invention utilizes the PyRender graphics rendering engine to automatically synthesize data pairs of images of objects without shadows and images of targets with shadows. Specifically, this invention uses the PyRender tool to configure a 3D scene containing a single aircraft, ship, or vehicle target, and then renders the same scene under different lighting conditions to obtain training image pairs, generating training data with fine-grained shadow details. The rendering process includes four steps: background image setting, 3D model configuration, camera angle setting, and lighting parameter adjustment. First, a suitable background image is selected from the LEVIR and DIOR datasets as a planar reference. Next, a 3D model is selected and loaded from the ShapeNet model library. The camera is fixed at a top-down view position perpendicular to the plane. Finally, lighting parameters including fixed ambient light and adjustable directional light are configured. The angle of the directional light is controlled by the elevation angle and azimuth angle, which affect the shadow direction of the rendered result, while its intensity controls the light intensity. After configuration, rendering begins, generating an RGB image and depth map of the rendered scene. By using the depth map to obtain the object mask, and translating the object mask along the lighting direction, an approximate shadow mask can be obtained, providing a supervision signal for the subsequent training of the shadow rendering network.

[0070] Step 2: Construct a virtual-real hybrid target generation network based on a diffusion model

[0071] The core model of the virtual-real hybrid target generation network constructed in this invention is the Latent Diffusion Model. The Latent Diffusion Model first encodes the input image into a low-dimensional representation in the latent space through an encoder, then gradually transforms the probability distribution of the target image into a standard normal distribution through a forward noise addition process, then gradually transforms the noisy image into the newly generated target image through an inverse diffusion process, and finally decodes the generated target image from the latent space to restore it to the pixel space through a decoder.

[0072] Assume the initial image distribution is equivalent to the true data distribution. Let x0 represent the original image, and x... T This represents the result obtained by adding T noise to the original image over T time steps. The forward process can be represented as q(x t|x t-1 That is, at each forward step, by feeding the image x t-1 Adding Gaussian noise to obtain x t The added Gaussian noise has a mean of . The variance is ∑ t =β t I, the noise addition process can be represented as:

[0073]

[0074] Since the forward process of the diffusion model is a Markov process, from x0 to x T The posterior probability distribution can be expressed as:

[0075]

[0076] To obtain operable closed-form sampling at any time step, a reparameterization technique is introduced: a two-dimensional standard normal distribution variable is introduced. Then through the reparameter β t From x t-1 Get x t :

[0077]

[0078] Similarly,

[0079]

[0080] Combining the two expressions above, we have:

[0081]

[0082] Define α t =1-β t By renormalizing the parameters, it can be transformed into a form containing only one random variable:

[0083]

[0084] Therefore, the state x at any given time can be obtained directly using parameter renormalization techniques. t :

[0085]

[0086] in Therefore, given the initial state x0 of the image and a pre-defined sequence of fixed values, In the case of the standard normal distribution, By sampling a standard noise ε, the result can be calculated directly in one step. t The state at time t. The reverse process q(x) in the diffusion model.t-1 |x t This is a denoising process. If we can obtain the distribution of the inverse diffusion process, we can sample from a Gaussian distribution. The diffusion model reconstructs an "original" sample that follows the initial distribution of the data. The core problem of the diffusion model is how to learn this denoising process.

[0087] In the diffusion model, the state x at time t t It is known that the essence of noise reduction is actually based on x. t Come and learn x t-1 The probability distribution function of the inverse diffusion process can be modeled as p. θ (x t-1 |x t ), where θ is the network model parameter. The constructed network is optimizable. In the latent diffusion model, we use the U-net network to predict noise.

[0088] Since we add Gaussian noise to the current image at each time step during the forward propagation process, the part filtered out in each denoising step during the backdiffusion process is also Gaussian noise. Theoretically, Gaussian noise can be determined by its mean and variance; therefore, model p... θ It can be written in the following form:

[0089]

[0090] Based on the Markov chain, we can obtain the expression for the inverse diffusion process:

[0091]

[0092] To predict the probability distribution of noise, we introduce the posterior diffusion conditional probability:

[0093]

[0094] Considering the posterior diffusion conditional probability, we use Bayes' theorem and substitute x0, x... into the derived values ​​above. t The relationship between them can be obtained as follows:

[0095]

[0096] After constructing the model, this invention employs self-supervised training to train the virtual-real combined target generation model. Specifically, based on the target-annotated mask in the training data, the target region in the input image is cropped out and then used as a conditional input into the latent diffusion model after a series of target augmentation operations. The background portion of the input image is then stitched together with the target mask after being encoded at the input end, and after an inverse diffusion process, a new image with the original target as a reference is generated in the mask region of the image.

[0097] Step 3: Construct a Transformer-based shadow rendering network

[0098] This invention further constructs a Transformer-based shadow rendering network to generate directionally controllable shadows from shadowless target images and to adjust the illumination intensity. The shadow rendering network consists of an encoding stage and a decoding stage, with skip connections between the encoder and decoder at the same resolution. Both the encoder and decoder are built based on Transformer modules employing a window self-attention mechanism, maintaining efficient feature extraction capabilities while reducing computational costs. In each Transformer module, the feature map sequentially passes through a conditional normalization layer, a window multi-head self-attention layer, a conditional normalization layer, and a feedforward layer for local enhancement to obtain the feature extraction result at the current scale. Then, it is upsampled or downsampled to the next scale for further processing. To achieve directionally controllable shadow generation and illumination adjustment, this invention designs separate shadow modulation and illumination modulation modules. First, this invention divides the lighting conditions into two parts: direction and intensity. The illumination direction is controlled by the solar altitude angle and solar azimuth angle. The solar altitude angle β represents the angle between sunlight and the horizontal plane, and the solar azimuth angle... This represents the angle between the northward rotation and the horizontal projection of the sun. These two parameters are encoded by the shadow modulation module. The brightness of the scene and the intensity of the shadows are controlled by the illumination intensity I, which is encoded by the illumination modulation module.

[0099] Specifically, this invention designs a shadow modulation module based on a convolutional neural network. At the input of the shadow modulation module, the values ​​of the solar altitude angle and solar azimuth angle are linearly processed and normalized, then filled into a tensor with the same shape as the target mask. This tensor is then concatenated with the target mask and input into the convolutional neural network. Different layers of the network output modulation information at different scales, which are then added to the corresponding scale features in the decoder, such as... Figure 6 As shown.

[0100] On the other hand, this invention designs an illumination modulation module based on a conditional normalization layer. The illumination modulation module is used to control the illumination intensity of the target image, responsible for regulating the brightness of shadows and the reflectivity of the target. The conditional layer normalization operation designed in this invention can be expressed as:

[0101]

[0102] Where, x i The current layer outputs features, μ represents the feature mean, and σ represents the feature mean. 2 Let ∈ represent the feature variance, where ∈ is a small quantity. To encode the illumination intensity and control the generation process, this invention designs a multilayer perceptron to map the illumination intensity parameter into a 2×C dimensional vector, which is then decomposed into γ... LN and β LN Two C-dimensional parameters are used as affine transformation parameters in the layer normalization of the above formula, thereby precisely adjusting the overall intensity of image features, where C is the number of feature channels in the window attention layer.

[0103] Step 4: Generating a target with controllable lighting and shadows

[0104] This invention first trains the virtual-real combined target generation network constructed in step two and the shadow rendering network constructed in step three, and then cascades the two networks after training to achieve target generation with controllable light and shadow.

[0105] In the virtual-real combined target generation stage, this invention proposes a virtual-real combined target generation network based on a diffusion model. Through the introduction of semantic masks and a self-supervised training process, virtual remote sensing targets are generated on a real remote sensing background. The input of this stage is a background image, a target mask, and a target reference image to be generated, and the output is a shadowless target image with virtual targets generated on the background.

[0106] In the shadow generation stage, this invention proposes a shadow rendering network. Through a shadow modulation module and an illumination modulation module, it achieves directional controllable shadow generation and illumination intensity adjustment on the targetless shadow image generated in the previous stage. This stage takes into account the solar azimuth angle, solar altitude angle, illumination intensity parameters, and the shadowless target image and its target mask generated in the virtual-real combined target generation stage. The output is a target image with generated shadows, where the shadow direction and illumination intensity match the input parameters.

[0107] Experimental results: Figure 7 The diagram demonstrates the results of target generation with controllable lighting and shadows. The upper left corner shows a schematic diagram of the entire generation process of this invention. The virtual-real hybrid target generation network trained in step two generates a shadowless target on a real remote sensing background. Then, the shadow rendering network trained in step three generates shadows with controllable light intensity and direction on the shadowless target image. The upper right corner shows the effect of the shadow rendering network on adjusting the target's lighting intensity, representing the generation process from low to high lighting intensity from left to right. Figure 7The section below shows the generation results of the aircraft target under various solar altitude and azimuth parameters, demonstrating the fine-grained controllability of the shadow rendering network over the direction of shadow generation.

Claims

1. A method for controllable generation of lighting and shadow of remotely sensed targets based on rendering perception, characterized in that: Includes the following steps: Step 1: Training Data Preparation We construct a dataset of remote sensing target images and their masks for a virtual-real combined target generation network, and use a graphics rendering engine to synthesize shadowless-shadowed target image pairs with illumination parameter labels for the training and validation process of the shadow rendering network. The dataset includes typical remote sensing target image data of aircraft, ships, and vehicles, along with their mask annotations. Step 2: Construct a virtual-real hybrid target generation network based on a diffusion model The core model of the virtual-real hybrid target generation network is a latent diffusion model. The encoder encodes the input image into a low-dimensional representation in the latent space. After a forward noise addition process, the probability distribution of the target image is gradually transformed into a standard normal distribution. Then, the noisy image is gradually transformed into a clean image through an inverse diffusion process. Finally, the decoder decodes the generated target image from the latent space to restore it to the pixel space. The virtual-real hybrid target generation network is trained in a self-supervised manner to learn the dataset distribution and generate new targets in the remote sensing background through sampling. Step 3: Construct a Transformer-based shadow rendering network A Transformer-based shadow rendering network is constructed. The main framework of the network includes an encoder stage and a decoder stage, which are composed of Transformer modules based on window attention mechanism. In addition, it includes a shadow modulation module for controlling shadow direction and a light modulation module for controlling light intensity. This network is used to render controllable lighting and shadow effects for generated shadowless targets. Step 4: Generating a target with controllable lighting and shadows The virtual-real combined target generation network and the shadow rendering network are trained separately, and the trained virtual-real combined target generation network and the shadow rendering network are cascaded to achieve remote sensing target generation with controllable light and shadow.

2. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 1, characterized in that: In step one, specifically: the training data of the shadow rendering network is synthesized by a graphics rendering engine. After loading the remote sensing background image and the three-dimensional target model, the image pairs with shadow direction and intensity labels are generated by configuring the lighting parameters.

3. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 1, characterized in that: In step two, specifically: the virtual-real combined target generation network is constructed based on a latent diffusion model, including an encoder, a latent space diffusion process, and a decoder; the diffusion process includes forward noise addition and reverse noise reduction processes, and noise is predicted through a U-Net network to achieve target generation in the mask region; during training, the target region in the image is used as a conditional input after image augmentation, and the background region of the input image is compressed into the latent space after being processed by the encoder and stitched with the mask to guide the generation of targets within a specified range.

4. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 2, characterized in that: The target image x0 to be generated follows a probability distribution q(x0), with x T Let q(x) represent the result obtained by adding noise T times to the original image over T time steps. Then the forward process can be represented as q(x) t |x t-1 That is, at each forward step, by feeding the image x t-1 Adding Gaussian noise to obtain x t The added Gaussian noise has a mean of . The variance is Σ t =β t I,β t These are predefined hyperparameters, linear interpolations ranging from 0.0001 to 0.

02. The noise addition process is represented as:

5. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 4, characterized in that: To obtain operable closed-form sampling at any time step, a reparameterization technique is introduced to obtain the state x at any time. t : in α t =1-β t Let ∈ be a random noise vector sampled from the standard normal distribution N(0,I). The above equation describes the state x at time t. t The relationship with the initial state x0 is such that when t is large enough, there are enough noise-adding steps, and x... t The closer it is to the standard normal distribution.

6. A method for controllable generation of remote sensing target lighting and shadow based on rendering perception, as described in claim 1 or 5, characterized in that: The denoising process involves constructing a neural network based on x. t Learning x t-1 The probability distribution function is expressed as: Based on the Markov chain, the expression for the inverse diffusion process is obtained. To predict the probability distribution of noise, we introduce the posterior diffusion conditional probability: Considering the posterior diffusion conditional probability, we use Bayes' theorem and substitute x0, x... into the derived values ​​above. t The relationship between them can be obtained as follows:

7. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 1, characterized in that: In step three, specifically: the input to the shadow rendering network is a remote sensing target image without shadows and its target mask, as well as the solar altitude angle, solar azimuth angle, and illumination intensity parameters input by the user. The output is a remote sensing target image with shadows, and the direction of the shadows and the illumination intensity of the image conform to the input parameters. Specifically, the shadow rendering network adopts an encoder-decoder architecture, built based on a Transformer module with a window self-attention mechanism, and the window size uses a fixed value. The shadow modulation module is built based on a convolutional neural network. This module takes the solar altitude angle, solar azimuth angle, and target mask as input, encodes them, and obtains multi-scale modulation information from the outputs of different convolutional layers, which is added to the output features of each Transformer module in the decoder. The illumination modulation module is based on conditional layer normalization. This module maps the illumination intensity parameters to affine transformation parameters through a multilayer perceptron, thereby regulating the feature intensity.

8. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 7, characterized in that: The transformer module employs a window attention mechanism, given a two-dimensional feature map. H and W represent the height and width of the feature map, respectively. Feature X is divided into M×M non-overlapping windows. Then, the features within each window are flattened and transposed to obtain... Feature vectors: X={X 1 ,X 2 ,…,X N },N=HW / M 2 Among them, X i Let represent the i-th window. Next, multi-head self-attention is calculated for the flattened features in each window. The self-attention of the k-th head in the window can be expressed as follows: in, These represent the projection matrices of the query, key, and value for the k-th header, respectively. The output of the k-th head, self-attention calculation can be expressed as: Where d k The dimension representing each attention head.

9. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 7, characterized in that: The shadow modulation module first adjusts the solar azimuth angle. The sine and cosine values ​​of the azimuth angle are calculated and used as modulation parameters, thereby achieving a linear mapping between the modulation parameters and the shadow azimuth; After being normalized to the solar elevation angle β, it is concatenated with the target mask and processed by a convolutional network to output a multi-scale modulation tensor. After each Transformer module in the decoder stage of the shadow rendering network, the features output by the module are flattened and added to the multi-scale modulation tensor output by the shadow modulation module. The flattened features are then reshaped back into the feature shape in the transformer module and input into the next module for further processing.

10. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 7, characterized in that: The illumination modulation module controls the illumination intensity of the generated image by using conditional layer normalization before and after the window attention mechanism in each transformer block. Specifically, the conditional layer normalization maps the illumination intensity I to γ ​​through a multilayer perceptron (MLP). LN and β LN Two parameters, used for the affine transformation in conditional layer normalization, are expressed as: γ LN ,β LN =MLP(I) Where, x i The current layer outputs features, μ represents the feature mean, and σ represents the feature mean. 2 Represents the characteristic variance, where ∈ is a small quantity.

11. The method for controllable generation of remote sensing target lighting and shadow based on rendering perception according to claim 1, characterized in that: In step four, specifically: the virtual-real combined target generation network takes a targetless remote sensing background image, a target mask, and a reference target image as input, and outputs a shadowless target image. The target mask used in the inference stage is provided by the user and can be obtained from mask annotations of existing data or generated through existing generative models. The shadow rendering network takes a shadowless target image, a target mask, a solar altitude angle, an azimuth angle, and an illumination intensity as input, and outputs a target image with shadows. By cascading the virtual-real combined target generation network and the shadow rendering network, it is possible to generate a specified remote sensing target and a shadow with controllable direction and intensity on a given remote sensing background image.

Images