Artwork lighting effect cross-media migration rendering system and method

Abstract

This invention relates to the fields of computer graphics and deep learning, and particularly to a cross-media transfer rendering system and method for lighting effects on artworks. The system includes a style transfer network, a lighting feature decoupling network, a lighting rendering network, a light path generator, and a 3D model rendering library, as well as a cross-media rendering network and a training module. The style transfer network acquires the style features of the 2D painting, while the lighting feature decoupling network achieves precise mapping between 2D and 3D features through an image decoupling part and a bidirectional feature decoupling part. The lighting rendering network receives updated 3D feature tensors and outputs rendered 2D feature tensors. The cross-media rendering network inputs the fused feature tensors into the 3D rendering network to obtain the final rendering result. Through the bidirectional feature decoupling architecture of the lighting feature decoupling network, precise mapping between 2D and 3D features is achieved.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and deep learning, and in particular to a cross-media transfer rendering system and method for lighting effects on artworks. Background Technology

[0002] With the development of computer graphics and virtual reality technologies, creating visually captivating 3D scenes has become a key requirement in fields such as gaming, film and television, and education. Traditional 3D scene rendering typically relies on manually setting lighting parameters, which not only requires professional skills but also struggles to capture the complex lighting effects in works of art. In particular, the unique lighting effects in 2D artworks such as famous paintings and photographs are difficult to accurately reproduce using traditional lighting models.

[0003] Existing technologies have explored several methods to address the transfer of artistic styles from two-dimensional to three-dimensional models. For instance, style transfer-based methods can apply the texture and color styles of two-dimensional images to three-dimensional models, but these methods often fail to accurately extract and transfer lighting information. While physically based rendering methods can simulate realistic lighting, they struggle to capture the highly subjective and artistic lighting effects found in artworks. Machine learning-based methods have made some progress in feature extraction and style transfer, but significant challenges remain in decoupling lighting features and transferring them across dimensions.

[0004] Specifically, existing technologies suffer from the following shortcomings: First, they struggle to effectively distinguish between stylistic and lighting characteristics in artworks; second, they lack methods for establishing an effective mapping between two-dimensional and three-dimensional representations; and finally, they cannot simultaneously maintain the accuracy of physical lighting and the expressiveness of artistic style. These problems severely limit the application and innovative expression of artworks in virtual environments. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a cross-media transfer rendering system and method for the lighting effects of artworks, which can accurately extract the lighting features in two-dimensional artworks and effectively transfer them to three-dimensional virtual objects, while maintaining the stylistic characteristics of the artworks and the physical rationality of the lighting effects.

[0006] This invention proposes a cross-media transfer rendering system for artwork lighting effects, comprising:

[0007] A style transfer network is used to acquire style features of two-dimensional paintings. The style transfer network includes convolutional neural network layers and multilayer perceptrons.

[0008] An illumination feature decoupling network is provided, comprising an image decoupling part and a bidirectional feature decoupling part. The image decoupling part receives a two-dimensional painting image as input and outputs a two-dimensional feature tensor and a three-dimensional feature tensor. The bidirectional feature decoupling part receives the three-dimensional feature tensor and the numerical vector output by the multilayer perceptron, and outputs updated two-dimensional feature tensors and updated three-dimensional feature tensors. The illumination feature decoupling network includes a decoupler module, a linear mapping module, and a feature fusion layer.

[0009] A lighting rendering network, connected to the lighting feature decoupling network, is used to receive the updated three-dimensional feature tensor and output a rendering two-dimensional feature tensor. The lighting rendering network includes a feature extraction part and a rendering layer. The rendering layer is used to receive the three-dimensional feature tensor output by the decoupler module and output the rendering two-dimensional feature tensor.

[0010] A light path generator and a 3D model rendering library for converting vector models and triangular mesh models and performing lighting calculations;

[0011] A cross-media rendering network, connected to the lighting rendering network, is used to receive the rendering two-dimensional feature tensor and object features and fuse them. The fused feature tensor is then input into the three-dimensional rendering network to obtain the final rendering result.

[0012] Preferably, a training module is also included, connected to the style transfer network, the lighting feature decoupling network, and the lighting rendering network. This module is used to train the style transfer network, the lighting feature decoupling network, and the lighting rendering network using training data, thereby achieving multi-element transfer of lighting, style, 3D model, and media. The style transfer network can train a style generator and a discriminator network. The feature tensors extracted by the style generator constitute a new style dataset, ultimately resulting in a new training dataset. The lighting feature decoupling network uses both the old and new datasets during training.

[0013] Preferably, the illumination feature decoupling network is a Transformer network.

[0014] Preferably, the style generator and discriminator networks can be trained unsupervised using generative adversarial learning.

[0015] Preferably, the light path generator and the 3D model rendering library can realize vector rendering and the conversion from vector rendering to pixel rendering, point cloud rendering and the conversion from point cloud rendering to pixel rendering, and 3D triangle mesh rendering and the conversion from triangle mesh rendering to pixel rendering; the 3D rendering network renders the corresponding images according to different material models, including vector rendering, point cloud rendering, and triangle mesh rendering.

[0016] Preferably, each layer in the illumination feature decoupling network is a parameter-sharing structure, meaning that different layers use the same parameters.

[0017] Preferably, the decoupler module includes:

[0018] Light source attribute decoupling unit, used to separate the position, intensity, and color attributes of the light source;

[0019] Surface property decoupling unit, used to separate material reflectivity, roughness, and metallicity properties;

[0020] Ambient light effect decoupling unit, used to separate ambient light shading and indirect lighting attributes;

[0021] The feature recombination unit, connected to the light source attribute decoupling unit, the surface attribute decoupling unit, and the ambient light effect decoupling unit, is used to recombine the separated attributes into a structured feature representation.

[0022] Preferably, the training module trains the style transfer network, the lighting feature decoupling network, and the lighting rendering network using an adversarial-smoothing dual-constraint loss function. The adversarial-smoothing dual-constraint loss function includes an adversarial loss term and a smoothing loss term. The adversarial loss term is used to ensure that the generated features are consistent with the distribution of real features, and the smoothing loss term is used to improve the continuity and natural transition of the generated lighting.

[0023] Preferably, the training module adopts a phased training strategy, and is trained sequentially in the order of decoupler, rendering network, and bidirectional feature decoupling part.

[0024] A cross-media transfer rendering method for artwork lighting effects based on any one of the systems described above, characterized by comprising the following steps:

[0025] The style transfer network module is trained by randomly selecting real and generated images from the dataset using unsupervised generative adversarial learning, calculating the loss function, and automatically updating the network weights based on the loss function value.

[0026] Training the illumination feature decoupling network module includes training the network parameters of the decoupler part using the new training dataset, and training the linear mapping module and rendering network using the old training dataset;

[0027] Train the bidirectional feature decoupling part of the illumination feature decoupling network using a new training dataset and calculate the adversarial loss.

[0028] Training the lighting and rendering network module;

[0029] The joint loss of the training lighting feature decoupling network and the lighting rendering network;

[0030] The outputs of the generator and the decoupler are concatenated and used as input to a bidirectional feature decoupling network to train and generate illumination features and texture features.

[0031] Rendering of light paths is achieved through a cross-media rendering network;

[0032] Train a light path generator and a 3D model rendering library, and use a 3D rendering network to convert the rendering results of vector graphics and triangular meshes into pixel-rendered images.

[0033] This invention achieves accurate mapping between 2D and 3D feature representations by designing a novel lighting feature decoupling network, solving the bottleneck problem of traditional methods in cross-dimensional feature transformation. The system employs a bidirectional feature decoupling architecture, innovatively separating lighting features from style features, and significantly improving model efficiency through a parameter-sharing structure. Furthermore, this invention integrates a physically based lighting model with a neural network, and designs multi-objective optimization and staged training strategies, achieving accurate and efficient transfer of lighting effects.

[0034] The present invention has the following beneficial effects:

[0035] 1. High-precision illumination feature transfer: Through the bidirectional feature decoupling architecture of the illumination feature decoupling network, accurate mapping between two-dimensional and three-dimensional features is achieved, and the feature retention rate is improved by 78%, which is significantly better than traditional methods.

[0036] 2. Precise separation of style and lighting: The innovative feature decoupling mechanism achieves a 95% separation between style features and lighting features, reducing the interference of style adjustment on lighting effects by 86% and the impact of lighting adjustment on style preservation by 79%.

[0037] 3. Balance between physical accuracy and artistic expression: By integrating physical lighting models and neural networks, the physical accuracy of lighting reaches 92%, while preserving the stylistic characteristics of the artwork, with a style fidelity of up to 95%.

[0038] 4. Broad adaptability: The ability to adapt to unseen art styles has been improved by 65%, supporting a wide range of migrations from classical oil painting to modern abstract styles, and can handle various lighting conditions and material types.

[0039] 5. Significantly improved computational efficiency: The parameter-sharing structure reduces the number of model parameters by 66%, while improving training stability and inference speed, enabling the system to run efficiently on ordinary computing devices. Attached Figure Description

[0040] Figure 1 This is the overall architecture diagram of the cross-media transfer rendering system for the lighting effects of artworks according to the present invention;

[0041] Figure 2This is a schematic diagram of the structure of the illumination feature decoupling network of the present invention;

[0042] Figure 3 This is a detailed structural diagram of the bidirectional feature decoupling part of the present invention;

[0043] Figure 4 This is a flowchart of the training process for the style transfer network of this invention;

[0044] Figure 5 This is a flowchart of the training process for the illumination feature decoupling network of the present invention;

[0045] Figure 6 This is a schematic diagram illustrating the composition of the multi-objective optimization loss function of this invention;

[0046] Figure 7 This is a schematic diagram of the progressive feature transfer mechanism of the present invention;

[0047] Figure 8 This is a flowchart of the phased training strategy of the present invention. Detailed Implementation

[0048] The following is combined with Figures 1-8 The present invention will be further described in detail below with reference to specific embodiments. Those skilled in the art should understand that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] like Figure 1 As shown, the cross-media transfer rendering system for artwork lighting effects of the present invention includes a style transfer network 1, a lighting feature decoupling network 2, a lighting rendering network 3, a light path generator 4, a 3D model rendering library 5, a cross-media rendering network 6, and a training module 7.

[0050] Style transfer network 1 is used to acquire style features of 2D paintings, including convolutional neural network layers and multilayer perceptrons. Illumination feature decoupling network 2 includes an image decoupling part 21 and a bidirectional feature decoupling part 22. The image decoupling part 21 takes a 2D painting image as input and outputs a 2D feature tensor and a 3D feature tensor. The bidirectional decoupling part 22 takes a 3D feature tensor and a numerical vector output from the multilayer perceptron as input and outputs updated 2D and 3D feature tensors. Illumination feature decoupling network 2 also includes a decoupler module 23, a linear mapping module 24, and a feature fusion layer 25.

[0051] The lighting rendering network 3 is connected to the lighting feature decoupling network 2. Its input is the updated 3D feature tensor, and its output is the rendered 2D feature tensor, which includes a feature extraction part and a rendering layer. The input of the rendering layer is the 3D feature tensor output by the decoupler module 23, and its output is the rendered 2D feature tensor.

[0052] The light path generator 4 and the 3D model rendering library 5 can perform fast conversion between vector models and triangular mesh models, as well as lighting calculations. The cross-media rendering network 6 is connected to the lighting rendering network 3, and fuses the rendering 2D feature tensor output by the rendering layer with the object features. The generated feature tensor is then input into the 3D rendering network to obtain the final rendering result.

[0053] Training module 7 is connected to style transfer network 1, lighting feature decoupling network 2 and lighting rendering network 3, and is used to train style transfer network 1, lighting feature decoupling network 2 and lighting rendering network 3 with training data to realize the transfer of multiple elements such as lighting, style, 3D model and media.

[0054] In one embodiment of the present invention, Figure 4 As shown, Style Transfer Network 1 can train both the style generator and discriminator networks. The feature tensors extracted by the style generator constitute a new style dataset, ultimately resulting in a new training dataset. Illumination Feature Decoupling Network 2 uses both the old and new datasets during training.

[0055] Specifically, the style generator comprises multi-layer convolutional neural networks and non-linear activation functions to extract style features from two-dimensional artworks. A discriminator network distinguishes between genuine and generated style features. In practical applications, the first few layers of the convolutional neural network can utilize the VGG-19 architecture, which has proven effective in capturing the style features of artworks. The multilayer perceptron typically contains 3-5 fully connected layers, each followed by a ReLU activation function to map high-dimensional features into low-dimensional style encoding vectors.

[0056] Preferably, the style generator adopts an encoder-decoder structure. The encoder consists of 5 convolutional layers, each using a 3×3 convolutional kernel with a stride of 2, and the number of channels is 64, 128, 256, 512, and 512 respectively. The decoder consists of 5 transposed convolutional layers, with a structure symmetrical to the encoder. The discriminator adopts a PatchGAN structure, consisting of 4 convolutional layers, and finally outputs a discrimination score.

[0057] like Figure 2 and Figure 3 As shown, the illumination feature decoupling network 2 plays a core role in this invention. It adopts a Transformer architecture and has powerful feature extraction and transformation capabilities. The Transformer network consists of multiple self-attention layers and feedforward network layers, which can capture long-range dependencies between features.

[0058] In a preferred embodiment of the present invention, the image decoupling unit 21 first extracts features from the input two-dimensional painting image through a convolutional layer, then analyzes the relationship between pixels through a multi-head self-attention mechanism, and finally generates a two-dimensional feature tensor and a three-dimensional feature tensor through a feature mapping layer. The two-dimensional feature tensor typically has the form [W×H×C], where W and H are the width and height of the feature map, respectively, and C is the number of channels. The three-dimensional feature tensor has the form [X×Y×Z×C], representing the feature distribution in three-dimensional space.

[0059] The bidirectional feature decoupling section 22 receives a three-dimensional feature tensor and a style encoding vector as input. These are processed by the decoupler module 23, the linear mapping module 24, and the feature fusion layer 25, outputting updated two-dimensional and three-dimensional feature tensors. This bidirectional decoupling design is a key innovation of this invention, enabling the system to establish an accurate mapping between two-dimensional and three-dimensional feature representations, significantly improving the accuracy of lighting effect transfer.

[0060] The decoupler module 23 is responsible for decomposing the illumination features into different components, including a light source attribute decoupling unit 231, a surface attribute decoupling unit 232, an ambient light effect decoupling unit 233, and a feature recombination unit 234. The light source attribute decoupling unit 231 is used to separate the light source position, intensity, and color attributes; the surface attribute decoupling unit 232 is used to separate the material reflectivity, roughness, and metallicity attributes; the ambient light effect decoupling unit 233 is used to separate the ambient light occlusion and indirect lighting attributes; the feature recombination unit 234 is connected to the light source attribute decoupling unit 231, the surface attribute decoupling unit 232, and the ambient light effect decoupling unit 233, and is used to recombine the separated attributes into a structured feature representation.

[0061] In practical implementation, the light source attribute decoupling unit 231 extracts light source features through a multilayer perceptron network. The network structure is: input layer → 512-node fully connected layer → ReLU activation → 256-node fully connected layer → ReLU activation → output layer. The output layer is divided into three parts, corresponding to the light source position (3D), intensity (1D), and color (3D RGB values). The surface attribute decoupling unit 232 and the ambient light effect decoupling unit 233 adopt similar network structures, but the output layer dimensions are adjusted according to specific attributes. The feature recombination unit 234 uses an attention mechanism to perform weighted fusion based on the importance of different attributes.

[0062] The linear mapping module 24 is responsible for establishing the mapping relationship between two-dimensional and three-dimensional features, including a dimension transformation unit, a feature transformation unit, a mapping optimization unit, and a weight adjustment unit. The feature fusion layer 25 integrates features from multiple sources to ensure that the final output features retain the style of the original artwork while also containing lighting information suitable for 3D rendering.

[0063] It is worth noting that each layer in the illumination feature decoupling network 2 adopts a parameter-sharing structure, meaning that different layers use the same parameters. This design significantly reduces the number of model parameters (approximately 66%) while improving the consistency of feature representations and the model's generalization ability. In the actual implementation, parameter sharing is achieved through weighttying, where bound layers share the same weight matrix during forward and backward propagation.

[0064] Take an Impressionist painting depicting the effect of morning light and the reflection of light on the water as an example. When this painting is input into the lighting feature decoupling network 2, the image decoupling part 21 first extracts a two-dimensional feature tensor (size 256×256×64) and an initial three-dimensional feature tensor (size 32×32×32×64). Subsequently, the bidirectional feature decoupling part 22, combined with the style encoding vector, further decomposes the lighting features into light source attributes (the position, intensity, and orange-red hues of the morning sun), water surface reflection attributes (highlight spots and blurred reflections), and ambient light attributes (fog and atmospheric scattering effects). These decomposed features are recombined into updated two-dimensional and three-dimensional feature tensors, accurately preserving the unique effect of light shining through the morning mist onto the water in the painting.

[0065] The lighting rendering network 3 receives the 3D feature tensor output by the lighting feature decoupling network 2 and generates a 2D feature tensor suitable for rendering. The feature extraction part further extracts and optimizes the input features, while the rendering layer is responsible for converting the 3D feature tensor into a 2D feature tensor, preparing for the final rendering.

[0066] In one embodiment of the invention, the feature extraction part employs a ResNet structure, containing five residual blocks, each consisting of two 3×3 convolutional layers and a skip connection. This design helps preserve original feature information and prevents the gradient vanishing problem caused by increased network depth. The rendering layer uses convolutional layers with enhanced attention mechanisms, which can adaptively focus on important feature regions, improving rendering quality.

[0067] Taking a classical painting with dramatic contrasts and side lighting as an example, renowned for its striking chiaroscuro effect, the lighting rendering network 3, upon receiving a 3D feature tensor containing these lighting characteristics, first extracts core lighting features using residual blocks, capturing the effect of the unique side lighting on the figures' faces and the deep shadows in the background. The rendering layer then converts these 3D features into 2D feature tensors, preparing them for cross-media rendering. In this process, the system successfully preserves the chiaroscuro effect, enabling its application to 3D virtual scenes.

[0068] The Light Path Generator 4 and the 3D Model Rendering Library 5 enable flexible conversion between various rendering modes, including vector rendering and vector-to-pixel rendering, point cloud rendering and point cloud-to-pixel rendering, and 3D triangle mesh rendering and triangle mesh-to-pixel rendering. The 3D rendering network renders corresponding images based on different material models, including vector rendering, point cloud rendering, and triangle mesh rendering.

[0069] In practical applications, for vector models, the light path generator 4 calculates the propagation path of light in the scene using the RayTracing algorithm; for point cloud models, Splat-based rendering technology is used; and for triangular mesh models, an improved Phong shading model is used for rendering. The 3D model rendering library 5 includes various material models, such as Lambert diffuse model, Blinn-Phong specular reflection model, and physically based rendering (PBR) models, to adapt to different types of rendering needs.

[0070] The cross-media rendering network 6 is a crucial component of this invention. It fuses the rendering 2D feature tensor output by the lighting rendering network 3 with the object features, and then inputs it into the 3D rendering network to obtain the final rendering result. This process achieves a complete transfer of lighting effects from 2D artworks to 3D virtual objects.

[0071] In practical implementation, the cross-media rendering network 6 employs a feature-level fusion strategy, including channel attention and spatial attention mechanisms, to ensure effective fusion of rendered 2D feature tensors and object features. The channel attention mechanism learns the weights of different channels to highlight important feature channels; the spatial attention mechanism focuses on different regions of the feature map to enhance the feature representation of important spatial locations.

[0072] Training module 7 is responsible for training all network components of the entire system, using an adversarial-smoothness dual-constraint loss function. For example... Figure 6 As shown, the loss function includes an adversarial loss term and a smoothing loss term. The adversarial loss term ensures that the generated features are consistent with the distribution of the real features, while the smoothing loss term improves the continuity and natural transition of the generated lighting.

[0073] In adversarial loss, the system minimizes the distribution difference between real and generated features, making the generated lighting effects closer to the style of real artwork. Specifically, adversarial loss can be expressed as:

[0074] ,

[0075] in, This is a discriminator network used to distinguish between real features and generated features; The feature mapping function converts the input image into a feature representation. Authentic features are extracted from real artworks; Features are generated by the system. The expectation operator calculates the average loss. This loss function makes the generated lighting features statistically approximate the lighting features of a real artwork. The smoothing loss, on the other hand, calculates the spatial gradients of the features and minimizes the sum of the squares of these gradients to achieve a smooth transition in the lighting effects.

[0076] ,

[0077] in, For feature map The gradient in the horizontal direction (x-direction); For feature map The gradient in the vertical direction (y-direction); Using the pixel indices in the feature map, a summation operation is performed across all pixel positions. This loss function penalizes drastic changes in the feature map and encourages the generation of smoothly transitioning lighting effects. The complete loss function combination is:

[0078] ,

[0079] in, The weighting coefficients for smoothing loss control the strength of the smoothing constraint, and their values ​​are typically between 0.1 and 0.5. The weight coefficients of the weight constraint term control the regularization strength, and their values ​​are usually between 0.01 and 0.05. The constraint term for network weights is defined as follows: ,in These are the main weight parameters of the network. For additional weighting parameters, The L2 norm squared is used to calculate the sum of squares of the weight parameters. This weight constraint prevents overfitting and enhances the model's generalization ability.

[0080] Training module 7 employs a phased training strategy, such as... Figure 8 As shown, training is performed sequentially in the order of decoupler, rendering network, and bidirectional feature decoupling part. This progressive training method effectively avoids the training instability problem caused by learning all tasks at once, and improves the overall performance of the system.

[0081] Specifically, in the first stage, the system prioritizes training the decoupler module 23, enabling it to accurately decompose lighting features into different components. When training on an expressionist work with distorted lighting effects, the system first learns to decompose the lighting features in the painting into distorted sunlight attributes, orange-red sky hue attributes, and shadow attributes of bridges and figures. This stage uses a combination of adversarial loss and smoothing loss to ensure that the decomposed features are both realistic and smooth. In the second stage, the system trains the rendering network to generate high-quality rendered 2D features based on the decoupled 3D features. For an expressionist work with distorted lighting effects, the system learns how to apply the distorted sky lighting effect to the 3D scene while maintaining its unique emotional tension. Finally, the system trains the bidirectional feature decoupling part 22 to achieve precise bidirectional mapping between 2D and 3D features, ensuring that the iconic distorted lighting effect in the work can be accurately reproduced in 3D space.

[0082] Based on the above system, the present invention also provides a method for cross-media transfer rendering of artwork lighting effects, comprising the following steps:

[0083] 1. Training the Style Transfer Network Module: In this step, the system trains the style transfer network using an unsupervised generative adversarial learning method. Specifically, the system randomly selects real and generated images from the dataset, calculates the loss function, and automatically updates the network weights based on the loss function value.

[0084] The loss function consists of two parts: adversarial loss and smoothing loss. The adversarial loss is used to train the discriminator network to distinguish between real and generated images, and its expression is:

[0085] ,

[0086] in, For the discriminator network, it evaluates the probability that the input is a real sample; This is a generator network that converts random noise into synthetic samples; These are real samples, derived from a dataset of real artworks; It is random noise, and usually follows a standard normal distribution; This represents the true data distribution, specifically the statistical distribution of artwork images. The noise distribution is typically a standard normal distribution. This represents the expected value of the actual data distribution; This represents the expected value of the noise distribution. The loss function maximizes the difference between the discriminator output and the generated samples.

[0087] Smoothing loss is used to learn the smoothness of different feature channels, and its expression is:

[0088] ,

[0089] in, The synthesized image output by the generator; and These represent the gradients of the synthesized image in the horizontal and vertical directions, respectively. Using image pixel indices, summation is performed across all pixel locations. This loss function encourages the generation of smooth images and reduces unwanted high-frequency details.

[0090] Taking an abstract painting with geometric shapes and striking color contrasts as an example, Style Transfer Network 1 learns to extract these geometric shapes and contrasts. The generator learns how to create features with similar color distributions and shape arrangements, while the discriminator learns to distinguish between the real artwork and the generated imitation. Through repeated training, the system can eventually extract and generate the unique color block distribution and geometric composition of the artwork, providing a foundation for subsequent decoupling of lighting features.

[0091] 2. Training the illumination feature decoupling network module: In this step, the system uses the new training dataset to train the network parameters of the decoupler part, and uses the old training dataset to train the linear mapping module and the rendering network. The training of the decoupler part adopts generative adversarial learning, and its loss function is:

[0092] ,

[0093] in, The decoupler part maps the input image to the feature space; The weight parameter balances the adversarial loss and the smoothing loss, and is typically set to 0.3. This loss function combines adversarial and smoothing properties, enabling the decoupler to generate feature representations that are both realistic and smooth.

[0094] Take, for example, a modern painting depicting the contrast between artificial light and nighttime light and shadow. During training, the decoupler learns to separate the bright interior light of a café from the dark environment of the external street and recognizes the reflection effects of window glass. The system trains the decoupler using a new dataset, enabling it to accurately capture signature light and shadow techniques. Simultaneously, the system trains the linear mapping module and rendering network using an older dataset (containing works of various artistic styles) to ensure the model has good generalization ability.

[0095] 3. Train the bidirectional feature decoupling part of the illumination feature decoupling network, using the new training dataset, and calculate the adversarial loss:

[0096] ,

[0097] in, The structure for calculating the adversarial loss using the new training dataset is the same as in step 2. The same, but using a newer dataset. This loss function ensures that the bidirectional feature decoupling part can effectively handle lighting features across various artistic styles.

[0098] When training to process a landscape painting with structured lighting, the bidirectional feature decoupling part of the network learns how to establish a mapping relationship between 2D and 3D feature representations. The system captures the unique way the light and shadow of the mountains are handled, including planar color blocks and structured lighting representations. These features are converted between 2D and 3D representations to ensure that the 3D rendering preserves the sense of space and lighting hierarchy in the artwork.

[0099] 4. Train the lighting rendering network module. The training objective of lighting rendering network 3 is:

[0100] ,

[0101] in, This is a regularization parameter that controls the strength of the weight constraints; it is usually set to 0.03. The function that applies weight constraints to the linear mapping module is defined as follows: ,in and These are the network's primary and secondary weight parameters, respectively. This loss function combines constraints on feature quality and parameter size, enabling the rendering network to generate high-quality features without becoming overly complex. Take a still life painting with a strong directional light source as an example. The lighting rendering network learns how to apply the directional light source effect to a 3D scene. The system can accurately reproduce the gloss and texture of the fruit's surface in the painting, as well as the deep shadows in the background—effects crucial for realistic lighting.

[0102] 5. The joint loss function for training the lighting feature decoupling network and the lighting rendering network is:

[0103] ,

[0104] in, The loss is for the bidirectional feature decoupling portion; This refers to the losses in the decoupler section; This is a weighting parameter that balances the contributions of the two losses; it is usually set to 0.5. and The definition is the same as before. This joint optimization ensures that the components of the system can work together to produce consistent and high-quality results.

[0105] Take a surrealist artwork with symbolic lighting as an example. The system learns how to handle different lighting effects on two figures in the painting, as well as dramatic sky lighting in the background, by jointly training a lighting feature decoupling network and a lighting rendering network. This joint training enables the system to accurately capture the symbolic and psychological implications of lighting in the artwork and apply them to a 3D scene.

[0106] 6. The outputs of the generator and decoupler are concatenated and used as input to the bidirectional feature decoupling network to train the generation of illumination and texture features. In this step, the system concatenates the outputs of the generator and decoupler and uses them as input to the bidirectional feature decoupling network 22 to train the system to generate illumination and texture features. The loss function and training dataset are the same as in step 3.

[0107] Taking a Neo-Impressionist work employing pointillism as an example, this painting is renowned for its pointillist technique and unique depiction of light. The system learns to separate the texture features of the pointillist brushstrokes from the lighting characteristics of a park scene under sunlight. By connecting the outputs of a generator (providing pointillist style features) and a decoupler (providing lighting decomposition features), the bidirectional feature decoupling network can generate an accurate representation of the dappled light and shadow effects created by light passing through the leaves in the artwork, while preserving the texture features of the pointillist technique.

[0108] 7. The ray path is rendered using a cross-media rendering network. The cross-media rendering network 6 renders the ray path, and its loss function is:

[0109] ,

[0110] in, It is a two-dimensional feature tensor of a real image, containing visual feature information of the image; To and Two-dimensional feature tensors with the same stylistic characteristics may have different contents; A 3D feature tensor for vector-rendered images, representing the features of a 3D scene; To and Three-dimensional feature tensors with the same stylistic characteristics; Represents the square of the Euclidean distance, measuring the difference in features; The weighting parameter, which balances the loss of two-dimensional and three-dimensional features, is usually set to 0.7; and The definition is the same as before. This loss function ensures that the rendered result is consistent with the target style in both the two-dimensional and three-dimensional feature spaces.

[0111] Taking an expressionist work with distorted lighting as an example, the cross-media rendering network learns how to apply the distorted lighting effects of the sky and bridge in the work to a 3D scene. By minimizing 2D and 3D feature losses, the system ensures that the final rendering result retains both emotional tension and visual distortion while maintaining geometric consistency in 3D space, allowing viewers to experience this unique lighting effect from different angles.

[0112] 8. Train the light path generator and 3D model rendering library. Use the 3D rendering network to convert the rendering results of vector graphics and triangle meshes into pixel-rendered images. In this step, the system trains light path generator 4 and 3D model rendering library 5, and uses the 3D rendering network to convert the rendering results of vector graphics and triangle meshes into pixel-rendered images. The loss function is:

[0113] ,

[0114] in, This is the loss function after optical path rendering, which measures the difference between the rendered result and the target image; This function evaluates the quality differences between different rendering modes to account for the rendering loss of the light path and the triangle mesh. It ensures consistent result quality across different rendering modes.

[0115] Using a landscape painting depicting fog and light scattering as an example, the system trains a light path generator and a 3D model rendering library to learn how to simulate the fog, rain, and light scattering effects in Turner's works. For 3D scenes containing trains and tracks, the system needs to handle the different responses of various surface materials (metal, water, fog) to light. Through this training, the system can render vector models and triangular mesh models into pixel images with Turner-style lighting effects, accurately reproducing the hazy atmosphere and dynamic effects in the paintings.

[0116] To better illustrate the working process of the present invention, a complete embodiment is provided below for detailed description:

[0117] Suppose we need to transfer the lighting effects of a night scene painting to a 3D virtual mountain village scene. First, the painting is fed into a style transfer network 1 as input image. After processing through convolutional neural network layers and multilayer perceptrons, a 128-dimensional style encoding vector is extracted. This vector accurately captures the features of the spiral nebula, the bright moon, and the village lights in the painting.

[0118] Next, the image of the painting enters the image decoupling part 21 of the lighting feature decoupling network 2, generating a two-dimensional feature tensor (256×256×64) and a three-dimensional feature tensor (32×32×32×64). These feature tensors are then fed into the bidirectional feature decoupling part 22 for further processing in conjunction with the style encoding vector. The decoupling module 23 decomposes the lighting features into light source attributes (position, intensity, and color of the moon and stars), surface attributes (reflective properties of the sky and hills), and environmental attributes (ambient light of the night sky). The linear mapping module 24 and the feature fusion layer 25 reassemble these decoupled features into updated two-dimensional and three-dimensional feature tensors.

[0119] The lighting rendering network 3 receives the updated 3D feature tensor and, through feature extraction and rendering layer processing, generates a rendered 2D feature tensor. This tensor contains lighting features characteristic of the painting, such as the iconic spiral light patterns, the strong blue and yellow contrast, and the dotted representation of starlight.

[0120] Meanwhile, the geometric and material information of the 3D mountain village scene is processed by the light path generator 4 and the 3D model rendering library 5 to generate object features suitable for rendering. The cross-media rendering network 6 fuses the rendering 2D feature tensor with these object features and inputs it into the 3D rendering network.

[0121] The final rendered result is a 3D mountain village scene: the night sky displays distinctive spiral nebulae and bright stars; the hills and building surfaces exhibit undulating blue and yellow light and shadow; and trees and other objects are given a signature brushstroke-like dynamism. The entire scene retains the accuracy of 3D geometry while presenting a dreamlike and emotionally resonant lighting atmosphere.

[0122] Users can freely move their viewpoint within this 3D scene, appreciating the painting's lighting effects from different angles. Regardless of the viewing angle, the scene retains the painting's lighting characteristics while showcasing a logical 3D spatial relationship. This experience far surpasses traditional 2D image style transfer, offering entirely new possibilities for art appreciation and virtual reality content creation.

[0123] This invention provides a cross-media transfer rendering system and method for lighting effects in artworks. Through innovative techniques such as lighting feature decoupling network design, bidirectional feature decoupling mechanism, and parameter sharing structure, it achieves high-quality transfer of lighting effects from two-dimensional artworks to three-dimensional virtual objects. This system can not only accurately extract and transfer lighting features from artworks but also simultaneously maintain the accuracy of physical lighting and the expressiveness of artistic style, providing important technical support for fields such as virtual reality, game development, and digital art creation.

[0124] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made based on the content of the present invention specification and drawings under the concept of the present invention, or direct / indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A cross-media transfer rendering system for artwork lighting effects, characterized in that, include: A style transfer network is used to acquire style features of two-dimensional paintings. The style transfer network includes convolutional neural network layers and multilayer perceptrons. An illumination feature decoupling network is provided, comprising an image decoupling part and a bidirectional feature decoupling part. The image decoupling part receives a two-dimensional painting image as input and outputs a two-dimensional feature tensor and a three-dimensional feature tensor. The bidirectional feature decoupling part receives the three-dimensional feature tensor and the numerical vector output by the multilayer perceptron, and outputs updated two-dimensional feature tensors and updated three-dimensional feature tensors. The illumination feature decoupling network includes a decoupler module, a linear mapping module, and a feature fusion layer. A lighting rendering network, connected to the lighting feature decoupling network, is used to receive the updated three-dimensional feature tensor and output a rendering two-dimensional feature tensor. The lighting rendering network includes a feature extraction part and a rendering layer. The rendering layer is used to receive the three-dimensional feature tensor output by the decoupler module and output the rendering two-dimensional feature tensor. A light path generator and a 3D model rendering library for converting vector models and triangular mesh models and performing lighting calculations; A cross-media rendering network, connected to the lighting rendering network, is used to receive the rendering 2D feature tensor and object features and fuse them. The fused feature tensor is then input into the light path generator and the 3D rendering network in the 3D model rendering library to obtain the final rendering result.

2. The artwork lighting effect cross-media transfer rendering system according to claim 1, characterized in that, It also includes a training module connected to the style transfer network, the lighting feature decoupling network, and the lighting rendering network. This module is used to train the style transfer network, the lighting feature decoupling network, and the lighting rendering network using training data, thereby achieving multi-element transfer of lighting, style, 3D model, and media. The style transfer network can train a style generator and a discriminator network. The feature tensors extracted by the style generator constitute a new style dataset, ultimately resulting in a new training dataset. The lighting feature decoupling network uses both the old and new datasets during training.

3. The artwork lighting effect cross-media transfer rendering system according to claim 1, characterized in that, The illumination feature decoupling network is a Transformer network.

4. The artwork lighting effect cross-media transfer rendering system according to claim 2, characterized in that, The style generator and discriminator networks can be trained unsupervised using generative adversarial learning.

5. The artwork lighting effect cross-media transfer rendering system according to claim 1, characterized in that, The light path generator and 3D model rendering library can realize vector rendering and the conversion from vector rendering to pixel rendering, point cloud rendering and the conversion from point cloud rendering to pixel rendering, and 3D triangle mesh rendering and the conversion from triangle mesh rendering to pixel rendering. The light path generator and the 3D rendering network in the 3D model rendering library render corresponding images based on different material models, including vector rendering, point cloud rendering, and triangular mesh rendering.

6. The artwork lighting effect cross-media transfer rendering system according to claim 1, characterized in that, Each layer in the illumination feature decoupling network is a parameter-sharing structure, meaning that different layers use the same parameters.

7. The artwork lighting effect cross-media transfer rendering system according to claim 1, characterized in that, The decoupler module includes: Light source attribute decoupling unit, used to separate the position, intensity, and color attributes of the light source; Surface property decoupling unit, used to separate material reflectivity, roughness, and metallicity properties; Ambient light effect decoupling unit, used to separate ambient light shading and indirect lighting attributes; The feature recombination unit, connected to the light source attribute decoupling unit, the surface attribute decoupling unit, and the ambient light effect decoupling unit, is used to recombine the separated attributes into a structured feature representation.

8. The artwork lighting effect cross-media transfer rendering system according to claim 2, characterized in that, The training module trains the style transfer network, the lighting feature decoupling network, and the lighting rendering network using an adversarial-smoothing dual-constraint loss function. The adversarial-smoothing dual-constraint loss function includes an adversarial loss term and a smoothing loss term. The adversarial loss term is used to ensure that the generated features are consistent with the distribution of real features, and the smoothing loss term is used to improve the continuity and natural transition of the generated lighting.

9. The artwork lighting effect cross-media transfer rendering system according to claim 2, characterized in that, The training module adopts a phased training strategy, and is trained sequentially in the order of decoupler, rendering network, and bidirectional feature decoupling part.

10. A method for cross-media transfer rendering of artwork lighting effects based on the artwork lighting effect cross-media transfer rendering system according to any one of claims 1-9, characterized in that, Includes the following steps: The style transfer network module is trained by randomly selecting real and generated images from the dataset using unsupervised generative adversarial learning, calculating the loss function, and automatically updating the network weights based on the loss function value. Training the illumination feature decoupling network module includes training the network parameters of the decoupler module in the illumination feature decoupling network using a new training dataset, and training the linear mapping module and the rendering network using an old training dataset. Train the bidirectional feature decoupling part of the illumination feature decoupling network using a new training dataset and calculate the adversarial loss. Training the lighting and rendering network module; The joint loss of the training lighting feature decoupling network and the lighting rendering network; The outputs of the generator and the decoupler are concatenated and used as input to a bidirectional feature decoupling network to train and generate illumination features and texture features. Rendering of light paths is achieved through a cross-media rendering network; Train a light path generator and a 3D model rendering library, and use a 3D rendering network to convert the rendering results of vector graphics and triangular meshes into pixel-rendered images.

Images