Essential image decomposition network and method based on unsupervised domain adaptive diffusion and application

Abstract

The invention belongs to the technical field of image processing. The invention provides an essential image decomposition network and method based on unsupervised domain adaptive diffusion and application. The embodiment of the invention provides a gradient flipping domain migration method, which is used for reducing the distribution difference between tagged synthetic data and an untagged natural image. According to the method, the diffusion network pre-trained based on the labeled data can be utilized to realize the reconstruction of the essential characteristics of the unlabeled image. A gradient inversion layer is adopted to carry out domain migration adversarial training, and maximum mean value difference loss is introduced to align feature distribution. A content transmission decomposition network is adopted to preliminarily decompose a natural scene image, a reflectivity graph is extracted, and a shadow-free and mirror-free loss function is introduced to improve the decomposition precision. And taking the preliminarily extracted reflectance image as input, combining domain invariant features as guide conditions, and finely adjusting a diffusion model pre-trained based on labeled synthetic data, so as to realize unsupervised diffusion reconstruction of the natural scene image reflectance image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an essential image decomposition network, method, and application based on unsupervised domain adaptive diffusion. Background Technology

[0002] Image recognition applications are ubiquitous in daily life, such as facial recognition, object tracking, and autonomous driving. However, during the imaging process, environmental factors such as light intensity, angle of incidence, and shadow occlusion can lead to poor image quality, making image recognition difficult and reducing accuracy. One solution is to extract features from multimodal images that are unaffected by environmental factors—the intrinsic image. The intrinsic image refers to the inherent features of an object, independent of its environment, including color, texture, and material. These inherent features do not change with environmental factors. If we can separate the intrinsic information of an object—color, texture, and material—from the environment and filter out the image parts affected by the environment, we can obtain a more accurate feature description of the object. Intrinsic image decomposition aims to extract these intrinsic features. It decomposes the image into two parts: a reflection map containing texture, color, and material, and an illumination map containing shape and lighting information. The reflection map is unchanging with the environment, and using the decomposed reflection map as input for other image understanding tasks will greatly reduce the difficulty of image analysis and give image understanding illumination-invariant robustness.

[0003] Intrinsic image decomposition is mainly divided into two categories according to algorithm type: the first is intrinsic image decomposition based on Retinex theory, and the second is intrinsic image decomposition problem based on deep learning.

[0004] Intrinsic image decomposition based on Retinex theory decomposes an image based on local gradient changes. It posits that large gradient changes are typically caused by differences in surface material, specifically color variations resulting from material reflection, leading to large gradient shifts. Small gradient changes are generally attributed to illumination. This theory assumes that illumination changes slowly and uniformly, without abrupt changes. However, in reality, illumination can change abruptly due to occlusion and other issues, resulting in large gradient changes, rendering Retinex theory inapplicable.

[0005] Intrinsic image decomposition methods based on deep learning have largely solved the aforementioned problems. However, the performance of current deep learning techniques is significantly constrained by data scale and annotation quality, especially in supervised training frameworks, where model performance is strongly correlated with the completeness of labeled data. Intrinsic image decomposition tasks face annotation bottlenecks: obtaining pixel-level physical decomposition annotations in real-world scenes requires sophisticated optical equipment and complex laboratory environments. This high-cost, resource-intensive annotation process severely limits the practical application of supervised methods. Therefore, traditional supervised learning methods have significant limitations in handling such problems, which has spurred a widespread demand for unsupervised learning methods.

[0006] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0008] The purpose of this disclosure is to provide an essential image decomposition network, method, and application based on unsupervised domain adaptive diffusion, thereby overcoming, to at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0009] According to a first aspect of the present disclosure, an essential image decomposition network based on unsupervised domain adaptive diffusion is provided, comprising: Domain adaptation network, content delivery decomposition network, diffusion model, encoder, and decoder; among them, Domain adaptation networks include feature extractors and neighborhood discriminators, which are used to extract features from the original image to output domain-invariant features; The encoder consists of k cascaded residual blocks, each of which downsamples the input by a factor of 2 through a max pooling layer to convert the original image into latent spatial features; Content transport decomposition network is used to decompose latent spatial features into preliminary reflectance maps; A diffusion model is used to guide the transformation of randomly sampled Gaussian noise into recovered features based on a preliminary reflectance map using domain-invariant features. The decoder is used to decode the recovered features to generate the final reflectance map.

[0010] Furthermore, the training process for the domain adaptation network is as follows: Use the source domain dataset and the target domain dataset as input; Randomly initialize the feature extractor and Domain Discriminator ; Source domain images were obtained by sampling from the source domain dataset: The target domain image is obtained by sampling from the target domain dataset: The source domain image is obtained through a feature extractor. and target domain image The Feature representation of each sample:

[0011] Using Gaussian kernel function to transform the source domain image and target domain image The feature representation is mapped to a high-dimensional feature space to obtain the source domain image. Distribution and target domain image Distribution:

[0012] In the formula, For kernel function mapping; Computing source domain image and target domain image Maximum mean difference:

[0013] In the formula, For source domain image The number of samples, For target domain image The number of samples, It is the norm in the characteristic space; Perform gradient descent until convergence to obtain a trained domain-invariant feature extractor. :

[0014] In the formula, For gradient descent operation, This is a hyperparameter.

[0015] 3. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 2, characterized in that the training process of the content transport decomposition network is as follows: Randomly initialize the parameters of the content transmission decomposition network; According to the labeled image pixel values For the initial lighting diagram and the initial reflectance map Make an estimate:

[0016] In the formula, It is a tiny constant; The initial illumination map is extracted using Convs with multiple convolutional blocks. and the initial reflectance map Based on the features, we obtain the embedding representations of the initial illumination map and the initial reflectance map:

[0017] A cross-self-attention module (CA) is introduced to compute the embedding representation of the initial illumination map. Embedded representation of the initial reflectance map The similarity between them will be used to embed the initial lighting map. The global information is aligned with the local features of the initial reflectance map's embedding representation, and then weighted and added to the initial reflectance map's embedding representation to obtain the enhanced reflectance map:

[0018] Introducing a self-attention module (SA) from the embedded representation of the initial illumination map. Extract the global dependencies to obtain the enhanced lighting map:

[0019] A convolution operation is performed on the enhanced illumination map and the enhanced reflectance map to obtain a preliminary reflectance map. and preliminary lighting diagram :

[0020] Perform gradient descent operation:

[0021] In the formula, To avoid shadow loss, No mirror loss; The process continues until convergence, resulting in a well-trained content transport decomposition network.

[0022] 4. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 3, characterized in that the training process of the diffusion model is as follows: Preliminary reflectance map Domain-invariant characteristics As input; Define the initial reflectivity map Initial state Construct a closed-type representation of the sample based on the noise level:

[0023] In the formula, The attenuation coefficient is... The image after adding noise at step t. This is the image after adding noise at step t-1. It follows a standard normal distribution; With field invariant features For guiding conditions Through diffusion model Estimated Image noise Using estimated noise Construct learnable closed-form expressions; where, for The variance; Hierarchical clustering was used to analyze... and Clustering is performed to obtain the respective cluster matrices. and ; right and Regularize the rank using the first norm and fit the result. and Deviation between:

[0024] In the formula, Represents the regularization loss term. For matrix The output of the singular value decomposition, This returns the vector along the main diagonal of the matrix. Gradually inject structure-aware regularization to construct the regularization function:

[0025] In the formula, For regularization loss term The adaptive factor, where T is the maximum time step; Introducing Uncertainty-Adaptive Attention Graphs Perform gradient descent operation:

[0026] In the formula, This is real noise gradually added during the forward process; Continue until convergence, and you will obtain a trained diffusion model.

[0027] 5. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 4, characterized in that the loss function of the diffusion model is:

[0028] In the formula, For the diffusion model loss, For global structural consistency loss, An adaptive attention map; The loss function of the domain adaptation network is:

[0029] In the formula, Domain discriminator classification loss, For the maximum mean difference loss, The distribution of the source domain image, The distribution of the target domain image; The loss function of the content delivery decomposition network includes shadowless loss and mirrorless loss; where the shadowless loss is:

[0030] In the formula, This represents the image reflection layer in the color space. A color image without shadows; No mirror loss is:

[0031] In the formula, For mapping from RGB space to a mirrorless image, This represents the value of the corresponding color channel in the non-mirror image.

[0032] 6. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 5, characterized in that the method further includes: Jointly train the content transport decomposition network and the diffusion model: The goal of the first phase is to optimize the encoder. Content delivery decomposition network and decoder At the same time, the parameters of the diffusion model are frozen; In the first stage, the encoder and decoder Optimization using content-aware loss:

[0033] In the formula, For content-aware loss function, The original input image; Content delivery decomposition networks are optimized using shadowless loss and mirrorless loss; In the second stage, freeze the encoder. Content delivery decomposition network and decoder The parameters are then used to fine-tune the diffusion model using noise prediction loss:

[0034] In the formula, It is the network's noise prediction at the current time step.

[0035] According to a second aspect of the present disclosure, an essential image decomposition method based on unsupervised domain adaptive diffusion is provided, comprising: The trained domain adaptation network extracts features from the original image to output domain-invariant features; The trained encoder converts the original image into latent spatial features; The trained content transport decomposition network decomposes latent spatial features into preliminary reflectance maps; The trained diffusion model is based on the initial reflectance map and guided by domain-invariant features to gradually transform randomly sampled Gaussian noise into recovered features. The trained decoder decodes the recovered features to generate the final reflectance map.

[0036] Furthermore, the aforementioned essential image decomposition network based on unsupervised domain adaptive diffusion can be applied in the field of image processing.

[0037] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, through the aforementioned network and method, firstly, a gradient-flipped domain transfer method is proposed to reduce the distribution difference between labeled synthetic data and unlabeled natural images. This method enables the reconstruction of essential features of unlabeled images using a diffusion network pre-trained on labeled data. This module employs a gradient-flipped layer for domain transfer adversarial training and introduces maximum mean difference loss to align feature distributions. Secondly, a content transfer decomposition network is used to initially decompose the natural scene image, extracting a reflectance map, and introducing shadow-free and mirror-free loss functions to further improve decomposition accuracy. Finally, the initially extracted reflectance map is used as input, combined with domain-invariant features as guiding conditions, to fine-tune the diffusion model pre-trained on labeled synthetic data, thereby achieving unsupervised diffusion reconstruction of the reflectance map of the natural scene image. Furthermore, this approach allows for precise control of the image generation process, effectively restoring details while overcoming the shortcomings of existing technologies that heavily rely on labeled data, and effectively performing essential decomposition of unlabeled natural scene images. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0039] Figure 1 This diagram illustrates the structure of an essential image decomposition network based on unsupervised domain adaptive diffusion in an exemplary embodiment of this disclosure. Figure 2 This diagram illustrates the structure of a domain adaptation network in an exemplary embodiment of this disclosure. Figure 3 This diagram illustrates the structure of a content delivery decomposition network in an exemplary embodiment of this disclosure. Figure 4 This diagram illustrates the structure of the diffusion model in an exemplary embodiment of this disclosure. Figure 5 Illustrations showing the results of exemplary embodiments of this disclosure; Figure 6 The diagram illustrates the steps of an essential image decomposition method based on unsupervised domain adaptive diffusion, an exemplary embodiment of this disclosure. Detailed Implementation

[0040] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0041] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0042] This example implementation provides an essential image decomposition network based on unsupervised domain adaptive diffusion. (Reference) Figure 1 As shown, the essential image decomposition network based on unsupervised domain adaptive diffusion can include: Domain adaptation network, content delivery decomposition network, diffusion model, encoder, and decoder; among them, Domain adaptation networks include feature extractors and neighborhood discriminators, which are used to extract features from the original image to output domain-invariant features; The encoder consists of k cascaded residual blocks, each of which downsamples the input by a factor of 2 through a max pooling layer to convert the original image into latent spatial features; Content transport decomposition network is used to decompose latent spatial features into preliminary reflectance maps; A diffusion model is used to guide the transformation of randomly sampled Gaussian noise into recovered features based on a preliminary reflectance map using domain-invariant features. The decoder is used to decode the recovered features to generate the final reflectance map.

[0043] The aforementioned unsupervised domain-adaptive diffusion-based essential image decomposition network addresses two main issues. First, it proposes a gradient-flipped domain transfer method to reduce the distributional differences between labeled synthetic data and unlabeled natural images. This method enables the reconstruction of essential features of unlabeled images using a diffusion network pre-trained on labeled data. This module employs a gradient-flipped layer for domain transfer adversarial training and introduces maximum mean difference loss to align feature distributions. Second, a content transfer decomposition network is used to initially decompose the natural scene image, extracting reflectance maps and introducing shadow-free and specular-free loss functions to further improve decomposition accuracy. Finally, the initially extracted reflectance map is used as input, combined with domain-invariant features as guiding conditions, to fine-tune the diffusion model pre-trained on labeled synthetic data, thereby achieving unsupervised diffusion reconstruction of the reflectance map of natural scene images. Furthermore, this approach allows for precise control of the image generation process, effectively restoring details while overcoming the shortcomings of existing technologies that heavily rely on labeled data, and effectively decomposes unlabeled natural scene images.

[0044] Below, we will refer to Figures 1 to 5 The steps of the essential image decomposition network based on unsupervised domain adaptive diffusion in this example implementation will be described in more detail.

[0045] In one embodiment, 1. Construct a training image sample library: A certain number of images are randomly selected from the test image dataset. Then, M small images of size N*N are randomly sampled from each image. These M small images are horizontally flipped to obtain M new small images. Thus, each image has 2*M small images. The small images obtained after performing the above operations on all the selected images constitute the training image sample library. 2. Constructing the domain adaptation module (i.e., the domain adaptation network) like Figure 2 The diagram shown is a schematic of the domain adaptation network.

[0046] The domain adaptation module consists of a feature extractor and a domain discriminator. It adopts a domain adversarial training strategy, introduces a gradient reversal layer and maximum mean difference loss. By introducing adversarial training into the network, the network learns shared features that do not depend on a specific domain. To enable the feature extractor to learn domain-invariant features, a gradient reversal layer (GRL) is introduced. By aligning features between the source and target domains, GRL can effectively facilitate the transfer of knowledge learned in the source domain to the target domain, thereby achieving cross-domain feature sharing.

[0047] The domain-adaptive training framework employs an adversarial collaborative optimization mechanism, the core of which lies in the dynamic game between the feature encoder and the domain discriminator. Technically, a gradient inversion layer (GRL) is introduced to construct a backpropagation channel: during the feature encoding stage, the gradient signal is inverted and transmitted to the front-end network, thus preventing the feature extraction network from directly adapting to the domain discrimination task. This backpropagation strategy forces the feature encoding network to generate deep feature representations with domain invariance, making it difficult for the domain discriminator to effectively distinguish the feature source. Under this strategy, the difference in feature distribution between the source and target domains is progressively resolved through adversarial game, ultimately achieving effective cross-domain knowledge transfer.

[0048] Maximum Mean Discrepancy (MMD) loss, a kernel-based statistical metric, aims to quantify inter-domain discrepancies through latent space distribution alignment strategies. It has been widely applied in domain adaptation tasks. The core objective of MMD loss is to minimize the difference between the feature distributions of the source and target domains, thereby enabling the model to learn feature representations applicable to both domains and improving performance on cross-domain tasks. MMD loss does not rely on data labeling information; it optimizes solely based on the statistical properties of the feature distributions of the source and target domains, making it particularly suitable for domain adaptation tasks. In these tasks, the source and target domains may contain samples of the same or different classes, but their distributions differ, and MMD aims to minimize this difference in distribution.

[0049] The specific training process of the domain adaptation network is as follows: Randomly initialize the feature extractor Domain Discriminator Synthetic dataset (source domain) images Images from natural scene datasets (target domain) As input, sampling The source and target domains are obtained through a feature extractor. Feature representation of each sample:

[0050] Using Gaussian kernel function Map the feature representations of the source and target domains to a high-dimensional feature space:

[0051] Calculate the maximum mean difference between the source and target domains:

[0052] In the formula and These represent the distributions in the source and target domains, respectively. and These are the sample sizes in the source domain and the target domain, respectively. and These are the source domain and the target domain, respectively. Feature representation of each sample; It is the norm in the characteristic space; The domain adaptation module converges continuously through gradient descent, eventually yielding a domain-invariant feature extractor. :

[0053] In the formula These are the true domain labels of the samples: the source domain label is 0, and the target domain label is 1. It is the output of the domain discriminator, representing the model's predicted probability that a sample belongs to the source domain or the target domain.

[0054] 3. Construct a content delivery decomposition network like Figure 3 The diagram shown is a structural diagram of the content delivery decomposition network.

[0055] Represents the pixel values ​​of the input image. and Given the initial illumination map and the initial reflectance map, respectively, the initial reflectance map and the initial illumination map are estimated using the following formula:

[0056]

[0057] In the formula It is a tiny constant that prevents division by zero and ensures the stability of the calculation; Multiple convolutional blocks (Convs) are used to extract features from the initial reflectance map and initial illumination map, resulting in richer embedding representations:

[0058]

[0059] A cross-attention (CA) module is introduced to process the illumination map. Feature information is incorporated into the reflectance map This enhances the content information of the reflectance map.

[0060] To further enhance the lighting diagram To extract global information from the illumination map, a self-attention (SA) module is used to extract global dependencies:

[0061] In the reflectivity diagram and lighting diagram After performing the above refinement operations, the results are further processed through convolution operations to obtain the final preliminary reflectance map. and preliminary lighting diagram :

[0062]

[0063] Through these steps, a detailed preliminary reflectance map can be obtained. It can represent some of the intrinsic content information of an image, while simultaneously obtaining a preliminary illumination map without content. This lighting map only includes the lighting conditions of the image, excluding specific object details.

[0064] Perform gradient descent operation:

[0065] In the formula, To avoid shadow loss, No mirror loss; The process continues until convergence, resulting in a well-trained content transport decomposition network.

[0066] 4. Construct a supervised essential image decomposition diffusion network (i.e., a diffusion model). like Figure 4 The diagram shown is a schematic representation of the diffusion model.

[0067] Supervised training aims to improve the diffusion model obtained through training. Given an original image I, generate a reflectance map R corresponding to the original image. Specifically, the supervised diffusion model performs the essential image decomposition process as follows: During the training phase, the network's input includes a preliminary reflectivity map. Domain-invariant characteristics Define the initial reflectivity map. Initial state According to noise level Constructing closed-type representations of samples:

[0068] Original image I as guiding condition Through diffusion model Estimated Image noise status Using estimated noise Constructing learnable closed expressions :

[0069] Learnable samples are processed using hierarchical clustering. truth value Clustering is performed to obtain the respective cluster matrices. , ; The rank of the cluster matrix is ​​regularized using a first norm to fit learnable samples. truth value Deviation between:

[0070] In the formula Represents the regularization function. For matrix The output of the singular value decomposition, Returns the vector along the main diagonal of the matrix; Simultaneously, an uncertainty-adaptive attention graph is introduced. This attention map dynamically adjusts attention weights based on noise intensity and edge details, optimizing the model's ability to recover details and thus improving accuracy and refinement in high-frequency regions. At each step, the diffusion model continuously fits the distance between the true value and the generated reflectance map through gradient descent. Through iterative steps, the reflectance map is finally obtained. :

[0071] In the formula This represents a pixel-level uncertainty mapping, i.e. , This represents the weight correspondence function of the denoising diffusion model. This is a hyperparameter, which is set to 10 based on experience.

[0072] Continue until convergence, and you will obtain a trained diffusion model.

[0073] 5. Define the loss function The loss function of the supervised essential image decomposition diffusion network is defined as follows:

[0074] In the formula It is the loss of the diffusion model. It is a loss of global structural consistency. It is an adaptive attention map. This is a hyperparameter, which is set to 10 based on experience;

[0075] In the formula It is the network's noise prediction at the current time step. It is the real noise added to the image at the same time step during the forward process. It represents the square of the L2 norm, which measures the Euclidean distance between the predicted and actual values.

[0076] In the formula Represents the regularization loss term The adaptive factor;

[0077] In the formula Represents the regularization function. For matrix The output of the singular value decomposition, Returns the vector along the main diagonal of the matrix; The domain-transition network loss function is as follows:

[0078] In the formula It is the overall loss function of the domain migration network. It is the classification loss of the domain discriminator. It is the loss due to the maximum mean difference; The joint training loss function for the content transport decomposition network and the diffusion model is defined as follows: The joint training of the content transport decomposition network and the diffusion model employs a two-stage network training strategy. The first stage aims to optimize the encoder. CTDN network (Content Delivery Decomposition Network) and decoder Simultaneously, the parameters of the diffusion model are frozen. In this stage, the encoder and decoder are optimized using content-aware loss:

[0079] The optimization of CTDN networks is achieved through shadowless loss and mirrorless loss:

[0080] In the formula It is a loss without shadows. It has no specular loss. Guided by prior knowledge, a relatively accurate reflectance image is initially obtained; The second stage involves freezing the encoder. CTDN network and decoder The parameters are then used to optimize and fine-tune the diffusion model using noise prediction loss, as follows:

[0081] 6. Using a training image sample library, the supervised reflectance diffusion generator combined with a domain adaptation network and the supervised illumination map generator combined with a domain adaptation network are trained respectively. The Adam optimization method is used to update the network parameters. Training stops when the defined loss function value is minimized, resulting in the final reflectance diffusion generator and illumination map diffusion generator. The supervised reflectance diffusion generator and the supervised illumination map generator have the same structure.

[0082] 7. Input the original image to be processed into the reflection map generator or the illumination map generator respectively, and the output image is the reflection map or illumination map obtained by decomposing the original image.

[0083] In one specific embodiment, a training image sample library is constructed: The MPI-Sintel dataset contains 18 computer-synthesized scene sequences, with 17 scenes containing 50 images each and another scene containing 40 images, for a total of 890 images. Each image has a resolution of 1024×436, but the high resolution makes training the diffusion model more difficult. Therefore, the original images were cropped before training. Specifically, 10 smaller images of 256×256 were randomly cropped from each 1024×436 image to train the supervised diffusion generation network. The IIW dataset is a large-scale image dataset of indoor scenes, containing over 5000 images with diverse scenes, covering various types of furniture, walls, floors, and lighting conditions. The IIW dataset was annotated using a crowdsourced method, primarily through manual sparse annotation on the Amazon Mechanical Turk platform. Due to the high resolution making training the diffusion model challenging, the original images were cropped before training. Specifically, ten 256×256 smaller images were randomly cropped from each image to train the unsupervised domain adaptation network and content transport decomposition network.

[0084] The SAW dataset provides detailed information on illumination variations, including annotations of illumination discontinuities (such as shadow boundaries) and constant illumination regions. This dataset contains over 7000 images and annotates nearly 15000 illumination boundary points and 24000 constant illumination regions. Similar to the IIW dataset, the SAW dataset also uses a crowdsourced annotation approach, but focuses on illumination annotation, especially the distinction between illumination boundaries and constant illumination regions. Because the high resolution makes training the diffusion model challenging, the original images were cropped before training. Specifically, ten 256×256 smaller images were randomly cropped from each image to train the unsupervised domain adaptation network and content transport decomposition network.

[0085] Experimental environment: This research uses an algorithm built on the PyTorch framework. The experimental environment was deployed on a hardware system equipped with a six-core Intel i7-6850K CPU and a dual-GPU heterogeneous computing platform (NVIDIA GTX 1080 Ti 11GB VRAM + TITAN Xp 16GB VRAM). The platform also included 31.9GB of RAM to support large-scale tensor operations. This computing architecture utilizes the CUDA acceleration framework to implement the model training and validation process, fully ensuring the computational performance requirements of the diffusion model in high-resolution image processing.

[0086] For the content delivery network and domain migration network, the Adam optimizer was used during the optimization process, with an initial learning rate set to 1×10⁻⁶. -4 The feature downsampling ratio k is set to 3, and it decays at a rate of 0.8 during training.

[0087] For training the diffusion model, this chapter uses random seeds for fixed parameter initialization. The total training iterations are 2 million. The optimizer used is Adam, with a fixed learning rate of 1×10⁻⁶. -4Furthermore, no weight decay was used. During parameter updates, an exponential moving average with a weight of 0.9999 was applied. During training, a fine-grained diffusion process was used, with T=500 steps, and linear noise scheduling was employed, with an initial noise value of 1×10⁻⁶. -4 The final value is 2×10 -2 Patch size and batch size are set to 96 and 8 respectively. Hyperparameters The value was determined to be 10 through experiments.

[0088] like Figure 5 As shown, the original image to be processed is input into the reflection map generator or the illumination map generator respectively, and the output image is the reflection map or illumination map obtained by decomposing the original image.

[0089] To quantitatively evaluate the performance of the method in this application, it was tested on the IIWSAW intrinsic image dataset and compared with other algorithms. The comparison results are shown in Table 1 (bold indicates the optimal index value).

[0090] Table 1 Performance metrics of several intrinsic image decomposition methods

[0091] As can be seen from Table 1, the method of this application has achieved excellent performance in both WHDR and AP, and has improved significantly in terms of indicators compared with existing methods, which fully demonstrates the effectiveness and practicality of the method of this application.

[0092] Furthermore, this example implementation provides an essential image decomposition method based on unsupervised domain adaptive diffusion. (See reference...) Figure 6 As shown, the essential image decomposition method based on unsupervised domain adaptive diffusion can include: The trained domain adaptation network extracts features from the original image to output domain-invariant features; The trained encoder converts the original image into latent spatial features; The trained content transport decomposition network decomposes latent spatial features into preliminary reflectance maps; The trained diffusion model is based on the initial reflectance map and guided by domain-invariant features to gradually transform randomly sampled Gaussian noise into recovered features. The trained decoder decodes the recovered features to generate the final reflectance map.

[0093] The aforementioned image decomposition network and method based on unsupervised domain-adaptive diffusion achieves the following: First, a gradient-flipped domain transfer method is proposed to reduce the distributional differences between labeled synthetic data and unlabeled natural images. This method enables the reconstruction of essential features of unlabeled images using a diffusion network pre-trained on labeled data. This module employs a gradient-flipped layer for domain transfer adversarial training and introduces maximum mean difference loss to align feature distributions. Second, a content transfer decomposition network is used to initially decompose the natural scene image, extracting reflectance maps and introducing shadow-free and specular-free loss functions to further improve decomposition accuracy. Finally, the initially extracted reflectance map is used as input, combined with domain-invariant features as guiding conditions, to fine-tune the diffusion model pre-trained on labeled synthetic data, thereby achieving unsupervised diffusion reconstruction of the reflectance map of natural scene images. Furthermore, this approach allows for precise control of the image generation process, effectively restoring details while overcoming the shortcomings of existing technologies that heavily rely on labeled data, and effectively decomposes unlabeled natural scene images.

[0094] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. An essential image decomposition network based on unsupervised domain adaptive diffusion, characterized in that, include: Domain adaptation network, content delivery decomposition network, diffusion model, encoder, and decoder; among them, Domain adaptation networks include feature extractors and neighborhood discriminators, which are used to extract features from the original image to output domain-invariant features; The encoder consists of k cascaded residual blocks, each of which downsamples the input by a factor of 2 through a max pooling layer to convert the original image into latent spatial features; Content transport decomposition network is used to decompose latent spatial features into preliminary reflectance maps; A diffusion model is used to guide the transformation of randomly sampled Gaussian noise into recovered features based on a preliminary reflectance map using domain-invariant features. The decoder is used to decode the recovered features to generate the final reflectance map.

2. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 1, characterized in that, The training process for the domain adaptation network is as follows: Use the source domain dataset and the target domain dataset as input; Randomly initialize the feature extractor and Domain Discriminator ; Source domain images were obtained by sampling from the source domain dataset: The target domain image is obtained by sampling from the target domain dataset: The source domain image is obtained through a feature extractor. and target domain image The Feature representation of each sample: Using Gaussian kernel function to transform the source domain image and target domain image The feature representation is mapped to a high-dimensional feature space to obtain the source domain image. Distribution and target domain image Distribution: In the formula, For kernel function mapping; Calculate source domain image and target domain image Maximum mean difference: In the formula, For source domain image The number of samples, For target domain image The number of samples, It is the norm in the characteristic space; Perform gradient descent until convergence to obtain a trained domain-invariant feature extractor. : In the formula, For gradient descent operation, This is a hyperparameter.

3. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 2, characterized in that, The training process of the Content Transport Decomposition Network is as follows: Randomly initialize the parameters of the content transmission decomposition network; According to the labeled image pixel values For the initial lighting diagram and the initial reflectance map Make an estimate: In the formula, It is a tiny constant; The initial illumination map is extracted using Convs with multiple convolutional blocks. and the initial reflectance map Based on the features, we obtain the embedding representations of the initial illumination map and the initial reflectance map: A cross-self-attention module (CA) is introduced to compute the embedding representation of the initial illumination map. Embedded representation of the initial reflectance map The similarity between them will be used to embed the initial lighting map. The global information is aligned with the local features of the initial reflectance map's embedding representation, and then weighted and added to the initial reflectance map's embedding representation to obtain the enhanced reflectance map: Introducing a self-attention module (SA) from the embedded representation of the initial illumination map. Extract the global dependencies to obtain the enhanced lighting map: A convolution operation is performed on the enhanced illumination map and the enhanced reflectance map to obtain a preliminary reflectance map. and preliminary lighting diagram : Perform gradient descent operation: In the formula, To avoid shadow loss, No mirror loss; The process continues until convergence, resulting in a well-trained content transport decomposition network.

4. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 3, characterized in that, The training process for the diffusion model is as follows: Preliminary reflectance map Domain-invariant characteristics As input; Define the initial reflectivity map Initial state Construct a closed-type representation of the sample based on the noise level: In the formula, The attenuation coefficient is... The image after adding noise at step t. This is the image after adding noise at step t-1. It follows a standard normal distribution; With field invariant features For guiding conditions Through diffusion model Estimated Image noise Using estimated noise Construct learnable closed-form expressions; where, for The variance; Hierarchical clustering was used to analyze... and Clustering is performed to obtain the respective cluster matrices. and ; right and Regularize the rank using the first norm and fit the result. and Deviation between: In the formula, Represents the regularization loss term. For matrix The output of the singular value decomposition, This returns the vector along the main diagonal of the matrix. Gradually inject structure-aware regularization to construct the regularization function: In the formula, For regularization loss term The adaptive factor, where T is the maximum time step; Introducing Uncertainty-Adaptive Attention Graphs Perform gradient descent operation: In the formula, This is real noise gradually added during the forward process; Continue until convergence, and you will obtain a trained diffusion model.

5. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 4, characterized in that, The loss function for the diffusion model is: In the formula, For the diffusion model loss, For global structural consistency loss, An adaptive attention map; The loss function of the domain adaptation network is: In the formula, Domain discriminator classification loss, For the maximum mean difference loss, The distribution of the source domain image, The distribution of the target domain image; The loss function of the content delivery decomposition network includes shadowless loss and mirrorless loss; where the shadowless loss is: In the formula, This represents the image reflection layer in the color space. A color image without shadows; No mirror loss is: In the formula, For mapping from RGB space to a mirrorless image, This represents the value of the corresponding color channel in the non-mirror image.

6. The essential image decomposition network based on unsupervised domain adaptive diffusion according to claim 5, characterized in that, The method also includes: Jointly train the content transport decomposition network and the diffusion model: The goal of the first phase is to optimize the encoder. Content delivery decomposition network and decoder At the same time, the parameters of the diffusion model are frozen; In the first stage, the encoder and decoder Optimization using content-aware loss: In the formula, For content-aware loss function, The original input image; Content delivery decomposition networks are optimized using shadowless loss and mirrorless loss; In the second stage, freeze the encoder. Content delivery decomposition network and decoder The parameters are then used to fine-tune the diffusion model using noise prediction loss: In the formula, It is the network's noise prediction at the current time step.

7. An essential image decomposition method based on unsupervised domain adaptive diffusion, characterized in that, include: The trained domain adaptation network extracts features from the original image to output domain-invariant features; The trained encoder converts the original image into latent spatial features; The trained content transport decomposition network decomposes latent spatial features into preliminary reflectance maps; The trained diffusion model is based on the initial reflectance map and guided by domain-invariant features to gradually transform randomly sampled Gaussian noise into recovered features. The trained decoder decodes the recovered features to generate the final reflectance map.

8. The application of the essential image decomposition network based on unsupervised domain adaptive diffusion according to any one of claims 1 to 6 in the field of image processing.

Images