A self-supervised fine-tuning method for re-illumination network, re-illumination method and system

Abstract

The present invention discloses a self-supervised fine-tuning method for a re-illumination network, a re-illumination method and a system, which belong to the field of re-illumination technology. Starting from an image sample N in the real world, its illumination condition E is obtained; N is compared with a random illumination condition E. * Input into the pre-trained re-illumination network and get N in E * The re-illuminated image N under * The re-illuminated image N* is then used as the image input to the re-illuminated network during fine-tuning, E is used as the illumination condition input for fine-tuning, and N is used as the ground-truth training value of the re-illuminated network. An adversarial illumination loss is then constructed to fine-tune the re-illuminated network. By creating paired training data from real-world images, this method ensures that the ground-truth training values ​​of the fine-tuned re-illuminated network remain real-world images, minimizing the discrepancy between the two images. This improves the generalization of the re-illuminated network and enhances the realism of the re-illuminated images output by the network.

Description

Technical Field

[0001] The present invention belongs to the field of re-illumination technology, and more specifically, relates to a self-supervised fine-tuning method for a re-illumination network, a re-illumination method and a system. Background Art

[0002] In the field of computational imaging, relighting is a technique used to adjust and edit the lighting properties of a given object to produce a relighted image that is consistent with specified lighting conditions and close to reality. For example, portrait relighting has attracted considerable attention from both academia and industry in recent years as a crucial component of metaverse and virtual reality applications, with significant application value in fields such as digital photography and face recognition. Current portrait relighting methods fully integrate traditional imaging models, illumination models, 3D facial morphing models, and deep learning technologies, achieving significant improvements in portrait relighting results.

[0003] At present, the basic steps of the mainstream deep learning-based relighting method are: inverse rendering the object to obtain feature information such as albedo and normal, and then re-rendering based on the feature information after inverse rendering to obtain the final relighting image.

[0004] LightStage, as a tool for obtaining datasets in most mainstream methods, provides training pairs of re-illuminated images under physical constraints. However, the datasets obtained by the above methods are synthetic in nature. The background is obtained by projecting from the environment map, and combined with the re-illuminated foreground subject to generate the training truth of the re-illuminated network. However, these synthetic images are somewhat different from images in the real world, resulting in a lack of realism in the output of the model. In addition, the cost of LightStage data acquisition limits the number and diversity of objects in the dataset. The diversity of background images is also limited by the scale of available environment maps (thousands) during rendering, which may affect the generalization ability of the network model to real images, and thus affect the final re-rendering effect, resulting in inconsistency between the target lighting and the re-rendering effect, and a lack of realism. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a self-supervised fine-tuning method, a relighting method and a system for a relighting network, the purpose of which is to improve the realism of the relighting image output by the network.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a self-supervised fine-tuning method for a re-illumination network, comprising:

[0007] For each image sample N in the pre-collected image sample set, obtain the illumination condition E of N; randomly obtain an illumination condition E * , N and E *Input into the pre-trained re-illumination network and get N in E * The re-illuminated image N under * ; Where N is an image sample in the real world;

[0008] Input N* and E into the relighting network to get N ** , calculate N ** The difference loss between N and N is used as the adversarial illumination loss, and the training loss including the adversarial illumination loss is constructed;

[0009] The relighting network is fine-tuned by minimizing the training loss.

[0010] Further preferably, the self-supervised fine-tuning method of the re-illumination network further includes:

[0011] Randomly obtain an illumination condition E', input N and E' into the re-illumination network, and obtain the re-illuminated image N' of N under E';

[0012] Divide E' into illumination conditions E according to the light intensity ratio l ' and E r ';N and E l 'Input the re-illumination network to get N l ', N and E r 'Input the re-illumination network to get N r ', calculate N l '+N r The difference loss between ' and N' is used as the consistent illumination loss;

[0013] The above training loss also includes consistent illumination loss, which is the weighted sum of adversarial illumination loss and consistent illumination loss.

[0014] Further preferably, the re-lighting network comprises: a cascaded albedo network and a lighting rendering network;

[0015] The albedo network is used to perform eigendecomposition on image samples and generate corresponding albedo maps;

[0016] The lighting rendering network is used to re-render the image sample based on the lighting conditions and the albedo map of the image sample to obtain the corresponding re-lit image; or re-render the image sample based on the lighting conditions, the image sample and its albedo map to obtain the corresponding re-lit image;

[0017] Fine-tuning the relighting network by minimizing the training loss involves freezing the parameters in the lighting rendering network and adjusting the parameters in the albedo network with the goal of minimizing the training loss.

[0018] Further preferably, the relighting network includes: a cascaded normal network and a lighting rendering network;

[0019] The normal network is used to perform eigendecomposition on image samples and generate corresponding normal maps;

[0020] The lighting rendering network is used to re-render the image sample based on the lighting condition and the normal map of the image sample to obtain a corresponding re-lit image; or, re-render the image sample based on the lighting condition, the image sample and its normal map and the lighting condition to obtain a corresponding re-lit image;

[0021] The fine-tuning of the relighting network by minimizing the training loss mentioned above includes freezing the parameters in the lighting rendering network and adjusting the parameters in the normal network with the goal of minimizing the training loss.

[0022] Further preferably, the re-illumination network includes: an albedo network, a normal network and a lighting rendering network;

[0023] Fine-tuning the relighting network by minimizing the training loss includes freezing the parameters in the lighting rendering network and adjusting the parameters in the albedo network and / or the normal network with the goal of minimizing the training loss.

[0024] Further preferably, the DECA algorithm is used to obtain the lighting condition E of N.

[0025] In a second aspect, the present invention provides a re-illumination method, comprising:

[0026] Input the image to be re-illuminated and the target illumination conditions into the re-illumination network to obtain the corresponding illumination image;

[0027] The re-illumination network is fine-tuned using the self-supervised fine-tuning method for the re-illumination network provided by the first aspect of the present invention.

[0028] In a third aspect, the present invention provides a re-lighting system, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the re-lighting method provided in the second aspect of the present invention when executing the computer program.

[0029] In a fourth aspect, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, it controls the device where the storage medium is located to execute the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the present invention or the re-illumination method provided in the second aspect of the present invention.

[0030] In a fifth aspect, the invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the invention or the re-illumination method provided in the second aspect of the invention.

[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0032] 1. This paper provides a self-supervised fine-tuning method for a re-illumination network. Starting from a real-world image sample N, its illumination condition E is obtained; N is compared with the random illumination condition E. * Input into the pre-trained re-illumination network and get N in E * The re-illuminated image N under * The re-illuminated image N* is then used as the image input to the re-illuminated network during fine-tuning, E is used as the illumination condition input for the re-illuminated network during fine-tuning, and N is used as the ground-truth training value of the re-illuminated network. An adversarial illumination loss is then constructed to fine-tune the re-illuminated network. By creating paired training data from real-world images, this method ensures that the ground-truth training values ​​of the fine-tuned re-illuminated network remain real-world images, minimizing the discrepancy between them and real-world images. This improves the generalization of the re-illuminated network and significantly reduces the reliance on obtaining difficult, high-quality images, thereby enhancing the realism of the re-illuminated images output by the network.

[0033] 2. Further preferably, the self-supervised fine-tuning method of the re-illumination network provided by the present invention divides one lighting condition into two parts according to the light intensity ratio as the lighting condition input of the re-illumination network, ensuring that the sum of the re-rendering effect values ​​under the two divided lighting conditions is equivalent to the re-rendering effect value under the undivided lighting condition, thereby constructing a consistent lighting loss, thereby ensuring that the output of the re-illumination network is consistent with the physical constraints in the real world, further improving the accuracy and generalization of the expression of the re-illumination network under various lighting conditions, and further improving the realism of the re-illumination image output by the network.

[0034] 3. Further preferably, the self-supervised fine-tuning method of the re-lighting network provided by the present invention takes into account that the estimation accuracy of most albedo and normal will directly affect the final re-rendering effect, resulting in inconsistency between the target lighting and the re-rendering effect, and lack of realism. When selecting the re-lighting network, the present invention preferably selects a re-lighting network with an albedo network and / or a normal network according to the user's requirements for albedo and / or normal accuracy. When fine-tuning, priority is given to parameter adjustment in the albedo network and / or normal network, and the parameters of the rest of the re-lighting network are frozen. With the goal of minimizing training loss, the parameters in the albedo network and / or normal network are adjusted. This can improve the albedo and / or normal accuracy while ensuring the realism of the re-lighting effect, and also improve the fine-tuning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of a self-supervised fine-tuning method for a re-illumination network provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a method for establishing synthetic data pairs to combat illumination loss provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a method for combating light loss provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of a method for establishing consistent illumination loss provided by an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of the self-supervised fine-tuning strategy flow provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0041] In order to achieve the above objectives, firstly, Figure 1 As shown, the present invention provides a self-supervised fine-tuning method for a re-illumination network, comprising:

[0042] For each image sample N in the pre-collected image sample set, obtain the illumination condition E of N; randomly obtain an illumination condition E * , N and E * Input into the pre-trained re-illumination network and get N in E *The re-illuminated image N under * ; Where N is an image sample in the real world;

[0043] Input N* and E into the relighting network to get N ** , calculate N ** The difference loss between N and N is used as the adversarial illumination loss, and the training loss including the adversarial illumination loss is constructed;

[0044] The relighting network is fine-tuned by minimizing the training loss.

[0045] It should be noted that the lighting condition E of N can be obtained by using methods such as DECA algorithm, DPR algorithm, SPLiT algorithm, etc., which are not limited here. Preferably, the lighting condition E of N is obtained by using DECA algorithm.

[0046] It should be noted that when constructing the training sample pairs required for adversarial illumination loss, the same image sample N can also be processed with different random illumination conditions based on the pre-trained re-illumination network to generate N re-illumination images under different random illumination conditions; through the above method, multiple pairs of training sample pairs required for adversarial illumination loss can be constructed through one image sample N, further expanding the training sample set.

[0047] In an optional implementation manner, the self-supervised fine-tuning method for the re-illumination network further includes:

[0048] Randomly obtain an illumination condition E', input N and E' into the re-illumination network, and obtain the re-illuminated image N' of N under E';

[0049] Divide E' into illumination conditions E according to the light intensity ratio l ' and E r ';N and E l 'Input the re-illumination network to get N l ', N and E r 'Input the re-illumination network to get N r ', calculate N l '+N r The difference loss between ' and N' is used as the consistent illumination loss;

[0050] The above training loss also includes consistent illumination loss, which is the weighted sum of adversarial illumination loss and consistent illumination loss.

[0051] It should be noted that the aforementioned re-illumination network is a pre-trained re-illumination network, and any existing pre-trained re-illumination network can be used. Preferably, in one optional embodiment, the pre-training method includes: training the re-illumination network using a pre-acquired pre-training set; the pre-training set includes multiple training pairs of re-illumination images under physical constraints; the training pairs include: image samples of an object under arbitrary lighting condition A and arbitrary lighting condition B, with the corresponding labels being images of the object under arbitrary lighting condition B from the same perspective; the image samples and their corresponding labels are composite images, which, in one optional embodiment, are synthesized using LightStage.

[0052] It should be noted that the loss function for calculating the difference loss can be MSE loss, VGG loss, etc., but is not limited to this.

[0053] It should be noted that random lighting conditions can be provided by randomly selected environment maps, spherical harmonic lighting, multi-point light sources, etc., which will not be described in detail here.

[0054] It should be noted that the re-illumination network in this embodiment can be any deep learning network, such as CNN, GAN, Transformer, Diffusion, etc., which is not limited here.

[0055] Considering that the estimation accuracy of most albedo and / or normals will directly affect the final re-rendering effect, preferably, the re-lighting network generally includes: an albedo network and / or a normal network, and a lighting rendering network.

[0056] In one optional implementation, the relighting network includes: a cascaded albedo network and a lighting rendering network;

[0057] The albedo network is used to perform eigendecomposition on image samples and generate corresponding albedo maps;

[0058] The lighting rendering network is used to re-render the image sample based on the lighting conditions and the albedo map of the image sample to obtain the corresponding re-lit image; or re-render the image sample based on the lighting conditions, the image sample and its albedo map to obtain the corresponding re-lit image;

[0059] In this case, users generally focus more on the accuracy of the albedo. To further improve training efficiency, the above-mentioned fine-tuning of the relighting network by minimizing the training loss includes: freezing the parameters in the lighting rendering network and adjusting the parameters in the albedo network with the goal of minimizing the training loss.

[0060] In one optional embodiment, the relighting network includes: a cascaded normal network and a lighting rendering network;

[0061] The normal network is used to perform eigendecomposition on image samples and generate corresponding normal maps;

[0062] The lighting rendering network is used to re-render the image sample based on the lighting condition and the normal map of the image sample to obtain a corresponding re-lit image; or, re-render the image sample based on the lighting condition, the image sample and its normal map and the lighting condition to obtain a corresponding re-lit image;

[0063] In this case, users generally focus more on the accuracy of normals. To further improve training efficiency, the above-mentioned fine-tuning of the relighting network by minimizing training loss includes: freezing the parameters in the lighting rendering network and adjusting the parameters in the normal network with the goal of minimizing training loss.

[0064] In an optional implementation, the relighting network includes: an albedo network, a normal network, and a lighting rendering network;

[0065] In this case, user needs generally focus on the accuracy of albedo or normal, or both. In order to further improve training efficiency, the above-mentioned fine-tuning of the re-lighting network by minimizing training loss includes: freezing the parameters in the lighting rendering network, and adjusting the parameters in the albedo network and / or normal network with the goal of minimizing training loss.

[0066] It should be noted that, at this time, there are many ways to connect the sub-networks in the re-illumination network, and there is no limitation. For example:

[0067] In an optional implementation, the albedo network and the normal network are connected in parallel and then in series with the lighting rendering network;

[0068] The albedo network is used to perform eigendecomposition on image samples and generate corresponding albedo maps;

[0069] The normal network is used to perform eigendecomposition on image samples and generate corresponding normal maps;

[0070] The lighting rendering network is used to re-render image samples based on lighting conditions, the albedo map of the image samples, and the normal map of the image samples to obtain corresponding re-lit images; or, based on lighting conditions, the image samples and their albedo map and normal map, to re-render image samples to obtain corresponding re-lit images.

[0071] In one optional implementation, the albedo network, the normal network, and the lighting rendering network are cascaded in sequence;

[0072] The albedo network is used to perform eigendecomposition on image samples and generate corresponding albedo maps;

[0073] The normal network is used to generate the corresponding normal map based on the image sample and the albedo map;

[0074] The lighting rendering network is used to re-render image samples based on lighting conditions, the albedo map of the image samples, and the normal map of the image samples to obtain corresponding re-lit images; or, based on lighting conditions, the image samples and their albedo map and normal map, to re-render image samples to obtain corresponding re-lit images.

[0075] In an optional implementation, the normal network, the albedo network, and the lighting rendering network are cascaded in sequence;

[0076] The normal network is used to perform eigendecomposition on image samples and generate corresponding normal maps;

[0077] The albedo network is used to generate the corresponding albedo map based on the image sample and the normal map;

[0078] The lighting rendering network is used to re-render image samples based on lighting conditions, the albedo map of the image samples, and the normal map of the image samples to obtain corresponding re-lit images; or, based on lighting conditions, the image samples and their albedo map and normal map, to re-render image samples to obtain corresponding re-lit images.

[0079] To further illustrate the self-supervised fine-tuning method of the re-illumination network provided by the present invention, the following is a detailed description using a portrait image as an example:

[0080] Figure 2 Schematic diagram of the method for establishing synthetic data pairs to combat illumination loss. This method creates paired training data pairs from real-world images to ensure that the training truth values ​​of the fine-tuned re-illumination network are still real-world images. The establishment process starts with a real-world portrait N, and the illumination information E of the portrait is extracted by existing methods (in this embodiment, the DECA algorithm is used to obtain the mask image of the portrait and combine the portrait with the mask image to extract the illumination information E of the portrait). This illumination information can be used as the illumination condition input for training. Next, by using a randomly selected environment map or spherical harmonic illumination as the illumination condition E * , N and E * Input the pre-trained re-illumination network and re-render N to get a picture that is the same as N under illumination E. * Portrait under heavy lighting N * .

[0081] Let N* be the portrait input of the re-illumination network during fine-tuning, E be the illumination condition input of the re-illumination network during fine-tuning, and N be the true training value of the re-illumination network.

[0082] Figure 3To establish a method for adversarial illumination loss, the output after inputting N* and E into the re-illumination network is N ** ; Figure 3 Where loss is a loss function, which can be MSE loss, VGG loss, etc., but is not limited to these.

[0083] Figure 4 To establish a consistent lighting loss method, a lighting information is divided into two parts according to the light intensity ratio as the lighting condition input of the re-lighting network, ensuring that the sum of the re-rendering effect values ​​under the two divided lighting conditions is equivalent to the re-rendering effect value under the undivided lighting condition. In this embodiment, the light intensity ratio is 1:1.

[0084] The process starts with a real-world portrait N, and uses a randomly selected environment map or spherical harmonic lighting as the lighting condition E', dividing E' into two lighting conditions E according to the light intensity ratio. l ' and E r '. Input N and E' into the relighting network to get the relighting image N', and then input N and E l 'Input the re-illumination network to get N l ', N and E r 'Input the re-illumination network to get N r '.

[0085] N l 'with N r 'Add together to get N", then the method for establishing the consistent illumination loss in this method is to calculate the loss between N" and N'; Figure 4 Where loss is a loss function, which can be MSE loss, VGG loss, etc., but is not limited to these.

[0086] Figure 5 A schematic flow chart of the self-supervised fine-tuning strategy provided for an embodiment of the present invention first utilizes the above-mentioned methods for establishing adversarial illumination loss and consistent illumination loss provided by the present invention to establish a loss function for self-supervised fine-tuning. The loss function is the weighted sum of the adversarial illumination loss and the consistent illumination loss. The weighting coefficients in the figure are all 1, and the specific weighting coefficients can be modified according to the optimization content and optimization situation.

[0087] The relighting network in this embodiment includes a cascaded albedo network and a lighting rendering network. The albedo network is fine-tuned using the final loss function. The fine-tuning method involves freezing the weights of all networks except the albedo network. Before each backpropagation of the loss function, synthetic data pairs with adversarial loss are generated based on the input image of the relighting network. The adversarial and consistent lighting losses are then calculated, and the weights of the fine-tuning network are updated through backpropagation.

[0088] In this embodiment, the methods for establishing robustness against illumination loss and consistent illumination loss, as well as the self-supervised fine-tuning method, are applicable to any re-illumination network that meets the requirements of the network framework of the present invention. It is easy to understand that the present invention does not require any additional details of network training; other hyperparameters can be set based on actual conditions.

[0089] In summary, to address the issues of insufficient albedo and normal accuracy estimated by existing deep learning relighting networks, as well as the resulting lack of realism in relighting results, this paper conducts a thorough analysis and research on the causes of these issues. The paper finds that traditional relighting networks directly use an albedo or normal sub-network to estimate the albedo and normal of the input image, and then use a re-rendering sub-network to combine the portrait, its albedo and normal, and the target lighting conditions to achieve relighting. In this approach, datasets must be acquired through LightStage. However, synthetic datasets acquired through LightStage differ from real-world portraits, resulting in a lack of realism in the relighting network's output. Furthermore, the cost of LightStage data acquisition limits the number and diversity of portraits in the dataset. The diversity of background images is also limited by the size of available environment maps (thousands of images) during rendering, which may affect the relighting network's generalization ability to real images. Furthermore, the accuracy of most albedo and normal estimates directly affects the final re-rendering results, resulting in inconsistencies between the target lighting and the re-rendered results, resulting in a lack of realism. Based on this discovery, the present invention accordingly proposes to create paired training data pairs from real-world images to ensure that the training truth values ​​of the fine-tuned relighting network are still real-world images, and at the same time, the output of the relighting network is consistent with the physical constraints in the real world.

[0090] Based on the above concepts, this paper proposes two new self-supervised fine-tuning loss methods: an adversarial illumination loss method and a consistent illumination loss method, and proposes a new self-supervised fine-tuning strategy for re-illumination deep networks. The adversarial illumination loss method creates paired training data from real-world images to ensure that the training ground truth of the fine-tuning network remains real-world images, improving the network's generalization performance. Furthermore, the ground truth during fine-tuning training is derived from the output of a pre-trained re-illumination network, eliminating the need for additional ground truth. This self-supervised fine-tuning method significantly reduces the reliance on high-quality, difficult data acquisition and, to a certain extent, the number of illumination conditions required by the re-illumination network, such as environment maps or spherical harmonic illumination. The consistent illumination loss method applies the physical consistency of illumination in the real world to deep learning network models, further improving the accuracy and generalization of the network's representation under diverse illumination conditions. The two loss methods are combined to create a final fine-tuning loss function. During fine-tuning, the weights of all networks except the albedo or normal network are frozen, and this loss function is used for network fine-tuning, further improving fine-tuning training efficiency.

[0091] In a second aspect, the present invention provides a re-illumination method, comprising:

[0092] Input the image to be re-illuminated and the target illumination conditions into the re-illumination network to obtain the corresponding illumination image;

[0093] The re-illumination network is fine-tuned using the self-supervised fine-tuning method for the re-illumination network provided by the first aspect of the present invention.

[0094] The related technical solutions are the same as the self-supervised fine-tuning method for the re-illumination network provided in the first aspect of the present invention, and will not be described in detail here.

[0095] In a third aspect, the present invention provides a re-lighting system, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the re-lighting method provided in the second aspect of the present invention when executing the computer program.

[0096] The related technical solutions are the same as the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the present invention and the re-illumination method provided in the second aspect, and will not be described in detail here.

[0097] In a fourth aspect, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, it controls the device where the storage medium is located to execute the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the present invention or the re-illumination method provided in the second aspect of the present invention.

[0098] The related technical solutions are the same as the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the present invention and the re-illumination method provided in the second aspect, and will not be described in detail here.

[0099] In a fifth aspect, the invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the invention or the re-illumination method provided in the second aspect of the invention.

[0100] The related technical solutions are the same as the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the present invention and the re-illumination method provided in the second aspect, and will not be described in detail here.

[0101] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A self-supervised fine-tuning method for re-illumination networks, characterized in that include: For each image sample N in the pre-collected image sample set, obtain the illumination condition E of N; Randomly obtain a lighting condition E * , N and E * Input into the pre-trained re-illumination network and get N in E * The re-illuminated image N under * ; Where N is an image sample in the real world; N * and E are input into the relighting network to obtain N ** , calculate N ** The difference loss between θ and N is used as the adversarial illumination loss, and a training loss including the adversarial illumination loss is constructed; Fine-tuning the relighting network by minimizing the training loss; The self-supervised fine-tuning method further includes: Randomly obtain an illumination condition E', input N and E' into the re-illumination network, and obtain the re-illumination image N' of N under E'; Divide E' into illumination conditions E according to the light intensity ratio l ' and E r ';N and E l 'Input the re-illumination network to get N l ', N and E r 'Input the re-illumination network to get N r ', calculate N l '+N r The difference loss between ' and N' is used as the consistent illumination loss; The training loss also includes the consistent illumination loss, specifically the weighted sum of the adversarial illumination loss and the consistent illumination loss.

2. The self-supervised fine-tuning method according to claim 1, characterized in that The DECA algorithm is used to obtain the lighting condition E of N.

3. The self-supervised fine-tuning method according to any one of claims 1 to 2, characterized in that: The re-illumination network includes: a cascaded albedo network and a lighting rendering network; The albedo network is used to perform eigendecomposition on image samples to generate corresponding albedo maps; The lighting rendering network is used to re-render the image sample based on the lighting conditions and the albedo map of the image sample to obtain a corresponding re-illuminated image; or re-render the image sample based on the lighting conditions, the image sample and its albedo map to obtain a corresponding re-illuminated image; Fine-tuning the relighting network by minimizing the training loss includes freezing parameters in the lighting rendering network and adjusting parameters in the albedo network with the goal of minimizing the training loss.

4. The self-supervised fine-tuning method according to any one of claims 1 to 2, characterized in that: The re-lighting network includes: a cascaded normal network and a lighting rendering network; The normal network is used to perform eigendecomposition on the image sample to generate a corresponding normal map; The lighting rendering network is used to re-render the image sample based on the lighting conditions and the normal map of the image sample to obtain a corresponding re-lit image; or to re-render the image sample based on the lighting conditions, the image sample and its normal map and the lighting conditions to obtain a corresponding re-lit image; Fine-tuning the relighting network by minimizing the training loss includes freezing parameters in the lighting rendering network and adjusting parameters in the normal network with the goal of minimizing the training loss.

5. The self-supervised fine-tuning method according to any one of claims 1 to 2, characterized in that: The re-illumination network includes: albedo network, normal network and lighting rendering network; Fine-tuning the relighting network by minimizing the training loss includes freezing parameters in the lighting rendering network and adjusting parameters in the albedo network and / or the normal network with the goal of minimizing the training loss.

6. A re-illumination method, characterized in that: include: Input the image to be re-illuminated and the target illumination conditions into the re-illumination network to obtain the corresponding illumination image; The re-illumination network is fine-tuned using the self-supervised fine-tuning method described in any one of claims 1 to 5.

7. A heavy lighting system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the re-illumination method according to claim 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the method according to any one of claims 1 to 6 when executed by a processor.

Images