Self-supervised fine tuning method of relighting network, relighting method and system

Abstract

The invention discloses a self-supervised fine tuning method of a relighting network, and a relighting method and system, and belongs to the technical field of relighting. Starting from an image sample N in the real world, obtaining an illumination condition E of the image sample N; inputting the N and a random illumination condition E * into a pre-trained relighting network to obtain a relighting image N * of the N under the E *; and then taking the relighting image N * as the image input of the relighting network during fine adjustment, E as the illumination condition input of the relighting network during fine adjustment, and N as the training true value of the relighting network, constructing adversarial illumination loss, and performing fine adjustment on the relighting network. According to the method, the paired training data pairs are created from the real world image, it can be ensured that the training truth value of the fine-tuning relighting network is still the real world image, the difference between the fine-tuning relighting network and the real world image is reduced, the generalization of the relighting network is high, and the reality sense of the relighting image output by the network is improved.

Description

Technical Field

[0001] The present invention belongs to the field of re-illumination technology, and more specifically, relates to a self-supervised fine-tuning method for a re-illumination network, a re-illumination method and a system. Background Art

[0002] In the field of computational imaging, relighting is a technique used to adjust and edit the lighting properties of a given object in order to present a relighted image that is consistent with the specified lighting conditions and close to reality. Taking the portrait relighting task as an example, in recent years, as an important part of the metaverse and virtual reality applications, the portrait relighting task has attracted much attention from academia and industry, and has important application value in digital photography, face recognition and other fields. The current portrait relighting method is fully combined with traditional imaging models, lighting models, three-dimensional face deformation models and deep learning technologies, and has made significant progress in the portrait relighting effect.

[0003] At present, the basic steps of the mainstream deep learning-based re-lighting method are: inverse rendering the object to obtain feature information such as albedo and normal, and then re-rendering combined with the feature information after inverse rendering to obtain the final re-lighting image.

[0004] LightStage, as a tool for obtaining datasets in most mainstream methods, provides re-illuminated image training pairs under physical constraints. However, the datasets obtained by the above methods are synthetic in nature. The background is obtained by projecting from the environment map, and the training truth of the re-illuminated network is generated by combining the foreground subject after re-illumination. However, these synthetic images are somewhat different from images in the real world, resulting in a lack of realism in the output of the model. In addition, due to the cost of LightStage data acquisition, the number and diversity of objects in the dataset are limited. The diversity of background images is also limited by the scale of available environment maps (several thousand) during rendering, which may affect the generalization ability of the network model to real images, and then affect the final re-rendering effect, resulting in inconsistency between the target lighting and the re-rendering effect, and a lack of realism. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a self-supervised fine-tuning method, a re-illumination method and a system for a re-illumination network, which aims to improve the realism of the re-illumination image output by the network.

[0006] In order to achieve the above objectives, in a first aspect, the present invention provides a self-supervised fine-tuning method for a re-illumination network, comprising:

[0007] For each image sample N in the pre-collected image sample set, obtain the illumination condition E of N; randomly obtain an illumination condition E * , N and E *Input into the pre-trained re-illumination network, and get N in E * The re-illuminated image N under * ; Where N is an image sample in the real world;

[0008] Input N* and E into the relighting network to get N ** , calculate N ** The difference loss between N and N is used as the adversarial illumination loss, and the training loss including the adversarial illumination loss is constructed;

[0009] The relighting network is fine-tuned by minimizing the training loss.

[0010] Further preferably, the self-supervised fine-tuning method of the re-illumination network further includes:

[0011] Randomly obtain a lighting condition E', input N and E' into the re-lighting network, and obtain the re-lighting image N' of N under E';

[0012] Divide E' into light conditions E according to the light intensity ratio l ' and E r '; Combine N and E l 'Input into the re-illumination network to get N l ', N and E r 'Input into the re-illumination network to get N r ', calculate N l '+N r The difference loss between ' and N' is used as the consistent illumination loss;

[0013] The above training loss also includes consistent illumination loss, which is the weighted sum of adversarial illumination loss and consistent illumination loss.

[0014] Further preferably, the re-illumination network comprises: a cascaded albedo network and a lighting rendering network;

[0015] The albedo network is used to perform intrinsic decomposition on image samples and generate the corresponding albedo map;

[0016] The lighting rendering network is used to re-render the image sample based on the lighting condition and the albedo map of the image sample to obtain the corresponding re-illuminated image; or, re-render the image sample based on the lighting condition, the image sample and its albedo map to obtain the corresponding re-illuminated image;

[0017] The fine-tuning of the relighting network by minimizing the training loss includes freezing the parameters in the lighting rendering network and adjusting the parameters in the albedo network with the goal of minimizing the training loss.

[0018] Further preferably, the re-illumination network comprises: a cascaded normal network and a lighting rendering network;

[0019] The normal network is used to perform eigendecomposition on image samples and generate the corresponding normal map;

[0020] The lighting rendering network is used to re-render the image sample based on the lighting condition and the normal map of the image sample to obtain the corresponding re-illuminated image; or, re-render the image sample based on the lighting condition, the image sample and its normal map and the lighting condition to obtain the corresponding re-illuminated image;

[0021] The above-mentioned fine-tuning of the relighting network by minimizing the training loss includes: freezing the parameters in the lighting rendering network and adjusting the parameters in the normal network with the goal of minimizing the training loss.

[0022] Further preferably, the re-illumination network includes: an albedo network, a normal network and a lighting rendering network;

[0023] The fine-tuning of the relighting network by minimizing the training loss includes freezing the parameters in the lighting rendering network and adjusting the parameters in the albedo network and / or the normal network with the goal of minimizing the training loss.

[0024] Further preferably, the DECA algorithm is used to obtain the lighting condition E of N.

[0025] In a second aspect, the present invention provides a re-illumination method, comprising:

[0026] Input the image to be re-illuminated and the target illumination conditions into the re-illumination network to obtain the corresponding illumination image;

[0027] Among them, the re-illumination network is fine-tuned using the self-supervised fine-tuning method for the re-illumination network provided by the first aspect of the present invention.

[0028] In a third aspect, the present invention provides a re-lighting system, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the re-lighting method provided in the second aspect of the present invention when executing the computer program.

[0029] In a fourth aspect, the present invention further provides a computer-readable storage medium, comprising a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the self-supervised fine-tuning method for the re-illumination network provided in the first aspect of the present invention or the re-illumination method provided in the second aspect of the present invention.

[0030] In a fifth aspect, the invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the invention or the re-illumination method provided in the second aspect of the invention.

[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0032] 1. The present invention provides a self-supervised fine-tuning method for a re-illumination network. Starting from an image sample N in the real world, its illumination condition E is obtained; N is compared with a random illumination condition E. * Input into the pre-trained re-illumination network, and get N in E * The re-illuminated image N under * ; Then, the re-illuminated image N* is used as the image input of the re-illuminated network during fine-tuning, E is used as the illumination condition input of the re-illuminated network during fine-tuning, and N is used as the training truth value of the re-illuminated network. The adversarial illumination loss is constructed to fine-tune the re-illuminated network. This method creates paired training data pairs from real-world images, which ensures that the training truth value of the fine-tuned re-illuminated network is still a real-world image, reducing the difference with the real-world image. The re-illuminated network has a high generalization ability, and it also greatly reduces the dependence on high-difficulty, high-quality image acquisition, improving the realism of the re-illuminated image output by the network.

[0033] 2. Further preferably, the self-supervised fine-tuning method of the re-illumination network provided by the present invention divides one lighting condition into two parts according to the light intensity ratio as the lighting condition input of the re-illumination network, ensuring that the sum of the re-rendering effect values ​​under the two divided lighting conditions is equivalent to the re-rendering effect value under the undivided lighting conditions, thereby constructing a consistent lighting loss, thereby ensuring that the output of the re-illumination network is consistent with the physical constraints in the real world, further improving the accuracy and generalization of the expression of the re-illumination network under a variety of lighting conditions, and further improving the realism of the re-illuminated image output by the network.

[0034] 3. Further preferably, the self-supervised fine-tuning method of the re-lighting network provided by the present invention takes into account that the estimation accuracy of most albedo and normal will directly affect the final re-rendering effect, resulting in inconsistency between the target lighting and the re-rendering effect, and lack of realism. When selecting the re-lighting network, the present invention preferably selects a re-lighting network with an albedo network and / or a normal network according to the user's requirements for albedo and / or normal accuracy, and when fine-tuning, gives priority to parameter adjustment in the albedo network and / or the normal network, freezes the parameters of the rest of the re-lighting network, and adjusts the parameters in the albedo network and / or the normal network with the goal of minimizing the training loss. This can improve the albedo and / or normal accuracy while ensuring the realism of the re-lighting effect, and also improves the fine-tuning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of a self-supervised fine-tuning method for a re-illumination network provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a method for establishing a synthetic data pair to combat illumination loss provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a method for counteracting light loss provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of a method for establishing consistent illumination loss provided by an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of the self-supervised fine-tuning strategy flow provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] In order to achieve the above objectives, firstly, Figure 1 As shown, the present invention provides a self-supervised fine-tuning method for a re-illumination network, comprising:

[0042] For each image sample N in the pre-collected image sample set, obtain the illumination condition E of N; randomly obtain an illumination condition E * , N and E * Input into the pre-trained re-illumination network, and get N in E *The re-illuminated image N under * ; Where N is an image sample in the real world;

[0043] Input N* and E into the relighting network to get N ** , calculate N ** The difference loss between N and N is used as the adversarial illumination loss, and the training loss including the adversarial illumination loss is constructed;

[0044] The relighting network is fine-tuned by minimizing the training loss.

[0045] It should be noted that the illumination condition E of N may be obtained by using methods such as DECA algorithm, DPR algorithm, SPLiT algorithm, etc., which are not limited here. Preferably, the illumination condition E of N is obtained by using DECA algorithm.

[0046] It should be noted that when constructing the training sample pairs required for adversarial illumination loss, the same image sample N can also be processed with different random illumination conditions based on the pre-trained re-illumination network to generate N re-illuminated images under different random illumination conditions; through the above method, multiple pairs of training sample pairs required for adversarial illumination loss can be constructed through one image sample N, further expanding the training sample set.

[0047] In an optional implementation manner, the self-supervised fine-tuning method of the re-illumination network further includes:

[0048] Randomly obtain a lighting condition E', input N and E' into the re-lighting network, and obtain the re-lighting image N' of N under E';

[0049] Divide E' into light conditions E according to the light intensity ratio l ' and E r '; Combine N and E l 'Input into the re-illumination network to get N l ', N and E r 'Input into the re-illumination network to get N r ', calculate N l '+N r The difference loss between ' and N' is used as the consistent illumination loss;

[0050] The above training loss also includes consistent illumination loss, which is the weighted sum of adversarial illumination loss and consistent illumination loss.

[0051] It should be noted that the above re-illumination network is a pre-trained re-illumination network, and any existing pre-trained re-illumination network can be used. Preferably, in an optional implementation, the pre-training method includes: using a pre-acquired pre-training set to train the re-illumination network; the pre-training set includes multiple re-illumination image training pairs under physical constraints; the training pairs include: image samples of an object under arbitrary lighting conditions A and arbitrary lighting conditions B, and the corresponding labels are images of the object under arbitrary lighting conditions B at the same viewing angle; the image samples and their corresponding labels are all synthetic images, which are synthesized using LightStage in an optional implementation. .

[0052] It should be noted that the loss function for calculating the difference loss may be MSE loss, VGG loss, etc., but is not limited thereto.

[0053] It should be noted that random lighting conditions can be provided by using randomly selected environment maps, spherical harmonic lighting, multi-point light sources, etc., which will not be elaborated here.

[0054] It should be noted that the re-illumination network in this embodiment can be any deep learning network, such as CNN, GAN, Transformer, Diffusion, etc., which is not limited here.

[0055] Considering that the estimation accuracy of most albedo and / or normal will directly affect the final re-rendering effect, therefore, preferably, the re-lighting network generally includes: an albedo network and / or a normal network, and a lighting rendering network.

[0056] In an optional implementation, the re-lighting network includes: a cascaded albedo network and a lighting rendering network;

[0057] The albedo network is used to perform intrinsic decomposition on image samples and generate the corresponding albedo map;

[0058] The lighting rendering network is used to re-render the image sample based on the lighting condition and the albedo map of the image sample to obtain the corresponding re-illuminated image; or, re-render the image sample based on the lighting condition, the image sample and its albedo map to obtain the corresponding re-illuminated image;

[0059] In this case, user needs generally focus more on the accuracy of the albedo. In order to further improve the training efficiency, the above-mentioned fine-tuning of the re-illumination network by minimizing the training loss includes: freezing the parameters in the lighting rendering network, and adjusting the parameters in the albedo network with the goal of minimizing the training loss.

[0060] In an optional implementation, the relighting network includes: a cascaded normal network and a lighting rendering network;

[0061] The normal network is used to perform eigendecomposition on image samples and generate the corresponding normal map;

[0062] The lighting rendering network is used to re-render the image sample based on the lighting condition and the normal map of the image sample to obtain the corresponding re-illuminated image; or, re-render the image sample based on the lighting condition, the image sample and its normal map and the lighting condition to obtain the corresponding re-illuminated image;

[0063] In this case, users generally focus more on the accuracy of normals. In order to further improve the training efficiency, the above-mentioned fine-tuning of the re-illumination network by minimizing the training loss includes: freezing the parameters in the lighting rendering network, and adjusting the parameters in the normal network with the goal of minimizing the training loss.

[0064] In an optional implementation, the re-lighting network includes: an albedo network, a normal network, and a lighting rendering network;

[0065] In this case, user needs generally focus on the accuracy of the albedo or the normal, or both. In order to further improve the training efficiency, the above-mentioned fine-tuning of the re-illumination network by minimizing the training loss includes: freezing the parameters in the lighting rendering network, and adjusting the parameters in the albedo network and / or the normal network with the goal of minimizing the training loss.

[0066] It should be noted that, at this time, there are many ways to connect the sub-networks in the re-illumination network, and there is no limitation. For example:

[0067] In an optional implementation, the albedo network and the normal network are connected in parallel and then in series with the lighting rendering network;

[0068] The albedo network is used to perform intrinsic decomposition on image samples and generate the corresponding albedo map;

[0069] The normal network is used to perform eigendecomposition on image samples and generate the corresponding normal map;

[0070] The lighting rendering network is used to re-render image samples based on lighting conditions, albedo maps of image samples and normal maps of image samples to obtain corresponding re-illuminated images; or, to re-render image samples based on lighting conditions, image samples and their albedo maps and normal maps to obtain corresponding re-illuminated images.

[0071] In an optional implementation, the albedo network, the normal network, and the lighting rendering network are cascaded in sequence;

[0072] The albedo network is used to perform intrinsic decomposition on image samples and generate the corresponding albedo map;

[0073] The normal network is used to generate the corresponding normal map based on the image sample and the albedo map;

[0074] The lighting rendering network is used to re-render image samples based on lighting conditions, albedo maps of image samples and normal maps of image samples to obtain corresponding re-illuminated images; or, to re-render image samples based on lighting conditions, image samples and their albedo maps and normal maps to obtain corresponding re-illuminated images.

[0075] In an optional implementation, the normal network, the albedo network, and the lighting rendering network are cascaded in sequence;

[0076] The normal network is used to perform eigendecomposition on image samples and generate the corresponding normal map;

[0077] The albedo network is used to generate the corresponding albedo map based on the image samples and the normal map;

[0078] The lighting rendering network is used to re-render image samples based on lighting conditions, albedo maps of image samples and normal maps of image samples to obtain corresponding re-illuminated images; or, to re-render image samples based on lighting conditions, image samples and their albedo maps and normal maps to obtain corresponding re-illuminated images.

[0079] In order to further illustrate the self-supervised fine-tuning method of the re-illumination network provided by the present invention, a portrait image is taken as an example to explain in detail as follows:

[0080] Figure 2 Schematic diagram of the method for establishing synthetic data pairs to combat illumination loss. This method creates paired training data pairs from real-world images to ensure that the training truth values ​​of the fine-tuned re-illumination network are still real-world images. The establishment process starts with a portrait N in the real world, and the illumination information E of the portrait is extracted by existing methods (the DECA algorithm is used in this embodiment to obtain the mask image of the portrait and extract the illumination information E of the portrait by combining the portrait and the mask image of the portrait). This illumination information can be used as the input of the training illumination condition. Next, by using a randomly selected environment map or spherical harmonic illumination as the illumination condition E * , N and E * Input the pre-trained re-illumination network and re-render N to obtain a similar image to N under illumination E. * Heavy lighting portrait N * .

[0081] Let N* be the portrait input of the re-illumination network during fine-tuning, E be the lighting condition input of the re-illumination network during fine-tuning, and N be the training truth value of the re-illumination network.

[0082] Figure 3To establish a method for adversarial illumination loss, the output of N* and E after inputting them into the re-illumination network is N ** ; Figure 3 Where loss is a loss function, which may be MSE loss, VGG loss, etc., but is not limited thereto.

[0083] Figure 4 In order to establish a method for consistent lighting loss, a lighting information is divided into two parts according to the light intensity ratio as the lighting condition input of the re-lighting network, ensuring that the sum of the re-rendering effect values ​​under the two divided lighting conditions is equivalent to the re-rendering effect value under the undivided lighting condition. In this embodiment, the light intensity ratio is 1:1.

[0084] The process starts with a real-world portrait N, and uses a randomly selected environment map or spherical harmonic lighting as the lighting condition E', dividing E' into two lighting conditions E according to the light intensity ratio. l ' and E r '. Input N and E' into the relighting network to get the relighting image N'. l 'Input into the re-illumination network to get N l ', N and E r 'Input into the re-illumination network to get N r '.

[0085] N l 'With N r 'Add together to get N", then the consistent illumination loss establishment method in this method is to calculate the loss between N" and N'; Figure 4 Where loss is a loss function, which may be MSE loss, VGG loss, etc., but is not limited thereto.

[0086] Figure 5 A schematic diagram of the self-supervised fine-tuning strategy flow provided for an embodiment of the present invention, firstly using the above-mentioned adversarial illumination loss establishment method and consistent illumination loss establishment method provided by the present invention to establish a loss function for self-supervised fine-tuning, the loss function is the weighted sum of the adversarial illumination loss and the consistent illumination loss, the weighting coefficients in the figure are all 1, and the specific weighting coefficients can be modified according to the optimization content and optimization situation.

[0087] The relighting network in this embodiment includes: a cascaded albedo network and a lighting rendering network. The albedo network is fine-tuned using the final loss function, and the fine-tuning method is: freezing the weights of other networks except the albedo network, generating synthetic data pairs of adversarial loss based on the input portrait of the relighting network before each back-propagation of the loss function, and then calculating the adversarial and consistent lighting losses, and then back-propagating to update the weights of the fine-tuning network.

[0088] In this embodiment, the method for establishing the anti-illumination loss and the consistent illumination loss, and the self-supervised fine-tuning method can be applied to any re-illumination network that meets the network framework requirements of the present invention. It is easy to understand that there is no additional requirement for other details of network training in the present invention, and other hyperparameters can be set according to actual conditions.

[0089] In summary, in order to solve the problems of insufficient accuracy of albedo and normal estimated by existing deep learning relighting networks, and lack of realism in the final relighting effect, the present invention has fully analyzed and studied the causes of the problem. The present invention finds that the traditional relighting network directly uses an albedo or normal sub-network to estimate the albedo and normal of the input image, and then uses a re-rendering sub-network to combine the portrait and its albedo and normal, and the target lighting conditions to achieve relighting. Under the above method, the data set needs to rely on LightStage to obtain. However, the synthetic data set obtained by LightStage has certain differences from the portrait photos in the real world, resulting in the lack of realism in the output of the relighting network. In addition, due to the cost of LightStage data acquisition, the number and diversity of portraits in the data set are limited. The diversity of background images is also limited by the scale of available environment maps (several thousand) during rendering, which may affect the generalization ability of the relighting network to real images. Moreover, the estimation accuracy of most albedo and normal will directly affect the final re-rendering effect, resulting in inconsistency between the target lighting and the re-rendering effect, and lack of realism. Based on this discovery, the present invention accordingly proposes to create paired training data pairs from real-world images to ensure that the training truth values ​​of the fine-tuned re-illumination network are still real-world images, and at the same time, the output of the re-illumination network is consistent with the physical constraints in the real world.

[0090] Based on the above concept, the present invention proposes a new self-supervised fine-tuning loss establishment method: an adversarial illumination loss establishment method and a consistent illumination loss establishment method, and proposes a new self-supervised fine-tuning strategy for re-illumination deep networks. The adversarial illumination loss establishment method creates paired training data pairs from real-world images to ensure that the training truth value of the fine-tuning network is still a real-world image, thereby improving the generalization performance of the network; and the truth value during fine-tuning training comes from the output of the pre-trained re-illumination network, and no additional truth value is required. This fine-tuning method is a self-supervised method, which greatly reduces the dependence on high-difficulty, high-quality data acquisition, and also reduces the number of illumination conditions such as environment maps or spherical harmonic illumination for the re-illumination network to a certain extent. The consistent illumination loss establishment method applies the illumination consistency in the physical sense of the real world to the deep learning network model, further improving the accuracy and generalization of the network expression under various illumination conditions. The final fine-tuning loss function is established by combining the above two loss establishment methods. During fine-tuning, the weights of other networks except the albedo network or the normal network are frozen, and the network is fine-tuned using this loss function, further improving the efficiency of fine-tuning training.

[0091] In a second aspect, the present invention provides a re-illumination method, comprising:

[0092] Input the image to be re-illuminated and the target illumination conditions into the re-illumination network to obtain the corresponding illumination image;

[0093] Among them, the re-illumination network is fine-tuned using the self-supervised fine-tuning method for the re-illumination network provided by the first aspect of the present invention.

[0094] The related technical solution is the same as the self-supervised fine-tuning method for the re-illumination network provided in the first aspect of the present invention, and will not be described in detail here.

[0095] In a third aspect, the present invention provides a re-lighting system, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the re-lighting method provided in the second aspect of the present invention when executing the computer program.

[0096] The related technical solutions are the same as the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the present invention and the re-illumination method provided in the second aspect, and will not be described in detail here.

[0097] In a fourth aspect, the present invention further provides a computer-readable storage medium, comprising a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the self-supervised fine-tuning method for the re-illumination network provided in the first aspect of the present invention or the re-illumination method provided in the second aspect of the present invention.

[0098] The related technical solutions are the same as the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the present invention and the re-illumination method provided in the second aspect, and will not be described in detail here.

[0099] In a fifth aspect, the invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the invention or the re-illumination method provided in the second aspect of the invention.

[0100] The related technical solutions are the same as the self-supervised fine-tuning method of the re-illumination network provided in the first aspect of the present invention and the re-illumination method provided in the second aspect, and will not be described in detail here.

[0101] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A self-supervised fine-tuning method for a re-illuminated network, characterized in that: include: For each image sample N in the pre-collected image sample set, obtain the illumination condition E of N; Randomly obtain a lighting condition E * , N and E * Input into the pre-trained re-illumination network, and get N in E * The re-illuminated image N under * ; Where N is an image sample in the real world; Input N* and E into the relighting network to get N ** , calculate N ** The difference loss between θ and N is used as the adversarial illumination loss, and a training loss including the adversarial illumination loss is constructed; The relighting network is fine-tuned by minimizing the training loss.

2. The self-supervised fine-tuning method according to claim 1, characterized in that: Also includes: Randomly obtain an illumination condition E', input N and E' into the re-illumination network, and obtain a re-illuminated image N' of N under E'; Divide E' into light conditions E according to the light intensity ratio l ' and E r '; Combine N and E l 'Input the re-illumination network to get N l ', N and E r 'Input the re-illumination network to get N r ', calculate N l '+N r The difference loss between ' and N' is used as the consistent illumination loss; The training loss also includes the consistent illumination loss, which is specifically a weighted sum of the adversarial illumination loss and the consistent illumination loss.

3. The self-supervised fine-tuning method according to claim 1, characterized in that: The DECA algorithm is used to obtain the lighting condition E of N.

4. The self-supervised fine-tuning method according to any one of claims 1 to 3, characterized in that: The re-illumination network includes: a cascaded albedo network and a lighting rendering network; The albedo network is used to perform intrinsic decomposition on the image samples to generate corresponding albedo maps; The lighting rendering network is used to re-render the image sample based on the lighting condition and the albedo map of the image sample to obtain the corresponding re-illuminated image; or re-render the image sample based on the lighting condition, the image sample and its albedo map to obtain the corresponding re-illuminated image; Fine-tuning the re-illumination network by minimizing the training loss includes: freezing parameters in the illumination rendering network, and adjusting parameters in the albedo network with the goal of minimizing the training loss.

5. The self-supervised fine-tuning method according to any one of claims 1 to 3, characterized in that: The re-illumination network includes: a cascaded normal network and a lighting rendering network; The normal network is used to perform eigendecomposition on the image sample to generate a corresponding normal map; The lighting rendering network is used to re-render the image sample based on the lighting condition and the normal map of the image sample to obtain a corresponding re-illuminated image; or, re-render the image sample based on the lighting condition, the image sample and its normal map and the lighting condition to obtain a corresponding re-illuminated image; Fine-tuning the re-illumination network by minimizing the training loss includes: freezing parameters in the illumination rendering network, and adjusting parameters in the normal network with the goal of minimizing the training loss.

6. The self-supervised fine-tuning method according to any one of claims 1 to 3, characterized in that: The re-illumination network includes: an albedo network, a normal network and a lighting rendering network; Fine-tuning the re-illumination network by minimizing the training loss includes: freezing parameters in the illumination rendering network, and adjusting parameters in the albedo network and / or the normal network with the goal of minimizing the training loss.

7. A re-illumination method, characterized in that: include: Input the image to be re-illuminated and the target illumination conditions into the re-illumination network to obtain the corresponding illumination image; Wherein, the re-illumination network is fine-tuned by using the self-supervised fine-tuning method described in any one of claims 1-6.

8. A heavy lighting system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the re-illumination method according to claim 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images