Rendering method and device, equipment and storage medium

Abstract

The invention provides a relighting method, device and equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC content generation based on artificial intelligence. According to the specific implementation scheme, original image features and illumination image features are obtained based on an original image and an illumination image; obtaining a first feature based on the original image feature and the illumination image feature; and processing the first feature by using a diffusion model to obtain a relighting map. According to the invention, the diffusion model can be used for processing the original image features and the illumination image features, and a high-quality relighting image is obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Background Art

[0002] There are many ways to implement relighting of scene images. Relighting methods based on multi-view images require a large amount of collected data to achieve full coverage of the scene. The rendering formula used by the inverse rendering method is an approximation of the scene, and the relighting effect is poor. Relighting methods based on single images require collecting specific data sets to train the network to decompose the image into multiple attribute graphs, and then synthesize the relighting image in the relighting scene by approximating the physical formula. The generated new image has many deficiencies compared to the actual lighting performance in the real scene. Summary of the invention

[0003] The present disclosure provides a re-illumination method, device, equipment and storage medium.

[0004] According to one aspect of the present disclosure, a re-illumination method is provided, comprising:

[0005] Based on the original image and the illumination map, the original image feature and the illumination map feature are obtained respectively;

[0006] Obtaining a first feature based on the original image feature and the illumination image feature;

[0007] The first feature is processed using a diffusion model to obtain a re-illumination map.

[0008] According to another aspect of the present disclosure, there is provided a re-illumination device, comprising:

[0009] A feature acquisition module, used to obtain original image features and illumination map features based on the original image and the illumination map respectively;

[0010] A feature processing module, used for obtaining a first feature based on the original image feature and the illumination image feature;

[0011] The re-illumination module is used to process the first feature using a diffusion model to obtain a re-illumination map.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] According to the present disclosure, a diffusion model can be used to process the original image features and the illumination map features to obtain a high-quality re-illumination map.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is a flow chart of a re-illumination method according to an embodiment of the present disclosure;

[0022] Figure 2 is a flow chart of a re-illumination method according to another embodiment of the present disclosure;

[0023] Figure 3 is a flow chart of a re-illumination method according to another embodiment of the present disclosure;

[0024] Figure 4 is a flow chart of a re-illumination method according to another embodiment of the present disclosure;

[0025] Figure 5 is a re-illumination flow chart of an embodiment of the present disclosure;

[0026] Figures 6a to 6f It is a rendering of the effect before and after heavy illumination using the embodiment of the present disclosure;

[0027] Figure 7 is a structural schematic diagram of a heavy illumination device according to an embodiment of the present disclosure;

[0028] Figure 8 is a structural schematic diagram of a re-illumination device according to another embodiment of the present disclosure;

[0029] Fig. 9 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] For the image relighting task, it can be implemented in the following forms, for example:

[0032] The first category is the multi-view image reconstruction method, which mainly collects multi-view images of the scene, obtains the pose of each image through the "Structure-from-Motion" method (SfM), obtains the geometry, physical material properties, ambient light estimation, etc. of the entire scene through the inverse rendering method (such as through NeRFactor, NVDiffrec, Neural-PBIR and other tools), and renders a new image under re-lighting by changing the ambient lighting. The method based on multi-view images requires a large amount of collected data to achieve full coverage of the scene, but it is difficult to obtain data in many actual business scenarios; the rendering formula used by the inverse rendering method is an approximation of the scene estimate, and inverse rendering itself is an ill-posed problem, resulting in visible defects in the re-lighting effect.

[0033] The second type of method is based on a single image. It collects a large amount of data to train a neural network that can decompose the image into different layers (including but not limited to normal vectors, material maps, highlight maps, etc.), and then re-illuminates the image by introducing new environmental lighting. The single-image-based method decomposes the image into multiple attribute maps by collecting a specific data set to train the network, and then synthesizes the re-illuminated image in the re-illuminated scene by approximating the physical formula. However, this type of method also causes defects in the way it synthesizes new images, and there are still many shortcomings compared to the lighting performance in real scenes.

[0034] Figure 1 1 is a flow chart of a re-illumination method 100 according to an embodiment of the present disclosure, the method comprising:

[0035] S101, obtaining original image features and illumination map features based on the original image and the illumination map respectively;

[0036] S102, obtaining a first feature based on the original image feature and the illumination image feature;

[0037] S103: Use a diffusion model to process the first feature to obtain a re-illumination map.

[0038] In the disclosed embodiment, the original image may include a scene image and a background image to be processed, which may be understood as an image that needs to be re-illuminated. Original image features may be extracted from the original image, and illumination map features may be extracted from the illumination map. Original image features may also be referred to as original image latent vectors, scene image latent vectors, background image latent vectors, etc., and illumination map features may also be referred to as illumination map latent vectors. In the disclosed embodiment, corresponding illumination map features may be directly extracted from the illumination map; or a corresponding illumination map may be generated based on the input illumination information, and then corresponding illumination map features may be extracted based on the generated illumination map.

[0039] In the embodiment of the present disclosure, the first feature can be obtained by splicing based on the original image feature and the illumination map feature. For example, the splicing method may include splicing based on the resolution of the original image and the illumination map. The first feature after splicing may include one or more of the size, resolution, content, light source type, light source position, illumination intensity, etc. of the image, which is not limited in the present disclosure.

[0040] In the disclosed embodiment, the diffusion model is a generative model. The diffusion model can perform a forward process (diffusion process) and a reverse process. Taking an image as an example, Gaussian noise can be continuously added to the image through a diffusion process, and a pure random noise distribution is finally obtained. The original image data is then restored from the random noise through the inverse process (reverse process) of the diffusion process to obtain the image distribution state transfer at each step in the inverse process. Using the diffusion model to process the first feature, noise such as Gaussian noise can be added to the first feature during the diffusion process, and in the reverse process, the feature after the noise is added can be denoised to obtain a high-quality re-illuminated map.

[0041] According to the embodiments of the present disclosure, the diffusion model can be used to process the original image features and the illumination map features to obtain a high-quality re-illumination map. The method of the embodiments of the present disclosure can reduce the number of images required for re-illumination and reduce the consumption of computing resources. The method of the embodiments of the present disclosure also does not need to decompose the image, which reduces the difficulty of re-illumination processing, and the re-illumination map has a stronger correlation with the original image features and the illumination map features.

[0042] Figure 2 2 is a flow chart of a relighting method 200 according to another embodiment of the present disclosure, and the method 200 can be used to implement step S103 in the relighting method 100. In one embodiment, the method 200 includes: processing the first feature using a diffusion model to obtain a relighting map, and further includes:

[0043] S201, extracting style structure features from the original image;

[0044] S202, using the first diffusion model to add noise and denoise the first feature and the style structure feature to obtain a second feature;

[0045] S203: Based on the second feature, obtain a re-illumination map.

[0046] In the embodiment of the present disclosure, the style structure features can be extracted from the original image through the style structure control network. The style structure control network can be a network outside the first diffusion model. The style structure control network can also be a part of the first diffusion model. For example, the first diffusion model includes a style control network and a diffusion network. The style structure control network can be called a style structure processing module. The style structure control network can include components such as sketch control, image prompt adapter (IP-Adapter), bootstrapping language-image pre-training (BLIP) splicing prompt word (Prompt), etc., which are not limited in the present disclosure.

[0047] In the disclosed embodiment, the style structure feature and the first feature can be input into the first diffusion model (or diffusion network). The first diffusion model (or diffusion network) can fuse the style structure feature and the first feature based on the cross attention mechanism, and add noise to the fused feature based on Gaussian noise, and then denoise the first feature after adding noise to obtain a new feature, i.e., the second feature. The second feature may include content information of the original image, new lighting information, style structure information of the original image, etc.

[0048] According to the embodiments of the present disclosure, the style structure features of the original image can be used again on the first feature including the lighting map feature to constrain the whole, thereby improving the consistency of the style and structure of the re-illuminated image with the original image, so as to keep the content and semantics of the original image intact and improve the quality of the re-illuminated image.

[0049] Figure 3 3 is a flow chart of a relighting method 300 according to another embodiment of the present disclosure, and the method 300 can be used to implement step S203 in the relighting method 200. In one embodiment, the method 300 includes: obtaining a relighting map based on the second feature, and further includes:

[0050] S301, extracting color features from the original image;

[0051] S302, using a second diffusion model to add noise and denoise the second feature and the color feature to obtain a third feature;

[0052] S303: Post-process the third feature to obtain the re-illumination map.

[0053] In the disclosed embodiment, color features can be extracted from the original image through a color control network. The color control network can be a network outside the second diffusion model. The color control network can be a part of the second diffusion model. For example, the second diffusion model includes a color control network and a diffusion network. The color control network can be called a color processing module.

[0054] In the disclosed embodiment, the color feature obtained by processing the original image with the color control network and the second feature obtained using the first diffusion model can be input into the second diffusion model (or diffusion network). The second diffusion model (or diffusion network) can fuse the color feature and the second feature based on the cross attention mechanism, and add noise to the fused feature based on Gaussian noise, and then denoise the second feature after adding noise to obtain a new feature, i.e., the third feature. The third feature may include the content information of the original image, the new lighting information, the style structure information of the original image, the color information of the original image, etc.

[0055] According to the embodiment of the present disclosure, the color feature of the original image can be used again to constrain the whole on the second feature including the lighting feature and the style structure feature, thereby improving the color consistency between the re-illuminated image and the original image, so as to keep the color of the original image intact and improve the quality of the re-illuminated image.

[0056] In one implementation, S303 post-processes the third feature to obtain the re-illumination map, including: using a decoder of a variational autoencoder to decode the third feature to generate a re-illumination map.

[0057] In the disclosed embodiment, the original image features and the illumination map features can be obtained based on the variational autoencoder (VAE). VAE is an improved autoencoder model that uses probabilistic encoding and decoding to generate new data by sampling from the latent space and can perform unsupervised learning. In the encoding of the variational autoencoder, the feature of each image is a probability distribution. In this way, each latent feature is represented as a concept distribution. When decoding from the latent state, random sampling can be performed from each latent state distribution to generate a vector as the input of the decoder model. This allows the decoder model to reconstruct the input.

[0058] In the disclosed embodiment, VAE is used to obtain the original image features and the illumination map features respectively, and after splicing and diffusion processing, a third feature including content information of the original image, new illumination information, style structure information of the original image, color information of the original image, etc. is obtained. Therefore, the decoder corresponding to the variational autoencoder can be used to decode the third feature, and the image can be reconstructed according to the third feature. The reconstructed new image may include the original image features and the new illumination effect, i.e., the re-illumination map.

[0059] According to the embodiment of the present disclosure, a re-lighting map of the original image can be generated based on the third feature including content information of the original image, new lighting information, style structure information of the original image, color information of the original image, etc., thereby improving the speed of re-lighting.

[0060] In one implementation, S303 post-processes the third feature to obtain the re-illumination map, and further includes at least one of the following:

[0061] Performing image frequency analysis on the re-illumination map;

[0062] A color matching algorithm is used to align the colors of the original image with the colors of the re-illumination map.

[0063] In the disclosed embodiment, the image spectrogram can describe the distribution of different frequency components in the image. By analyzing the spectrogram, the texture, structure and other information of the image can be deeply understood. In a digital image, the pixel value is usually a two-dimensional matrix, which represents the brightness or color information of the image. The spectrogram is a Fourier transform of this two-dimensional matrix to obtain a complex matrix, which contains the frequency information of the image. By analyzing the distribution of the spectrogram, the high-frequency and low-frequency components in the image and their relative intensities can be obtained. By analyzing the real and imaginary spectrograms, the high-frequency and low-frequency components in the image can be distinguished. By analyzing the distribution of the spectrogram, the image can be filtered, denoised, and feature extracted.

[0064] In the disclosed embodiment, after the relighting map is generated, frequency analysis can be performed on the relighting map and the original image, and the high-frequency portion of the original image can be compared with the high-frequency portion of the relighting map to obtain the edges, textures and other details of the original image, and the high-frequency of the original image can be used to supplement the high-frequency details of the relighting map.

[0065] In the disclosed embodiment, a color matching algorithm may also be used to compare the colors of the re-illumination map and the original image, and the color of the re-illumination map may be adjusted using the color information of the original image to align the color of the re-illumination map with the original image.

[0066] According to the embodiments of the present disclosure, the generated re-lighting map can be further processed according to the original image to improve the consistency of the re-lighting map with the original image in color, structure, and details, thereby improving the quality of the re-lighting map.

[0067] Figure 4 4 is a flow chart of a re-illumination method 400 according to another embodiment of the present disclosure, and the method 400 can be used to implement step S101 in the re-illumination method 100. In one embodiment, the method 400 includes: obtaining original image features and illumination map features based on the original image and the illumination map, respectively, and further includes:

[0068] S401, using a variational autoencoder to process the original image to obtain features of the original image;

[0069] S402: Use a variational autoencoder to process the illumination map to obtain illumination map features.

[0070] In the disclosed embodiment, a variational autoencoder may be used to encode the original image to obtain features of the original image including original image content information, resolution information, size information, and the like.

[0071] In the disclosed embodiment, after the image is input into the variational autoencoder, image features with a resolution and a number of channels of w×h×c can be obtained, where w can represent the width (horizontal direction) of the image, h can represent the height (vertical direction) of the image, and c can represent the number of channels of the image. The channels of the image can include features related to the content of the image.

[0072] In the disclosed embodiment, the original image can be encoded using a variational autoencoder to obtain the original image features with the original image content information, and the illumination map can be encoded using a variational autoencoder to obtain the illumination map features representing the illumination information. For example, after the original image is input into the variational autoencoder, the original image features of w×h×c can be obtained, and after the illumination map is input into the variational autoencoder, the illumination map features of w×h×c can be obtained. After the two are combined, the resolution and number of channels of the combined features are w×h×2c.

[0073] In the disclosed embodiment, a variational autoencoder based on different training data amounts, resolutions, etc. can obtain features (latent vectors) with different resolutions and numbers of channels.

[0074] According to the embodiments of the present disclosure, the original image and the illumination map may be encoded to obtain features in a consistent format, thereby facilitating subsequent processing of the image features.

[0075] In one implementation, S102 obtains a first feature based on the original image feature and the illumination map feature, including: concatenating the original image feature and the illumination map feature in a channel dimension to obtain the first feature.

[0076] In the disclosed embodiment, the original image features and the illumination map features can be spliced. When splicing, the w×h of the original image can be kept unchanged, that is, the size of the original image is kept unchanged, and the original image features and the illumination map features are spliced ​​in the channel dimension to obtain a first feature of w×h×2c. The first feature may include features related to information such as the size, resolution, and content of the original image, and may also include features related to illumination information obtained from the illumination map.

[0077] According to the embodiment of the present disclosure, the original image features and the illumination map features may be spliced ​​to obtain a first feature containing the original image information and the illumination information, thereby providing a basis for a subsequent diffusion model to fuse the features.

[0078] In one implementation, the diffusion model is obtained by fine-tuning an initial diffusion model based on a training set, where the training set includes a plurality of training samples, and one training sample includes an original image sample, an illumination image sample, and a re-illumination image sample.

[0079] In the disclosed embodiment, the diffusion model may be a large model, and the initial diffusion model may be trained or fine-tuned based on the training set. The initial first diffusion model may include a style structure control network, and the initial first diffusion model may be fine-tuned using a training set of multiple groups of training samples including original image samples, illumination map samples, and re-illumination map samples. During fine-tuning, information such as color may be ignored first, and the consistency of the style structure between the image generated using the original image samples and the illumination map samples and the re-illumination map samples may be mainly compared. When the consistency of the style structure between the generated image and the re-illumination map sample is lower than the style threshold, the fine-tuning ends, and the fine-tuned first diffusion model is obtained.

[0080] In the disclosed embodiment, the initial second diffusion model may include a color control network, and the initial second diffusion model may be fine-tuned using a training set of multiple groups of training samples including original image samples, illumination map samples, and re-illumination map samples. During fine-tuning, information such as style structure may be ignored, and the color consistency between the image generated using the original image samples and the illumination map samples and the re-illumination map samples may be mainly compared. When the color consistency between the generated image and the re-illumination map samples is lower than the color threshold, the fine-tuning ends and the fine-tuned second diffusion model is obtained. In another embodiment, the training samples of the second diffusion model may also be images and re-illumination map samples output by the trained first diffusion model, and the color consistency between the image output by the first diffusion model and the re-illumination map samples may be compared.

[0081] According to the embodiments of the present disclosure, a first diffusion model that can perform style structure constraints on the re-illumination map and a second diffusion model that can perform color constraints on the re-illumination map can be trained to improve the image processing capability and efficiency of the diffusion model.

[0082] In one embodiment, the method further includes: performing illumination image conversion on the illumination parameters to generate an illumination map.

[0083] In the embodiments of the present disclosure, the illumination information may include the illumination requirements input by the user, or the illumination requirements queried according to the needs or selections of the user, and the present disclosure does not limit this. For example, the illumination information may be information such as a high dynamic range image (HDR) panorama, a point light source position direction intensity, a surface light source position direction intensity, etc. A illumination image conversion model may be pre-trained, and the illumination image conversion model may generate an illumination image based on the illumination information.

[0084] According to the embodiments of the present disclosure, after the user inputs the lighting information, the lighting information can be automatically used to generate a corresponding lighting image, so that when the user uses the present disclosure, the corresponding re-lighting map can be generated by inputting the original image and his own lighting requirements, thereby increasing the scope of application of the re-lighting and improving the ease of use of the re-lighting.

[0085] The re-lighting method proposed in the present disclosure includes a scene re-lighting method based on a single image, which does not require reconstruction of the scene, and improves the stability and effect of the re-lighting of the entire scene by introducing a stable diffusion model. For example, the scene re-lighting method based on a single image includes a re-lighting method based on an indoor scene image, which is used for re-lighting under the condition of a single image of the scene without the need for 3D reconstruction. Unlike previous methods, the embodiments of the present disclosure do not require dense reconstruction of the scene, and introduce a stable diffusion model to solve the defects of re-lighting images based on physical formulas or their approximate synthesis.

[0086] Figure 5 It is a re-lighting flow chart of an embodiment of the present disclosure, wherein the input part includes two parts: the "input scene graph" is the original scene graph of the target re-lighting, that is, the original image (referred to as the original image) whose original lighting the user wishes to change. The input lighting may include lighting information specified by the user (referred to as lighting) or lighting information generated according to the original image. Optionally, the lighting representation may include information such as a high dynamic range image (HDR) panorama, the position direction intensity of point light sources, and the position direction intensity of surface light sources. The output part may include a "re-lighting scene graph" (referred to as the re-lighting graph), which is a re-lighting image with the same resolution as the "input scene graph", unchanged image elements and theme colors, and only changes in lighting and shadows.

[0087] The disclosed embodiment inputs the scene graph and the illumination map converted from the illumination image into the encoder of the variational auto encoder (VAE) to obtain the background image latent vector (Background Latent) and the illumination image latent vector (Light Latent). Optionally, the variational auto encoder based on different training data amounts and resolutions can obtain latent vectors of different resolutions and numbers of channels, and the resolution and number of channels are denoted as w×h×c. By splicing these two latent vectors in the channel dimension, a new latent vector is obtained, whose size is w×h×2c.

[0088] Next, the merged vector can be sent to the trained diffusion model for denoising and denoising to obtain a new latent vector. In this process, the disclosed embodiment introduces a style and structure control network to keep the content and semantics of the original image intact. Optionally, Sketch control, IP-Adapter, BLIP splicing prompt, etc. can be used as modules to maintain the style and structure control network.

[0089] In the next stage, the denoised latent vector is fed into the next pre-trained diffusion model, and the features extracted by the color control network are added to the diffusion model in this stage to maintain the overall color consistency, and a smaller noise ratio is used to avoid a large change in the latent vector input to this stage.

[0090] After completing the above steps, the latent vector is sent to the post-processing module for post-processing. This stage includes but is not limited to the following methods: obtaining an image through the VAE decoder, supplementing the high-frequency details of the original image through image frequency analysis and other methods, and slightly aligning the overall color of the original image with the overall color of the re-lighting through a color matching algorithm, and finally obtaining the output re-lighting image. Figures 6a to 6f : is a picture showing the effect of using the embodiment of the present disclosure before and after re-illumination. Figure 6a is the input scene graph, Figure 6b yes Figure 6a The corresponding heavily illuminated scene graph; Figure 6c is the input scene graph, Figure 6d yes Figure 6c The corresponding heavily illuminated scene graph; Figure 6e is the input scene graph, Figure 6f yes Figure 6e The corresponding re-lit scene graph.

[0091] Optionally, the diffusion model can be fine-tuned on synthetic paired data based on a model trained on large-scale vinculum models, retaining its high-quality image generation capabilities.

[0092] Figure 7is a schematic structural diagram of a re-illumination device 700 according to an embodiment of the present disclosure. The device 700 may include:

[0093] A feature acquisition module 701 is used to obtain original image features and illumination map features based on the original image and the illumination map respectively;

[0094] A feature processing module 702, configured to obtain a first feature based on the original image feature and the illumination image feature;

[0095] The re-illumination module 703 is used to process the first feature using a diffusion model to obtain a re-illumination map.

[0096] Figure 8 800 according to another embodiment of the present disclosure, the device 800 includes: a feature acquisition module 801, a feature processing module 802 and a re-illumination module 803. The functions of the above modules can refer to the functions of the modules of the re-illumination device in the above embodiment. In one implementation, the re-illumination module 803 may include:

[0097] The style submodule 8031 ​​is used to extract style structure features from the original image;

[0098] A first diffusion submodule 8032 is used to use the first diffusion model to add noise and denoise the first feature and the style structure feature to obtain a second feature;

[0099] In one implementation, the re-lighting module 803 is further configured to obtain a re-lighting map based on the second feature.

[0100] In one embodiment, if Figure 8 As shown, the re-illumination module 803 may further include:

[0101] A color submodule 8033 is used to extract color features from the original image;

[0102] A second diffusion submodule 8034 is used to add noise and denoise the second feature and the color feature using a second diffusion model to obtain a third feature;

[0103] The post-processing submodule 8035 is used to post-process the third feature to obtain the re-illumination map.

[0104] In one implementation, the post-processing submodule 8035 is further configured to use a decoder of a variational autoencoder to decode the third feature and generate a re-illumination map.

[0105] In one implementation, the post-processing submodule 8035 is further used for at least one of the following:

[0106] Performing image frequency analysis on the re-illumination map;

[0107] A color matching algorithm is used to align the colors of the original image with the colors of the re-illumination map.

[0108] In one embodiment, if Figure 8 As shown, the feature acquisition module 801 includes:

[0109] The original image feature submodule 8011 is used to process the original image using a variational autoencoder to obtain the original image features;

[0110] The illumination map feature submodule 8012 is used to process the illumination map using a variational autoencoder to obtain illumination map features.

[0111] In one implementation, the feature processing module 802 is further configured to concatenate the original image feature and the illumination image feature in a channel dimension to obtain the first feature.

[0112] In one implementation, the diffusion model is obtained by fine-tuning an initial diffusion model based on a training set, where the training set includes a plurality of training samples, and one training sample includes an original image sample, an illumination image sample, and a re-illumination image sample.

[0113] In one embodiment, if Figure 8 As shown, the device 800 also includes:

[0114] The illumination map generation module 804 is used to convert the illumination parameters into illumination images to generate illumination maps. The specific functions and examples of each module and submodule of the device of the present disclosure embodiment can be found in the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0115] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0117] Fig. 9A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0118] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0119] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the relighting method. For example, in some embodiments, the relighting method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the relighting method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the relighting method in any other appropriate manner (e.g., by means of firmware).

[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0126] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0127] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A re-illumination method, comprising: Based on the original image and the illumination map, the original image feature and the illumination map feature are obtained respectively; Obtaining a first feature based on the original image feature and the illumination image feature; The first feature is processed using a diffusion model to obtain a re-illumination map.

2. The method according to claim 1, wherein: The first feature is processed using a diffusion model to obtain a re-illumination map, including: Extracting style structure features from the original image; Using a first diffusion model to add noise and denoise the first feature and the style structure feature to obtain a second feature; Based on the second feature, a re-illumination map is obtained.

3. The method according to claim 2, wherein: Based on the second feature, a re-illumination map is obtained, including: Extracting color features from the original image; Using a second diffusion model, adding noise and denoising the second feature and the color feature to obtain a third feature; The third feature is post-processed to obtain the re-illumination map.

4. The method according to claim 3, wherein: Post-processing the third feature to obtain the re-illumination map includes: The third feature is decoded using a decoder of the variational autoencoder to generate a re-illumination map.

5. The method according to claim 3 or 4, wherein: Post-processing the third feature to obtain the re-illumination map further includes at least one of the following: performing image frequency analysis on the re-illumination map; A color matching algorithm is used to align the color of the original image with the color of the re-illumination image.

6. The method according to any one of claims 1 to 5, wherein: Based on the original image and the illumination map, the original image features and the illumination map features are obtained respectively, including: Processing the original image using a variational autoencoder to obtain features of the original image; The illumination map is processed using a variational autoencoder to obtain illumination map features.

7. The method according to any one of claims 1 to 6, wherein: Obtaining a first feature based on the original image feature and the illumination image feature includes: The original image feature and the illumination image feature are concatenated in the channel dimension to obtain the first feature.

8. The method according to any one of claims 1 to 7, wherein: The diffusion model is obtained by fine-tuning an initial diffusion model based on a training set, wherein the training set includes a plurality of training samples, and one training sample includes an original image sample, an illumination image sample, and a re-illumination image sample.

9. The method according to any one of claims 1 to 8, further comprising: Perform illumination image conversion on the illumination parameters to generate an illumination map.

10. A heavy illumination device, comprising: A feature acquisition module, used to obtain original image features and illumination map features based on the original image and the illumination map respectively; A feature processing module, used for obtaining a first feature based on the original image feature and the illumination image feature; The re-illumination module is used to process the first feature using a diffusion model to obtain a re-illumination map.

11. The device according to claim 10, wherein: The re-illumination module comprises: A style submodule, used to extract style structure features from the original image; A first diffusion submodule, configured to use a first diffusion model to perform noise addition and noise removal on the first feature and the style structure feature to obtain a second feature; The re-lighting module is further used to obtain a re-lighting map based on the second feature.

12. The device according to claim 11, wherein The re-illumination module further includes: A color submodule, used to extract color features from the original image; A second diffusion submodule, configured to add noise and denoise the second feature and the color feature using a second diffusion model to obtain a third feature; A post-processing submodule is used to post-process the third feature to obtain the re-illumination map.

13. The device according to claim 12, wherein: The post-processing submodule is further used to decode the third feature using a decoder of the variational autoencoder to generate a re-illumination map.

14. The device according to claim 12 or 13, wherein: The post-processing submodule is further used for at least one of the following: performing image frequency analysis on the re-illumination map; A color matching algorithm is used to align the color of the original image with the color of the re-illumination image.

15. The device according to any one of claims 10 to 14, wherein: The feature acquisition module comprises: An original image feature submodule, used to process the original image using a variational autoencoder to obtain the original image features; The illumination map feature submodule is used to process the illumination map using a variational autoencoder to obtain illumination map features.

16. The device according to any one of claims 10 to 15, wherein: The feature processing module is further used to splice the original image feature and the illumination image feature in the channel dimension to obtain the first feature.

17. The device according to any one of claims 10 to 16, wherein: The diffusion model is obtained by fine-tuning an initial diffusion model based on a training set, wherein the training set includes a plurality of training samples, and one training sample includes an original image sample, an illumination image sample, and a re-illumination image sample.

18. The device according to any one of claims 10 to 17, further comprising: The illumination map generation module is used to convert illumination parameters into illumination images to generate illumination maps.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images