Image relighting method and system based on asymmetric frequency space modulation

Abstract

The application relates to the field of image processing, and provides an image relighting method and system based on asymmetric frequency space modulation, which comprises the following steps: performing encoder decoupling to obtain spatial intrinsic features and global illumination embedding vectors; performing group normalization and convolution operation to obtain intermediate features, performing linear projection and spatial dimension broadcast to obtain spatial broadcast illumination features, and obtaining a spatial modulation map according to the spatial broadcast illumination features and the intermediate features; performing normalization and feature extraction activation on the intermediate features to obtain dynamic weights, so as to obtain low-frequency components and high-frequency components; obtaining modulated low-frequency components by using the spatial modulation map and the low-frequency components, performing high-frequency correction on the high-frequency components based on the spatial modulation map to obtain modulated high-frequency components, and using the modulated low-frequency components and the modulated high-frequency components to perform relighting on an original image to obtain an output image; and training model parameters and performing image output to obtain a target output image. The application is used for completing image relighting.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image relighting method and system based on asymmetric frequency spatial modulation. Background Technology

[0002] Image relighting aims to alter the illumination conditions of a source image to match the target illumination while strictly preserving the original intrinsic geometry and texture structure. While current generative models have made significant progress in this area, several insurmountable shortcomings remain in feature processing and modulation. First, existing techniques generally suffer from frequency-domain coupling between illumination and texture features, directly leading to detail degradation in the relighting results. Physical optics theory indicates that global illumination typically exhibits smooth low-frequency variations, while the intrinsic reflectivity and texture of an object manifest as sharp high-frequency details. However, most existing relighting networks perform frequency-agnostic feature modulation within a unified feature space, incorrectly entangled these two types of features with distinctly different physical properties. This mechanism makes the network highly susceptible to interfering with high-frequency signals when performing ray redirection tasks, inadvertently erasing or blurring the original high-frequency intrinsic texture.

[0003] Regarding lighting control mechanisms, existing technologies, which rely on spatially uniform global affine transformations, cannot effectively handle the generation of complex shadows and highlights. Current methods typically extend the extracted global lighting feature vectors directly into a spatially uniform mapping, and then apply a globally uniform affine transformation to the feature map. This approach severely violates the highly non-uniform nature of light transmission in the real physical world. For example, when faced with complex scenes such as strong local self-shadows or specular highlights, spatially uniform modulation weights cannot apply differentiated lighting effects to the geometry of different regions. This makes it difficult for the model to generate physically consistent and realistic lighting effects, and may even produce extremely unnatural structural distortions at shadow edges.

[0004] At the model optimization and training level, existing solutions rely excessively on pixel-level supervision, leading to significant spectral aliasing. Current mainstream optimization objectives almost entirely depend on absolute error loss in the spatial domain. Limited by the mathematical properties of such loss functions, when the model faces the prediction of high-frequency shadow edges or complex textures, in order to minimize the global average error, the network tends to output a "safe," extremely blurry, gradient transition pixel. This compromise output strategy severely sacrifices the high-frequency sharpness of the image, resulting in a visually unrealistic final image with obvious spectral aliasing and edge smoothing problems, greatly limiting the model's generalization and application capabilities in complex real-world scenes. Summary of the Invention

[0005] The present invention aims to at least solve one of the technical problems existing in related technologies. To this end, the present invention provides an image relighting method and system based on asymmetric frequency spatial modulation, which realizes high-fidelity image relighting without losing the original texture details of the image.

[0006] This invention provides an image relighting method based on asymmetric frequency spatial modulation, comprising:

[0007] S1: Obtain the original image and reference image, decouple the original image and reference image by encoder, and obtain spatial intrinsic features and global illumination embedding vector;

[0008] S2: Perform group normalization and convolution operations on the spatial intrinsic features to obtain intermediate features. Perform linear projection and spatial dimension broadcasting on the global illumination embedding vector to obtain spatial broadcast illumination features. Obtain the spatial modulation map based on the spatial broadcast illumination features and intermediate features.

[0009] S3: Normalize and extract features from intermediate features to obtain dynamic weights, and obtain low-frequency and high-frequency components through dynamic weights;

[0010] S4: Obtain the modulated low-frequency component through the spatial modulation map and the low-frequency component, perform high-frequency correction on the high-frequency component based on the spatial modulation map to obtain the modulated high-frequency component, and use the modulated low-frequency component and the modulated high-frequency component to re-illuminate the original image to obtain the output image.

[0011] S5: Obtain the image after lighting, calculate the spectral consistency loss between the image after lighting and the output image, obtain the relighting loss through the spectral consistency loss, train the model parameters based on the relighting loss to obtain the target model, and output the target image from the target model to obtain the target output image, thus completing the relighting of the image.

[0012] According to the image relighting method based on asymmetric frequency spatial modulation provided by the present invention, in step S1, a noise scaling factor is obtained by generating random numbers, Gaussian noise interference is obtained, and a noise interference term is obtained by using Gaussian noise interference and noise scaling factor.

[0013] The original image and the noise interference term are decoupled by an encoder to obtain the spatial intrinsic features. The reference image is then decoupled by an encoder to obtain the global illumination embedding vector.

[0014] According to the image relighting method based on asymmetric frequency spatial modulation provided by the present invention, in step S2, the spatial broadcast illumination features and intermediate features are stitched together, and convolution and activation are performed cyclically to obtain the spatial modulation map.

[0015] According to the image relighting method based on asymmetric frequency spatial modulation provided by the present invention, in step S3, the intermediate features are re-normalized to obtain re-normalized features, and the intermediate features are subjected to global context extraction, convolution and batch normalization to obtain the dynamic weights.

[0016] The low-frequency component is obtained by smoothing the renormalized features using dynamic weights, and the high-frequency component is obtained by combining the renormalized features and the low-frequency component.

[0017] According to the present invention, an image relighting method based on asymmetric frequency spatial modulation is provided. In step S4, the low-frequency component is injected into the spatial modulation map to obtain the modulated low-frequency component. A high-frequency correction unit is constructed by multiple convolutional layers and nonlinear activation functions between the convolutional layers. The high-frequency component is corrected by the high-frequency correction unit and the spatial modulation map to obtain the modulated high-frequency component. In the high-frequency correction unit, the terminal convolution is initialized to zero.

[0018] According to the image relighting method based on asymmetric frequency spatial modulation provided by the present invention, in step S4, the modulation low-frequency component and the modulation high-frequency component are added element-wise, and activation, convolution projection and residual connection are performed to obtain output features, a decoder is determined, and the output features are decoded by the decoder to obtain the output image.

[0019] According to the image re-illumination method based on asymmetric frequency spatial modulation provided by the present invention, in step S5, a fast Fourier transform is performed on the illuminated image to obtain the illuminated logarithmic amplitude spectrum, a fast Fourier transform is performed on the output image to obtain the output logarithmic amplitude spectrum, and the L1 norm of the difference between the illuminated logarithmic amplitude spectrum and the output logarithmic amplitude spectrum is calculated to obtain the spectral consistency loss.

[0020] The present invention also provides an image relighting system based on asymmetric frequency spatial modulation, comprising:

[0021] Encoder decoupling module: used to acquire the original image and the reference image, decouple the original image and the reference image from the encoder, and obtain spatial intrinsic features and global illumination embedding vector;

[0022] Spatial Modulation Map and Intermediate Feature Module: This module is used to perform group normalization and convolution operations on spatial intrinsic features to obtain intermediate features, perform linear projection and spatial dimension broadcasting on global illumination embedding vectors to obtain spatial broadcast illumination features, and obtain spatial modulation map based on spatial broadcast illumination features and intermediate features.

[0023] High and low frequency component module: used to normalize intermediate features and extract and activate features to obtain dynamic weights, and obtain low frequency components and high frequency components through dynamic weights;

[0024] Output image module: used to obtain the modulated low-frequency component through the spatial modulation map and the low-frequency component, perform high-frequency correction on the high-frequency component based on the spatial modulation map to obtain the modulated high-frequency component, and use the modulated low-frequency component and the modulated high-frequency component to re-illuminate the original image to obtain the output image;

[0025] Parameter training module: used to acquire the image after lighting, calculate the spectral consistency loss between the image after lighting and the output image, obtain the relighting loss through the spectral consistency loss, train the model parameters based on the relighting loss to obtain the target model, and output the target image through the target model to obtain the target output image, thus completing the image relighting.

[0026] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0027] This invention provides an image relighting method based on asymmetric frequency spatial modulation. It adaptively learns the frequency distribution of features in an image and eliminates interference with intrinsic texture during illumination modulation by modulating low-frequency and high-frequency components, generating a physically consistent relighted image from the source image. Furthermore, dynamic weighting ensures explicit decoupling of low-frequency illumination features from high-frequency intrinsic texture. By injecting a spatial modulation map into the low-frequency path and performing high-frequency correction in the high-frequency path, it strictly preserves high-frequency intrinsic geometric and textural features, achieving better reproduction of complex local shadows and highlights.

[0028] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is a schematic flowchart of an image relighting method based on asymmetric frequency spatial modulation provided by the present invention.

[0031] Figure 2 This is a schematic diagram of the first test result of an image relighting method based on asymmetric frequency spatial modulation provided by the present invention.

[0032] Figure 3 This is a magnified view of the first test result of an image relighting method based on asymmetric frequency spatial modulation provided by the present invention.

[0033] Figure 4 This is a schematic diagram of the second test result of an image relighting method based on asymmetric frequency spatial modulation provided by the present invention.

[0034] Figure 5 This is a magnified view of the second test result of an image relighting method based on asymmetric frequency spatial modulation provided by the present invention.

[0035] Figure 6 This is a schematic diagram of the structure of an image re-illumination system based on asymmetric frequency spatial modulation provided by the present invention.

[0036] Figure label:

[0037] 100. Encoder decoupling module; 200. Spatial modulation map and intermediate feature module; 300. High and low frequency component module; 400. Output image module; 500. Parameter training module. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0039] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second" and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0040] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0041] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0042] The following is combined Figures 1 to 6 Specific embodiments of the present invention are described below. Figure 1 A flowchart illustrating an image relighting method based on asymmetric frequency spatial modulation provided by the present invention includes:

[0043] S1: Obtain the original image and reference image, decouple the original image and reference image by encoder, and obtain spatial intrinsic features and global illumination embedding vector;

[0044] Furthermore, the objective of this stage is to decouple the encoders of the original image and the reference image to obtain spatial intrinsic features and global illumination embedding vectors. Specifically, in step S1, a noise scaling factor is obtained by generating random numbers, Gaussian noise interference is acquired, and a noise interference term is obtained through the Gaussian noise interference and the noise scaling factor.

[0045] The original image and the noise interference term are decoupled by an encoder to obtain the spatial intrinsic features. The reference image is then decoupled by an encoder to obtain the global illumination embedding vector.

[0046] The specific implementation method for the above steps in this embodiment is as follows:

[0047] First, we need to obtain the image waiting to be relit as the original image. It is also necessary to obtain other scenes that have been lit according to the desired lighting conditions after relighting as reference images. Then, a random number n following a natural normal distribution is generated, thus obtaining the noise scaling factor. :

[0048]

[0049] Here, exp() is the exponential function. Then, the Gaussian noise interference caused by Gaussian noise is obtained. Thus, multiplying the Gaussian noise interference by the noise scaling factor yields the noise interference term.

[0050] Subsequently, the original image and the noise interference were decoupled using an encoder to obtain spatial intrinsic features. :

[0051]

[0052] Where E() represents the global illumination embedding vector obtained by decoupling the reference image through the encoder. :

[0053] E( ).

[0054] S2: Perform group normalization and convolution operations on the spatial intrinsic features to obtain intermediate features. Perform linear projection and spatial dimension broadcasting on the global illumination embedding vector to obtain spatial broadcast illumination features. Obtain the spatial modulation map based on the spatial broadcast illumination features and intermediate features.

[0055] Furthermore, the objective of this stage is to obtain intermediate features, perform linear projection and spatial dimension broadcasting, and finally obtain a spatial modulation map. Specifically, in step S2, the spatially broadcast illumination features and intermediate features are concatenated, and convolution and activation are performed iteratively to obtain the spatial modulation map.

[0056] The specific implementation method for the above steps in this embodiment is as follows:

[0057] First, the spatial intrinsic features need to be input into the frequency domain sensing spatial modulation unit. In the frequency domain sensing spatial modulation unit, the spatial intrinsic features undergo group normalization and convolution operations to obtain intermediate features. A linear projection is performed on the one-dimensional global illumination embedding vector, and then broadcast in the spatial dimension to form spatially broadcast illumination features that match the intermediate feature dimension. .

[0058] Subsequently, the spatial broadcast illumination features and intermediate features are concatenated in dimensionality, followed by two rounds of convolution and activation to obtain the spatial modulation map. :

[0059]

[0060] in, The parentheses indicate that the elements are concatenated. Conv() represents convolution, which is a 3×3 convolution in this embodiment. SiLU() represents the sigmoid colon linear unit activation function. tanh() represents the hyperbolic tangent activation function, which truncates the values ​​to ensure numerical stability. α is a scalar scaling factor.

[0061] S3: Normalize and extract features from intermediate features to obtain dynamic weights, and obtain low-frequency and high-frequency components through dynamic weights;

[0062] Furthermore, the objective of this stage is to obtain dynamic weights, thereby obtaining low-frequency and high-frequency components. Specifically, in step S3, the intermediate features are re-normalized to obtain re-normalized features, and the intermediate features are subjected to global context extraction, convolution, and batch normalization to obtain the dynamic weights;

[0063] The low-frequency component is obtained by smoothing the renormalized features using dynamic weights, and the high-frequency component is obtained by combining the renormalized features and the low-frequency component.

[0064] The specific implementation method for the above steps in this embodiment is as follows:

[0065] By renormalizing the intermediate features, we can obtain the renormalized features. Next, the global context of the features is extracted using the global average pooling function, and then passed through 1×1 convolution, batch normalization, and normalized exponential activation in sequence. This allows us to obtain dynamic weights based on the input content. :

[0066]

[0067] Where GAP() is the global average pooling function. () is a 1×1 convolution function, BN() is a batch normalization operation, and softmax() is a softmax transform function used to normalize exponential activation.

[0068] Thus, since illumination in an image typically changes slowly and is a low-frequency signal, while texture, edges, and object contours are high-frequency signals, using dynamic weights to smooth the renormalized features and extract the trend of light and dark transitions can separate the low-frequency components representing the illumination distribution. :

[0069]

[0070] Here, Dconv() is a depthwise convolution operation based on dynamic weights, which allows for the further extraction of high-frequency components. :

[0071] -

[0072] As mentioned above, high-frequency components represent the texture, edges, and outlines of objects in an image.

[0073] S4: Obtain the modulated low-frequency component through the spatial modulation map and the low-frequency component, perform high-frequency correction on the high-frequency component based on the spatial modulation map to obtain the modulated high-frequency component, and use the modulated low-frequency component and the modulated high-frequency component to re-illuminate the original image to obtain the output image.

[0074] Furthermore, the objective of this stage is to obtain the modulated low-frequency component and perform high-frequency correction on the high-frequency component, thereby relighting and obtaining the output image. Specifically, in step S4, the low-frequency component is injected into the spatial modulation map with illumination direction to obtain the modulated low-frequency component. A high-frequency correction unit is constructed through multiple convolutional layers and nonlinear activation functions between the convolutional layers. The high-frequency component is then subjected to high-frequency correction using the high-frequency correction unit and the spatial modulation map to obtain the modulated high-frequency component. In the high-frequency correction unit, the terminal convolution is initialized to zero.

[0075] In step S4, the modulation low-frequency component and modulation high-frequency component are added element-wise, and activation, convolution projection and residual connection are performed to obtain output features. The decoder is determined, and the output features are decoded by the decoder to obtain the output image.

[0076] The specific implementation method for the above steps in this embodiment is as follows:

[0077] Since directly applying the spatial modulation map to global features would destroy high-frequency features such as texture, edges, and object contours, an asymmetric strategy is needed. Because illumination changes are primarily manifested in low frequencies, the spatial modulation map is used through a multiplicative residual mechanism to apply only to low-frequency components, achieving directional injection of target illumination and thus obtaining the modulated low-frequency components. :

[0078]

[0079] For high-frequency components, since they represent the intrinsic texture of an object, a high-frequency correction unit is needed to avoid destroying the texture when the lighting changes, while also adapting to reasonable deformations at shadow edges. Here, the high-frequency correction unit can perform lightweight detail correction while adapting to reasonable deformations at shadow edges without destroying the object's intrinsic texture. It is implemented by performing residual correction on the original high-frequency components, rather than completely replacing them. The high-frequency correction unit consists of two convolutional layers and a non-linear activation function between them, with the convolutions at the end of the unit initialized to zero. The first convolutional layer extracts features, and the second convolutional layer generates the correction amount, with its weights initialized to 0 to act as the initial identity mapping. The weights are adjusted during training. As training progresses, the weights can be adjusted to learn when and where correction is needed, avoiding the destruction of the original high-frequency information from the outset.

[0080] The high-frequency components can be corrected using a high-frequency correction unit to obtain the modulated high-frequency components. :

[0081] +H( , ])

[0082] In this unit, H() is a high-frequency correction unit. It adds the modulated low-frequency component and the modulated high-frequency component element by element, and then performs activation, convolution projection, and residual connection to obtain the output features. :

[0083]

[0084] Here, the modulated low-frequency and high-frequency components are activated and convolved, and then residually connected to the spatial intrinsic features. This allows the modulated re-illumination to be projected onto the spatial intrinsic features, achieving re-illumination. Finally, the output features are processed by the decoder network to generate the re-illumination output image. To learn the correct high-frequency signal distribution and completely eliminate spectral aliasing, the decoder extracts the spatial intrinsic features as features for decoding with a 90% probability, and extracts both the spatial intrinsic features and the illumination vector of the reference image as features for decoding with a 10% probability.

[0085] S5: Obtain the image after lighting, calculate the spectral consistency loss between the image after lighting and the output image, obtain the relighting loss through the spectral consistency loss, train the model parameters based on the relighting loss to obtain the target model, and output the target image from the target model to obtain the target output image, thus completing the relighting of the image.

[0086] Furthermore, the objective of this stage is to calculate the spectral consistency loss, thereby obtaining the re-illumination loss, and then training the model parameters to obtain the target output image. Specifically, in step S5, a Fast Fourier Transform is performed on the illuminated image to obtain the illuminated logarithmic amplitude spectrum, a Fast Fourier Transform is performed on the output image to obtain the output logarithmic amplitude spectrum, and the L1 norm of the difference between the illuminated logarithmic amplitude spectrum and the output logarithmic amplitude spectrum is calculated to obtain the spectral consistency loss.

[0087] The specific implementation method for the above steps in this embodiment is as follows:

[0088] Here, we need to obtain the image obtained after the original image has been illuminated by the light source in the reference image, i.e., the illuminated image. Then, we calculate the loss between the illuminated image and the output image. This requires calculating the L1 loss and gradient loss in the spatial domain between the illuminated image and the output image. Furthermore, to learn the correct high-frequency signal distribution and completely eliminate spectral aliasing, we also need to calculate the spectral consistency loss. The calculation method for the spectral consistency loss is as follows:

[0089] The illuminated image is subjected to Fast Fourier Transform and logarithmic operation to obtain the logarithmic amplitude spectrum. The output image is then subjected to Fast Fourier Transform and logarithmic operation to obtain the output logarithmic amplitude spectrum. Subsequently, the L1 norm of the difference between the two logarithmic amplitude spectra is calculated to obtain the spectral consistency loss. :

[0090]

[0091] Where F() represents the Fast Fourier Transform, and ln() represents the logarithmic operation. Indicates the output image. This represents the image after lighting. Operations representing the L1 norm.

[0092] Next, the spectral consistency loss, spatial domain L1 loss, and gradient loss are weighted and summed to obtain the relighting loss. To minimize the relighting loss, the method consisting of steps S1 to S4 is trained multiple times to adjust the parameters in operations such as convolution and normalization until the relighting loss is reduced to a predetermined ideal value or the maximum number of training iterations is reached. This completes the training and yields the target model. The output image of the trained target model is then used as the target output image, thus completing the relighting of the image.

[0093] This invention also verifies the effectiveness of an image relighting method based on asymmetric frequency spatial modulation. Here, the method provided by this invention and other existing methods were relighted on the same publicly available dataset, and the relighting results were evaluated. The experimental results are compared in Table 1:

[0094] Table 1. Comparison of relighting experiment results between the method provided by this invention and other methods.

[0095]

[0096] Among them, Input Img, SA-AE, S3Net, Latent-instrinsics, RGB<->X, and Luminet are all existing methods. The number of parameters is in millions, with upward arrows indicating that higher values ​​are better and downward arrows indicating that lower values ​​are better. It can be seen that this invention achieves optimal relighting results with a relatively moderate number of parameters, thus achieving good relighting effects with relatively small computational resources. A schematic diagram of the relighting effect of this method on images from a public dataset is also shown here. Figure 2 This is a schematic diagram of the first test results. Figure 4 This is a schematic diagram of the second test results. Latent-instrinsics, Luminet, and Ground truth are all existing image relighting methods. Figure 2 This is a schematic diagram of the overall first test results. Figure 3 This is a magnified view of the area within the red box in the first test result. Figure 4 This is a schematic diagram of the overall second test results. Figure 5 This is a magnified view of the area within the red box in the second test results.

[0097] The following describes an image relighting device based on asymmetric frequency spatial modulation provided by the present invention. The image relighting device based on asymmetric frequency spatial modulation described below and the image relighting method based on asymmetric frequency spatial modulation described above can be referred to and correspond to each other.

[0098] Figure 6 An example is a schematic diagram of an image re-illumination system based on asymmetric frequency spatial modulation, as shown below. Figure 6 As shown, an image relighting method based on asymmetric frequency spatial modulation, as described above, includes:

[0099] Encoder decoupling module 100: used to acquire the original image and the reference image, decouple the original image and the reference image from the encoder, and obtain spatial intrinsic features and global illumination embedding vector;

[0100] Spatial Modulation Map and Intermediate Feature Module 200: This module is used to perform group normalization and convolution operations on spatial intrinsic features to obtain intermediate features, perform linear projection and spatial dimension broadcasting on global illumination embedding vectors to obtain spatial broadcast illumination features, and obtain spatial modulation map based on spatial broadcast illumination features and intermediate features.

[0101] High and low frequency component module 300: used to normalize intermediate features and extract and activate features to obtain dynamic weights, and obtain low frequency components and high frequency components through dynamic weights;

[0102] Output image module 400: used to obtain a modulated low-frequency component through a spatial modulation map and a low-frequency component, perform high-frequency correction on the high-frequency component based on the spatial modulation map to obtain a modulated high-frequency component, and use the modulated low-frequency component and the modulated high-frequency component to re-illuminate the original image to obtain an output image;

[0103] Parameter training module 500: used to acquire the image after lighting, calculate the spectral consistency loss between the image after lighting and the output image, obtain the relighting loss through the spectral consistency loss, train the model parameters based on the relighting loss to obtain the target model, and output the target image through the target model to obtain the target output image, thus completing the relighting of the image.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image relighting method based on asymmetric frequency spatial modulation, characterized in that, include: S1: Obtain the original image and reference image, decouple the original image and reference image by encoder, and obtain spatial intrinsic features and global illumination embedding vector; S2: Perform group normalization and convolution operations on the spatial intrinsic features to obtain intermediate features. Perform linear projection and spatial dimension broadcasting on the global illumination embedding vector to obtain spatial broadcast illumination features. Obtain the spatial modulation map based on the spatial broadcast illumination features and intermediate features. S3: Normalize and extract features from intermediate features to obtain dynamic weights, and obtain low-frequency and high-frequency components through dynamic weights; S4: Obtain the modulated low-frequency component through the spatial modulation map and the low-frequency component, perform high-frequency correction on the high-frequency component based on the spatial modulation map to obtain the modulated high-frequency component, and use the modulated low-frequency component and the modulated high-frequency component to re-illuminate the original image to obtain the output image. S5: Obtain the image after lighting, calculate the spectral consistency loss between the image after lighting and the output image, obtain the relighting loss through the spectral consistency loss, train the model parameters based on the relighting loss to obtain the target model, and output the target image from the target model to obtain the target output image, thus completing the relighting of the image.

2. The image relighting method based on asymmetric frequency spatial modulation according to claim 1, wherein, In step S1, a noise scaling factor is obtained by generating random numbers, Gaussian noise interference is obtained, and a noise interference term is obtained by using Gaussian noise interference and noise scaling factor. The original image and the noise interference term are decoupled by an encoder to obtain the spatial intrinsic features. The reference image is then decoupled by an encoder to obtain the global illumination embedding vector.

3. The method of claim 1, wherein the asymmetric frequency space modulation is based on a function of the form: ###0001### where f is the frequency, and a is a constant. In step S2, the spatial broadcast illumination features and intermediate features are spliced ​​together, and convolution and activation are performed cyclically to obtain the spatial modulation map.

4. The image relighting method based on asymmetric frequency space modulation according to claim 1, wherein, In step S3, the intermediate features are re-normalized to obtain re-normalized features, and the intermediate features are subjected to global context extraction, convolution and batch normalization to obtain the dynamic weights. The low-frequency component is obtained by smoothing the renormalized features using dynamic weights, and the high-frequency component is obtained by combining the renormalized features and the low-frequency component.

5. The method of claim 1, wherein the asymmetric frequency space modulation is based on a function of the form: ###0001### where f is the frequency, and a is a constant. In step S4, the low-frequency component is injected with light in a directional manner into the spatial modulation map to obtain the modulated low-frequency component. A high-frequency correction unit is formed by multiple convolutional layers and nonlinear activation functions between the convolutional layers. The high-frequency component is corrected by the high-frequency correction unit and the spatial modulation map to obtain the modulated high-frequency component. In the high-frequency correction unit, the terminal convolution is initialized to zero.

6. The image relighting method based on asymmetric frequency spatial modulation according to claim 1, wherein, In step S4, the modulation low-frequency component and modulation high-frequency component are added element-wise, and activation, convolution projection and residual connection are performed to obtain output features. The decoder is determined, and the output features are decoded by the decoder to obtain the output image.

7. The image relighting method based on asymmetric frequency spatial modulation according to claim 1, characterized in that, In step S5, a fast Fourier transform is performed on the illuminated image to obtain the illuminated logarithmic amplitude spectrum. A fast Fourier transform is performed on the output image to obtain the output logarithmic amplitude spectrum. The L1 norm of the difference between the illuminated logarithmic amplitude spectrum and the output logarithmic amplitude spectrum is calculated to obtain the spectral consistency loss.

8. An image relighting system based on asymmetric frequency spatial modulation, used to perform an image relighting method based on asymmetric frequency spatial modulation as described in any one of claims 1 to 7, characterized in that, include: Encoder decoupling module: used to acquire the original image and the reference image, decouple the original image and the reference image from the encoder, and obtain spatial intrinsic features and global illumination embedding vector; Spatial Modulation Map and Intermediate Feature Module: This module is used to perform group normalization and convolution operations on spatial intrinsic features to obtain intermediate features, perform linear projection and spatial dimension broadcasting on global illumination embedding vectors to obtain spatial broadcast illumination features, and obtain spatial modulation map based on spatial broadcast illumination features and intermediate features. High and low frequency component module: used to normalize intermediate features and extract and activate features to obtain dynamic weights, and obtain low frequency components and high frequency components through dynamic weights; Output image module: used to obtain the modulated low-frequency component through the spatial modulation map and the low-frequency component, perform high-frequency correction on the high-frequency component based on the spatial modulation map to obtain the modulated high-frequency component, and use the modulated low-frequency component and the modulated high-frequency component to re-illuminate the original image to obtain the output image; Parameter training module: used to acquire the image after lighting, calculate the spectral consistency loss between the image after lighting and the output image, obtain the relighting loss through the spectral consistency loss, train the model parameters based on the relighting loss to obtain the target model, and output the target image through the target model to obtain the target output image, thus completing the image relighting.

Images