Lighting estimation method, terminal and storage medium

Abstract

This application discloses an illumination estimation method, a terminal, and a storage medium. The terminal trains an initial autoencoder model and an initial decoding model using first training data to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector. The initial autoencoder model and the initial decoding model are optimized using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model. An initial prediction model is trained using second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model. Illumination estimation is performed on the input image based on the target prediction model and the target decoding model to obtain a sky-light panoramic image corresponding to the input image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an illumination estimation method, a terminal and a storage medium. Background Art

[0002] The lighting information of outdoor scenes can vary greatly depending on the weather or time of day, for example, bright highlights and dark shadows at noon and warm orange tones at dusk. At present, the methods for realizing lighting estimation of outdoor images mainly include methods with the help of auxiliary markers, methods with the help of auxiliary shooting equipment, and image analysis methods without auxiliary markers or shooting equipment. However, these outdoor lighting prediction methods using parameterized lighting models have many limitations, such as only being able to handle sunny days, or the outdoor lighting parameters they output cannot cover all sky light forms, resulting in poor lighting estimation results. Summary of the invention

[0003] The embodiments of the present application provide a lighting estimation method, a terminal, and a storage medium that can process the sky light morphology of outdoor scenes in any weather or time period and have better lighting estimation capabilities.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for estimating illumination, the method comprising:

[0006] Using the first training data to train an initial autoencoder model and an initial decoding model, obtaining a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector;

[0007] Optimizing the initial autoencoder model and the initial decoding model using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model;

[0008] Training an initial prediction model using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model;

[0009] The input image is subjected to illumination estimation processing according to the target prediction model and the target decoding model to obtain a sky light panorama corresponding to the input image.

[0010] In a second aspect, an embodiment of the present application provides a terminal, wherein the terminal includes a training unit, a processing unit, and an estimation unit.

[0011] The training unit is used to train an initial autoencoder model and an initial decoding model using first training data to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; and train an initial prediction model using second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model; wherein the target prediction model is used to obtain the latent vector corresponding to the input image

[0012] The processing unit is used to optimize the initial autoencoder model and the initial decoding model by using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model;

[0013] The estimation unit is used to perform illumination estimation processing on the input image according to the target prediction model and the target decoding model to obtain a sky light panorama corresponding to the input image.

[0014] In a third aspect, an embodiment of the present application provides a terminal, which further includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the illumination estimation method as described above is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which is applied to a terminal. When the program is executed by a processor, the illumination estimation method as described above is implemented.

[0016] An embodiment of the present application provides a method for illumination estimation, a terminal, and a storage medium. The terminal uses first training data to train an initial autoencoder model and an initial decoding model to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; uses the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to optimize the initial autoencoder model and the initial decoding model to obtain a target autoencoder model and a target decoding model; uses the second training data, the first latent vector, the second latent vector, and the first panoramic image to train an initial prediction model to obtain a target prediction model; performs illumination estimation processing on an input image according to the target prediction model and the target decoding model to obtain a sky light panoramic image corresponding to the input image. It can be seen that the present application optimizes the initial autoencoder model and the initial decoding model by training the first latent vector, the second latent vector, the first panoramic image and the second panoramic image obtained by the initial autoencoder model and the initial decoding model to obtain the target autoencoder model and the target decoding model, and then uses the first latent vector, the second latent vector and the first panoramic image in the above training process to train the initial prediction model to obtain the target prediction model. Finally, the latent vector corresponding to the input image can be obtained through the constructed target prediction model, and then the latent vector is decoded by the target decoding model to obtain the sky light panorama corresponding to the latent vector. Since the sky light panorama can display all the lighting information in the input image, it can produce a better lighting estimation effect. Therefore, the lighting estimation method based on the target prediction model and the target decoding model proposed in the present application has better lighting estimation capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 1 ;

[0018] Figure 2 Schematic diagram of the implementation of the illumination estimation method proposed in the embodiment of the present application Figure 1 ;

[0019] Figure 3 A schematic diagram of color distortion parameters proposed in an embodiment of the present application;

[0020] Figure 4 Schematic diagram of the application of the illumination estimation method proposed in the embodiment of the present application Figure 1 ;

[0021] Figure 5 Schematic diagram of the application of the illumination estimation method proposed in the embodiment of the present application Figure 2 ;

[0022] Figure 6 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 2 ;

[0023] Figure 7 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 3 ;

[0024] Figure 8 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 4 ;

[0025] Fig. 9 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 5 ;

[0026] Fig.10 Schematic diagram of the implementation of the illumination estimation method proposed in the embodiment of the present application Figure 2 ;

[0027] Fig.11 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 6 ;

[0028] Fig.12 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 7 ;

[0029] Fig.13 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 8 ;

[0030] Fig.14 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 9 ;

[0031] Fig.15 The structure of the terminal proposed in the embodiment of the present application is shown in FIG. Figure 1 ;

[0032] Fig.16 The structure of the terminal proposed in the embodiment of the present application is shown in FIG. Figure 2 . DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for ease of description, only the parts related to the related applications are shown in the drawings.

[0034] Lighting consistency means that virtual objects have the same lighting effects as real scenes. The goal of lighting consistency is to make the lighting conditions of virtual objects consistent with those in real scenes, so that virtual objects have consistent light and shadow effects with real objects to enhance the realism of virtual objects. However, the key to achieving lighting consistency is how to estimate the lighting information of real scenes.

[0035] At present, the methods for realizing the illumination information of the scene include the method with the help of auxiliary markers, the method with the help of auxiliary shooting equipment and the image analysis method. Among them, the method with the help of auxiliary markers refers to some special objects with known geometric structures in the scene, which are used to capture or infer the illumination information in the real scene; the method with the help of auxiliary shooting equipment refers to the use of special shooting equipment such as depth cameras, fisheye cameras, light field cameras, etc. to more quickly calculate the illumination conditions in the scene; the image analysis method refers to the use of machine learning, image processing and other technologies to analyze the illumination information of the real scene. In the image analysis method, when using the deep learning method, an illumination model will be introduced, such as the Hosek-Wilkie illumination model, which is fitted to a large outdoor illumination database, and then a convolutional neural network (CNN) is trained to regress the Hosek-Wilkie parameters from the input image. However, this type of method using parameterized illumination models to realize illumination estimation has many limitations. For example, this type of parameterized illumination model can only handle sunny days, or the illumination parameters output by it cannot cover all sky light forms, and the illumination estimation effect is poor.

[0036] In order to solve this problem, the present application proposes a lighting estimation method, a terminal and a storage medium, wherein the terminal uses first training data to train an initial autoencoding model and an initial decoding model to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; the initial autoencoding model and the initial decoding model are optimized using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoding model and a target decoding model; the initial prediction model is trained using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model; the input image is subjected to lighting estimation processing according to the target prediction model and the target decoding model to obtain a sky light panoramic image corresponding to the input image, thereby being able to process the sky light morphology of outdoor scenes under any weather or time period, and having better lighting estimation capabilities.

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0038] An embodiment of the present application provides a method for estimating illumination. Figure 1 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the illumination estimation method may include the following steps:

[0039] Step 101: Use the first training data to train an initial autoencoder model and an initial decoding model to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector.

[0040] In an embodiment of the present application, first, the terminal uses the first training data to train an initial autoencoder model and an initial decoding model to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector.

[0041] It should be noted that, in an embodiment of the present application, the first training data is a data set used to train an initial autoencoder model and an initial decoding model, and the first training data only includes sky information, but does not include non-sky components, such as buildings, plants and other information.

[0042] Furthermore, in an embodiment of the present application, the first training data is composed of third training data and fourth training data, wherein the third training data is training data obtained by stripping non-sky components in the second training data, and then the first training data is determined based on the third training data and the fourth training data.

[0043] Exemplarily, in an embodiment of the present application, the second training data is 6700 high dynamic range images (HDR) panoramas sampled from the SUN360-HDR dataset, and the fourth training data is 5000 HDR panoramas sampled from the Laval dataset; since the second training data contains non-sky components, such as buildings, plants and other information, these non-sky components in the second training data are stripped and determined as the third training data. Finally, the third training data and the fourth training data are jointly determined as the first training data for training the initial autoencoding model and the initial decoding model.

[0044] It should be noted that, in the embodiments of the present application, the first training data are all HDR panoramas adjusted to a uniform size. For example, the first training data are all HDR panoramas of 32×128 dimensions.

[0045] It should be noted that, in the embodiment of the present application, the structure of the initial autoencoder model may include a convolutional layer and two residual blocks, and the structure of the initial decoding model may also include a convolutional layer and two residual blocks; illustratively, Figure 2 Schematic diagram of the implementation of the illumination estimation method proposed in the embodiment of the present application Figure 1 ,like Figure 2 As shown in the figure, the structure of the initial autoencoder model can be a 64×7 convolutional layer, a 64×16 residual block and a 32×8 residual block; the structure of the initial decoding model can be a 64×16 residual block, a 128×32 residual block and a 3×3 convolutional layer; then a 32×128-dimensional panoramic image is input into the initial autoencoder model to obtain a 64-dimensional latent vector, and then the 64-dimensional latent vector is input into the initial decoding model to restore the 64-dimensional latent vector to a 32×128-dimensional panoramic image.

[0046] Further, in an embodiment of the present application, the autoencoder model can be used to perform feature dimensionality reduction on the HDR panorama, thereby obtaining a latent vector of the HDR panorama, wherein the dimension of the latent vector is lower than that of the HDR panorama. For example, a 32×128-dimensional HDR panorama is input into the initial autoencoder model to obtain a 64-dimensional latent vector.

[0047] It should be noted that in the embodiments of the present application, the dimension of the latent vector is not limited to 64 dimensions, and can be 32 dimensions, 128 dimensions, or other integer power of 2 dimensions.

[0048] Furthermore, in an embodiment of the present application, the decoding model can be used to decode the low-dimensional latent vector to obtain a corresponding HDR panoramic image. For example, by inputting a 64-dimensional latent vector into the initial decoding model, a 32×128-dimensional HDR panoramic image can be obtained.

[0049] It should be noted that, in the embodiment of the present application, the first latent vector refers to a normal latent vector obtained after the first training data is input into the initial autoencoder model; the second latent vector refers to a latent vector output after the first training data that generates color distortion is input into the initial autoencoder model. Figure 3 This is a schematic diagram of the color distortion parameters proposed in the embodiment of the present application, such as Figure 3 The diagram shows the types of color distortion and parameter settings, where the types of color distortion include exposure, white balance, and gamma.

[0050] Furthermore, in an embodiment of the present application, the first panoramic image refers to a normal panoramic image obtained by decoding the initial decoding model after the first latent vector is input into the initial decoding model, and the second panoramic image refers to a color distorted panoramic image obtained by decoding the second latent vector is input into the initial decoding model.

[0051] Step 102: Optimize the initial autoencoder model and the initial decoding model using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model;

[0052] In an embodiment of the present application, after the terminal uses the first training data to train the initial autoencoding model and the initial decoding model to obtain the first latent vector and the second latent vector corresponding to the first training data, the first panoramic image corresponding to the first latent vector, and the second panoramic image corresponding to the second latent vector, the terminal can use the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to optimize the initial autoencoding model and the initial decoding model to obtain the target autoencoding model and the target decoding model.

[0053] It can be understood that, in the embodiments of the present application, the target decoding model obtained after optimization can be used to decode the latent vector, thereby obtaining a panoramic image corresponding to the latent vector.

[0054] It should be noted that in the embodiments of the present application, the corresponding loss function is calculated by the first latent vector, the second latent vector, the first panoramic image and the second panoramic image, so that the initial autoencoder model and the initial decoding model are optimized using the loss function, and finally the target autoencoder model and the target decoding model are obtained.

[0055] Specifically, in an embodiment of the present application, a first norm loss function is first calculated using the first latent vector and the second latent vector, and then the second norm loss function is calculated using the first panoramic image, the second panoramic image and the first training data. Finally, the first loss function is calculated according to the first norm loss function and the second norm loss function, and the initial autoencoder model and the initial decoding model are optimized using the first loss function to obtain a target autoencoder model and a target decoding model.

[0056] Step 103: Train an initial prediction model using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model.

[0057] In an embodiment of the present application, after the terminal uses the first training data to train an initial autoencoder model and an initial decoding model to obtain the first latent vector and the second latent vector corresponding to the first training data, the first panoramic image corresponding to the first latent vector, and the second panoramic image corresponding to the second latent vector, the terminal can use the second training data, the first latent vector, the second latent vector, and the first panoramic image to train an initial prediction model to obtain a target prediction model.

[0058] It can be understood that in an embodiment of the present application, the target prediction model obtained after training can be used to obtain the latent vector corresponding to the input image, and the latent vector contains the sky light feature information of the input image, so that the latent vector can be input into the target decoding model to obtain a panoramic image corresponding to the latent vector.

[0059] Furthermore, in an embodiment of the present application, the prediction model can be constructed based on DenseNet-161. The overall architecture is consistent with DenseNet-161, but the last layer is a fully connected layer with 64-bit output; the prediction model can also use other CNN networks, such as EfficientNet, ResNet, MobileNet and multi-scale Transformer.

[0060] It should be noted that, in an embodiment of the present application, the second loss function is calculated using the second training data, the first latent vector, the second latent vector and the first panoramic image, so that the initial prediction model is optimized using the second loss function to obtain the target prediction model.

[0061] Specifically, in an embodiment of the present application, the fifth norm loss function is first calculated using the first latent vector and the prediction vector, and then the sixth norm loss function is calculated using the first panoramic image and the third panoramic image corresponding to the prediction vector. Finally, the second loss function is obtained based on the fifth norm loss function and the sixth norm loss function. The initial prediction model is optimized using the second loss function to obtain the target prediction model.

[0062] Step 104: Perform illumination estimation processing on the input image according to the target prediction model and the target decoding model to obtain a sky light panorama corresponding to the input image.

[0063] In an embodiment of the present application, the terminal optimizes the initial autoencoding model and the initial decoding model using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain the target autoencoding model and the target decoding model, and trains the initial prediction model using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain the target prediction model. After that, the input image can be subjected to illumination estimation processing according to the target prediction model and the target decoding model to obtain a sky light panoramic image corresponding to the input image.

[0064] It should be noted that in the embodiment of the present application, the skylight panorama is an image that only contains information about the sun and light in the sky, that is, the skylight panorama does not contain any ground objects. The skylight panorama can fully describe the state of the skylight.

[0065] It can be understood that in an embodiment of the present application, by inputting the input image into the target prediction model, a latent vector of the input image can be obtained, and the latent vector contains the sky light feature information of the input image. Then, by inputting the latent vector into the target decoding model, a sky light panorama corresponding to the latent vector can be obtained. The sky light panorama can fully display the sky light status in the input image, thereby completing the lighting estimation processing of the input image.

[0066] Furthermore, in an embodiment of the present application, the sky light panorama after illumination estimation processing can also be applied to an augmented reality (AR) scene. For example, a virtual scene and the sky light panorama can be fused so that the virtual scene can reflect the sky light form in the sky light panorama, and the light in the virtual scene is the same as the sky light form in the sky light panorama. Figure 4 Schematic diagram of the application of the illumination estimation method proposed in the embodiment of the present application Figure 1 , Figure 4 Schematic diagrams of an input image, a skylight panorama, and a virtual scene are shown, and the output effects of the target prediction model and the target decoding model proposed in the present application are compared with the output effects of different existing models. The skylight panorama output by the target prediction model and the target decoding model proposed in the present application is more consistent with the illumination information of the input image, and can better reflect the skylight information of the input image; Figure 4 As shown, the input image can be any photo taken of an outdoor environment. After a single photo is input into the target prediction model and the target decoding model, a skylight panorama corresponding to the input image can be obtained. The skylight panorama is an image that only contains the sun and light information in the sky. Further, the obtained skylight panorama can be fused with the virtual scene for drawing, so that in the virtual scene, the shadow form of the virtual object corresponds to the skylight panorama. Furthermore, Figure 5 Schematic diagram of the application of the illumination estimation method proposed in the embodiment of the present application Figure 2 , Figure 5 The picture shown is a picture displayed on a mobile terminal by fusing the sky light panorama and the virtual scene. It can be seen that the lighting estimation method of the present application has a good lighting estimation effect. After being fused with the virtual scene, it can vividly display the ambient lighting status, thereby enhancing the realism of AR drawing.

[0067] It should be noted that, in the embodiments of the present application, the input image may be any picture of an outdoor environment taken by a terminal, and the input image may contain ground object information.

[0068] Further, in the embodiment of the present application, the illumination estimation process refers to inputting the input image into the target prediction model and the target decoding model, thereby obtaining a sky light panorama corresponding to the input image, and realizing light estimation of the input image. That is, the sky light panorama obtained after the light estimation process contains the complete sky light information of the input image.

[0069] Figure 6 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 2 ,like Figure 6 As shown, in an embodiment of the present application, the terminal optimizes the initial autoencoding model and the initial decoding model using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain the target autoencoding model and the target decoding model, that is, step 102 may include the following steps:

[0070] Step 102a, calculating a first loss function corresponding to the initial autoencoder model and the initial decoding model according to the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image.

[0071] In an embodiment of the present application, the terminal optimizes the initial autoencoding model and the initial decoding model using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain the target autoencoding model and the target decoding model. Specifically, the terminal calculates the first loss function corresponding to the initial autoencoding model and the initial decoding model based on the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image.

[0072] It can be understood that, in the embodiment of the present application, the first loss function is used to optimize the initial autoencoding model and the initial decoding model.

[0073] Furthermore, in an embodiment of the present application, a first loss function is calculated based on a first latent vector, a second latent vector, a first panoramic image, and a second panoramic image. Specifically, a first norm loss function is first calculated based on the first latent vector and the second latent vector, and a second norm loss function is calculated based on the first panoramic image, the second panoramic image, and the first training data. Finally, the first loss function is obtained based on the first norm loss function and the second norm loss function.

[0074] Step 102b: Optimize the initial autoencoding model and the initial decoding model according to the first loss function to obtain a target autoencoding model and a target decoding model.

[0075] In an embodiment of the present application, after the terminal calculates the first loss function corresponding to the initial autoencoding model and the initial decoding model based on the first latent vector, the second latent vector, the first panoramic image and the second panoramic image, the initial autoencoding model and the initial decoding model can be optimized according to the first loss function to obtain the target autoencoding model and the target decoding model.

[0076] It can be understood that in the embodiments of the present application, the initial autoencoding model and the initial decoding model are optimized using the first loss function to continuously reduce the error between the initial model and the target model, and finally the target autoencoding model and the target decoding model are obtained through optimization.

[0077] Figure 7 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 3 ,like Figure 7 As shown, in an embodiment of the present application, the terminal calculates a first loss function corresponding to the initial autoencoder model and the initial decoding model according to the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image, that is, step 102a may include the following steps:

[0078] Step 102a1: Calculate the first norm loss function corresponding to the first latent vector and the second latent vector.

[0079] In an embodiment of the present application, the terminal calculates a first loss function corresponding to an initial autoencoding model and an initial decoding model based on the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image. Specifically, the first norm loss function corresponding to the first latent vector and the second latent vector can be calculated.

[0080] It can be understood that, in the embodiments of the present application, the norm loss function refers to the L1 norm loss function, also known as the least absolute deviations (LAD), which minimizes the sum of the absolute differences between the target value and the estimated value.

[0081] Further, in an embodiment of the present application, the first norm loss function refers to an L1 norm loss function calculated according to the first latent vector and the second latent vector;

[0082] Exemplarily, in an embodiment of the present application, the first latent vector is Z, and the second latent vector is Z d , then the first norm loss function is expressed as the following formula:

[0083] L d =||Z d -Z|| (1)

[0084] Among them, L d is the first norm loss function.

[0085] Step 102a2: Calculate the second norm loss function corresponding to the first panoramic image, the second panoramic image, and the first training data.

[0086] In an embodiment of the present application, the terminal calculates a first loss function corresponding to an initial autoencoding model and an initial decoding model based on the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image. Specifically, the second norm loss function corresponding to the first panoramic image, the second panoramic image, and the first training data can be calculated.

[0087] It can be understood that, in the embodiment of the present application, the second norm loss function refers to the L1 norm loss function calculated according to the first panoramic image, the second panoramic image and the first training data.

[0088] Exemplarily, in an embodiment of the present application, the first panoramic image is The second panorama is The first training data is P, then the second norm loss function is expressed as the following formula:

[0089]

[0090] Step 102a3: Calculate the first loss function according to the first norm loss function and the second norm loss function.

[0091] In an embodiment of the present application, after calculating the first norm loss function corresponding to the first latent vector and the second latent vector, and calculating the second norm loss function corresponding to the first panoramic image, the second panoramic image and the first training data, the terminal can calculate the first loss function based on the first norm loss function and the second norm loss function.

[0092] It should be noted that in an embodiment of the present application, the first loss function is obtained by adding the first norm loss function and the second norm loss function. At the same time, weight values ​​can also be added to the first norm loss function and the second norm loss function to obtain the first loss function.

[0093] Exemplarily, in an embodiment of the present application, the first norm loss function is A, the second norm loss function is B, and a weight value λ is added at the same time, then the first loss function L can be expressed as the following formula:

[0094] L=A+λB (3)

[0095] Figure 8 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 4 ,like Figure 8As shown, in an embodiment of the present application, the terminal trains an initial prediction model using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model, that is, step 103 may include the following steps:

[0096] Step 103a: input the second training data into the initial prediction model, and output a prediction vector corresponding to the second training data.

[0097] In an embodiment of the present application, the terminal uses the second training data, the first latent vector, the second latent vector and the first panoramic image to train an initial prediction model to obtain a target prediction model. Specifically, the terminal inputs the second training data into the initial prediction model and outputs a prediction vector corresponding to the second training data.

[0098] It can be understood that, in the embodiment of the present application, the prediction vector is obtained by inputting the second training data into the initial prediction model.

[0099] Step 103b: input the prediction vector into the target decoding model, and output the third panoramic image corresponding to the prediction vector.

[0100] In an embodiment of the present application, after inputting the second training data into the initial prediction model and outputting the prediction vector corresponding to the second training data, the terminal can input the prediction vector into the target decoding model and output the third panoramic image corresponding to the prediction vector.

[0101] It can be understood that, in the embodiment of the present application, after the prediction vector is input into the target decoding model, the target decoding model can decode the prediction vector to obtain a third panoramic image corresponding to the prediction vector.

[0102] Further, in an embodiment of the present application, the third panoramic image is a panoramic image obtained according to the prediction vector, that is, the third panoramic image may include sky light information, and the sky light information included in the third panoramic image is obtained according to the prediction vector and corresponds to the prediction vector.

[0103] Step 103c, calculating a second loss function corresponding to the initial prediction model according to the prediction vector, the first latent vector, the first panoramic image, and the third panoramic image.

[0104] In an embodiment of the present application, after the terminal inputs the prediction vector into the target decoding model and outputs the third panoramic image corresponding to the prediction vector, it can calculate the second loss function corresponding to the initial prediction model based on the prediction vector, the first latent vector, the first panoramic image and the third panoramic image.

[0105] Furthermore, in an embodiment of the present application, the second loss function is used to optimize the initial prediction model, and the calculation of the second loss function requires combining the prediction vector, the first latent vector, the first panoramic image, and the third panoramic image.

[0106] Specifically, in an embodiment of the present application, the fifth norm loss function is first obtained using the predicted vector and the first latent vector, and the sixth norm loss function is obtained using the first panoramic image and the third panoramic image, and finally the second loss function is obtained according to the fifth norm loss function and the sixth norm loss function.

[0107] Furthermore, in an embodiment of the present application, the second loss function is obtained by adding the fifth norm loss function and the sixth norm loss function, and a second weight value can also be added to the fifth norm loss function and the sixth norm loss function, so as to determine the second loss function based on the second weight value, the fifth norm loss function and the sixth norm loss function.

[0108] Exemplarily, in an embodiment of the present application, the fifth norm loss function is D, the sixth norm loss function is E, and a weight value α is added, then the second loss function can be expressed as the following formula:

[0109] L N =D+αE (4)

[0110] Among them, L N is the second loss function.

[0111] Step 103d: Optimize the initial prediction model according to the second loss function to obtain the target prediction model.

[0112] In an embodiment of the present application, after the terminal calculates the second loss function corresponding to the initial prediction model based on the prediction vector, the first latent vector, the first panoramic image and the third panoramic image, the initial prediction model can be optimized according to the second loss function to obtain the target prediction model.

[0113] It can be understood that in the embodiments of the present application, the initial prediction model is optimized by the second loss function, the error between the initial prediction model and the target prediction model is continuously reduced, and finally the target prediction model is obtained by optimization.

[0114] Fig. 9 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 5 ,like Fig. 9 As shown, in an embodiment of the present application, the terminal uses the first training data to train the initial autoencoder model and the initial decoding model, and before obtaining the first latent vector and the second latent vector corresponding to the first training data, the first panoramic image corresponding to the first latent vector, and the second panoramic image corresponding to the second latent vector, that is, before step 101, the following steps are also included:

[0115] Step 105: Strip the non-sky components in the second training data to obtain third training data.

[0116] In an embodiment of the present application, the terminal uses the first training data to train an initial autoencoder model and an initial decoding model, and before obtaining the first latent vector and the second latent vector corresponding to the first training data, the first panoramic image corresponding to the first latent vector, and the second panoramic image corresponding to the second latent vector, the terminal also strips the non-sky components in the second training data to obtain the third training data.

[0117] It should be noted that in the embodiment of the present application, the second training data is sampled from the SUN360-HDR dataset. Specifically, each panoramic image in the dataset is randomly sampled 10 times to generate a sampled image with restricted viewpoints. The elevation angle of the sampled image is limited to [-20°, 20°], and the field of view (FOV) is limited to [60°, 80°].

[0118] Further, in an embodiment of the present application, since the second training data includes non-sky components, the non-sky components in the second training data are stripped to obtain the third training data. Therefore, the third training data is training data that only includes sky components.

[0119] Step 106: Determine first training data according to the third training data and the fourth training data.

[0120] In an embodiment of the present application, after the terminal performs stripping processing on the non-sky components in the second training data to obtain the third training data, the first training data can be determined according to the third training data and the fourth training data.

[0121] It should be noted that, in the embodiment of the present application, the fourth training data is training data sampled from the Laval data set.

[0122] In summary, the trained target prediction network can obtain the illumination information of the input image, and then the target decoder is used to restore the illumination information, thereby obtaining a sky light panorama, which can fully display the sky light information corresponding to the input image; the present application designs the structure of the autoencoder and decoder, and in the training process, the design of the loss function includes a color distortion module, Fig.10 Schematic diagram of the implementation of the illumination estimation method proposed in the embodiment of the present application Figure 2 ,like Fig.10 As shown in the figure, considering the color distortion factor in the training of the autoencoder, that is, designing the loss function in combination with the second latent vector for training, can make the model more robust; then combining the autoencoder and decoder to train the prediction network, and finally using the trained target autoencoder and target prediction network for illumination estimation, it can handle the sky light morphology of outdoor scenes in any weather or time period, and has better illumination estimation capabilities.

[0123] An embodiment of the present application provides a method for illumination estimation, in which a terminal uses first training data to train an initial autoencoder model and an initial decoding model to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; the initial autoencoder model and the initial decoding model are optimized using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model; an initial prediction model is trained using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model; an input image is subjected to illumination estimation processing according to the target prediction model and the target decoding model to obtain a sky light panoramic image corresponding to the input image. It can be seen that the present application optimizes the initial autoencoder model and the initial decoding model by training the first latent vector, the second latent vector, the first panoramic image and the second panoramic image obtained by the initial autoencoder model and the initial decoding model to obtain the target autoencoder model and the target decoding model, and then uses the first latent vector, the second latent vector and the first panoramic image in the above training process to train the initial prediction model to obtain the target prediction model. Finally, the input image can be decoded by the constructed target prediction model to obtain the latent vector corresponding to the input image, and then the latent vector is decoded by the target decoding model to obtain the sky light panorama corresponding to the latent vector. The sky light panorama can display all the lighting information in the input image, thereby producing a better lighting estimation effect. Therefore, the target prediction model and target decoding model proposed in the present application have better lighting estimation capabilities.

[0124] Based on the above embodiment, in another embodiment of the present application, Fig.11 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 6 ,like Fig.11 As shown, in an embodiment of the present application, the terminal may calculate the second norm loss function corresponding to the first panoramic image, the second panoramic image, and the first training data, including the following steps:

[0125] Step 201: Calculate the third norm loss function corresponding to the first panoramic image and the first training data.

[0126] In an embodiment of the present application, the terminal calculates a second norm loss function corresponding to the first panoramic image, the second panoramic image, and the first training data. Specifically, the terminal may calculate a third norm loss function corresponding to the first panoramic image and the first training data.

[0127] It can be understood that, in the embodiment of the present application, the third norm loss function is calculated based on the first panoramic image and the first training data.

[0128] Exemplarily, in an embodiment of the present application, the first panoramic image is The first training data is P, then the third norm loss function is expressed as the following formula:

[0129]

[0130] Among them, L1 is the third norm loss function.

[0131] Step 202: Calculate the fourth norm loss function corresponding to the second panoramic image and the first training data.

[0132] In an embodiment of the present application, the terminal calculates a second norm loss function corresponding to the first panoramic image, the second panoramic image, and the first training data. Specifically, the terminal may calculate a fourth norm loss function corresponding to the second panoramic image and the first training data.

[0133] It can be understood that, in the embodiment of the present application, the fourth norm loss function is calculated based on the second panoramic image and the first training data.

[0134] Exemplarily, in an embodiment of the present application, the second panoramic image is The first training data is P, then the fourth norm loss function is expressed as the following formula:

[0135]

[0136] Among them, L2 is the fourth norm loss function.

[0137] Step 203: Add the third norm loss function and the fourth norm loss function to obtain the second norm loss function.

[0138] In an embodiment of the present application, after calculating the third norm loss function corresponding to the first panoramic image and the first training data, and calculating the fourth norm loss function corresponding to the second panoramic image and the first training data, the terminal can add the third norm loss function and the fourth norm loss function to obtain the second norm loss function.

[0139] Exemplarily, in an embodiment of the present application, the third norm loss function is L1, the fourth norm loss function is L2, and the second norm loss function can be expressed as the following formula:

[0140] L r =L1+L2 (7)

[0141] Among them, L r is the second norm loss function.

[0142] Fig.12 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 7 ,like Fig.12As shown, in an embodiment of the present application, the terminal calculates the first loss function according to the first norm loss function and the second norm loss function, that is, step 203 may include the following steps:

[0143] Step 203a: Add a first weight value to the first norm loss function and the second norm loss function.

[0144] In an embodiment of the present application, the terminal calculates the first loss function according to the first norm loss function and the second norm loss function. Specifically, the terminal may add a first weight value to the first norm loss function and the second norm loss function.

[0145] It should be noted that, in the embodiment of the present application, the first weight value includes a weight value corresponding to the first norm loss function and a weight value corresponding to the second norm loss function. The value of the first weight value can be any natural number, and the weight value corresponding to the first norm loss function and the weight value corresponding to the second norm loss function can be the same or different.

[0146] Exemplarily, in an embodiment of the present application, the first norm loss function is A, the second norm loss function is B, and the first weight values ​​added to the first norm loss function A and the second norm loss function B are λ and 1, respectively, that is, the weight value corresponding to the first norm loss function is λ, and the weight value corresponding to the second norm loss function is 1.

[0147] Step 203b: Generate a first loss function according to the first weight value, the first norm loss function and the second norm loss function.

[0148] In an embodiment of the present application, after adding the first weight value to the first norm loss function and the second norm loss function, the terminal can generate the first loss function according to the first weight value, the first norm loss function and the second norm loss function.

[0149] Exemplarily, in an embodiment of the present application, assuming that the first weight values ​​corresponding to the first norm loss function A and the second norm loss function B are λ and 1 respectively, the terminal can calculate the first loss function L by the following formula:

[0150] L=λA+B (8)

[0151] That is, the first loss function L can be generated according to the first weight value λ and 1, the first norm loss function A and the second norm loss function B.

[0152] Fig.13 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 8 ,like Fig.13As shown, in an embodiment of the present application, the terminal may calculate the second loss function corresponding to the initial prediction model according to the prediction vector, the first latent vector, the first panoramic image, and the third panoramic image, including the following steps:

[0153] Step 301: Calculate the fifth norm loss function corresponding to the prediction vector and the first latent vector.

[0154] In an embodiment of the present application, the terminal calculates the second loss function corresponding to the initial prediction model based on the prediction vector, the first latent vector, the first panoramic image and the third panoramic image. Specifically, the fifth norm loss function corresponding to the prediction vector and the first latent vector can be calculated.

[0155] It can be understood that, in the embodiment of the present application, the fifth norm loss function is calculated based on the prediction vector and the first latent vector.

[0156] Exemplarily, in an embodiment of the present application, the prediction vector is The first latent vector is Z, then the fifth norm loss function can be expressed as the following formula:

[0157]

[0158] Among them, L z is the fifth norm loss function.

[0159] Step 302: Calculate the sixth norm loss function corresponding to the first panoramic image and the third panoramic image.

[0160] In an embodiment of the present application, the terminal calculates the second loss function corresponding to the initial prediction model based on the prediction vector, the first latent vector, the first panoramic image and the third panoramic image. Specifically, the sixth norm loss function corresponding to the first panoramic image and the third panoramic image can be calculated.

[0161] It can be understood that, in the embodiment of the present application, the sixth norm loss function is calculated based on the first panoramic image and the third panoramic image.

[0162] Exemplarily, in the embodiment of the present application, the first panoramic image is dec(Z), and the third panoramic image is Then the sixth norm loss function can be expressed as the following formula;

[0163]

[0164] Among them, L c is the sixth norm loss function, and dΩ represents the stereo angle of each pixel in the panorama.

[0165] Step 303: Calculate a second loss function according to the fifth norm loss function and the sixth norm loss function.

[0166] In an embodiment of the present application, after calculating the fifth norm loss function corresponding to the prediction vector and the first latent vector, and calculating the sixth norm loss function corresponding to the first panoramic image and the third panoramic image, the terminal can calculate the second loss function based on the fifth norm loss function and the sixth norm loss function.

[0167] It should be noted that in an embodiment of the present application, the second loss function is obtained based on the fifth norm loss function and the sixth norm loss function. Specifically, the fifth norm loss function and the sixth norm loss function are added to obtain the second loss function. At the same time, a second weight value can be added to the fifth norm loss function and the sixth norm loss function.

[0168] Fig.14 Schematic diagram of the implementation process of the illumination estimation method proposed in the embodiment of the present application Figure 9 ,like Fig.14 As shown, in an embodiment of the present application, the terminal calculates the second loss function according to the fifth norm loss function and the sixth norm loss function, that is, step 303 may include the following steps:

[0169] Step 303a: Add a second weight value to the fifth norm loss function and the sixth norm loss function.

[0170] In an embodiment of the present application, the terminal calculates the second loss function based on the fifth norm loss function and the sixth norm loss function. Specifically, the second weight value can be first added to the fifth norm loss function and the sixth norm loss function.

[0171] It should be noted that, in the embodiment of the present application, the second weight value includes the weight value corresponding to the fifth norm loss function and the weight value corresponding to the sixth norm loss function. The value of the second weight value can be any natural number, and the weight value corresponding to the fifth norm loss function and the weight value corresponding to the sixth norm loss function can be the same or different.

[0172] Exemplarily, in an embodiment of the present application, the fifth norm loss function is L z , the sixth norm loss function is L c , for the fifth norm loss function L z And the sixth norm loss function L c The second weight values ​​added are 1 and α respectively, that is, the weight value corresponding to the fifth norm loss function is 1, and the weight value corresponding to the sixth norm loss function is α.

[0173] Step 303b: Generate a second loss function according to the second weight value, the fifth norm loss function and the sixth norm loss function.

[0174] In an embodiment of the present application, after adding the second weight value to the fifth norm loss function and the sixth norm loss function, the terminal can generate a second loss function according to the second weight value, the fifth norm loss function and the sixth norm loss function.

[0175] Exemplarily, in an embodiment of the present application, it is assumed that the fifth norm loss function L z And the sixth norm loss function L c The corresponding first weight values ​​are 1 and α respectively, then the terminal can calculate the second loss function L by the following formula N :

[0176] L N =L z +αL c (11)

[0177] An embodiment of the present application provides a method for illumination estimation, in which a terminal uses first training data to train an initial autoencoder model and an initial decoding model to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; the initial autoencoder model and the initial decoding model are optimized using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model; an initial prediction model is trained using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model; an input image is subjected to illumination estimation processing according to the target prediction model and the target decoding model to obtain a sky light panoramic image corresponding to the input image. It can be seen that the present application optimizes the initial autoencoder model and the initial decoding model by training the first latent vector, the second latent vector, the first panoramic image and the second panoramic image obtained by the initial autoencoder model and the initial decoding model to obtain the target autoencoder model and the target decoding model, and then uses the first latent vector, the second latent vector and the first panoramic image in the above training process to train the initial prediction model to obtain the target prediction model. Finally, the input image can be decoded by the constructed target prediction model to obtain the latent vector corresponding to the input image, and then the latent vector is decoded by the target decoding model to obtain the sky light panorama corresponding to the latent vector. The sky light panorama can display all the lighting information in the input image, thereby producing a better lighting estimation effect. Therefore, the target prediction model and target decoding model proposed in the present application have better lighting estimation capabilities.

[0178] Based on the above embodiment, in another embodiment of the present application, Fig.15 The structure of the terminal proposed in the embodiment of the present application is shown in FIG. Figure 1 ,like Fig.15As shown, the terminal 10 proposed in the embodiment of the present application may include a training unit 11, a processing unit 12 and an estimation unit 13.

[0179] The training unit 11 is used to train an initial autoencoder model and an initial decoding model using first training data to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; and to train an initial prediction model using second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model; wherein the target prediction model is used to obtain the latent vector corresponding to the input image.

[0180] The processing unit 12 is used to optimize the initial autoencoding model and the initial decoding model by using the first latent vector, the second latent vector, the first panoramic image and the second panoramic image to obtain a target autoencoding model and a target decoding model;.

[0181] The estimation unit 13 is used to perform illumination estimation processing on the input image according to the target prediction model and the target decoding model to obtain a sky light panorama corresponding to the input image.

[0182] Furthermore, the processing unit 12 is specifically used to calculate a first loss function corresponding to the initial autoencoder model and the initial decoding model based on the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image; and to optimize the initial autoencoder model and the initial decoding model according to the first loss function to obtain the target autoencoder model and the target decoding model.

[0183] Furthermore, the processing unit 12 is specifically used to calculate a first norm loss function corresponding to the first latent vector and the second latent vector; calculate a second norm loss function corresponding to the first panoramic image, the second panoramic image and the first training data; and calculate the first loss function based on the first norm loss function and the second norm loss function.

[0184] Furthermore, the processing unit 12 is specifically used to calculate a third norm loss function corresponding to the first panoramic image and the first training data; calculate a fourth norm loss function corresponding to the second panoramic image and the first training data; and add the third norm loss function and the fourth norm loss function to obtain the second norm loss function.

[0185] Furthermore, the processing unit 12 is specifically used to add a first weight value to the first norm loss function and the second norm loss function; and generate the first loss function according to the first weight value, the first norm loss function and the second norm loss function.

[0186] Furthermore, the training unit 11 is specifically used to input the second training data into the initial prediction model, and output a prediction vector corresponding to the second training data; input the prediction vector into the target decoding model, and output a third panoramic image corresponding to the prediction vector; calculate a second loss function corresponding to the initial prediction model based on the prediction vector, the first latent vector, the first panoramic image and the third panoramic image; optimize the initial prediction model according to the second loss function to obtain the target prediction model.

[0187] Furthermore, the training unit 11 is specifically used to calculate the fifth norm loss function corresponding to the prediction vector and the first latent vector; calculate the sixth norm loss function corresponding to the first panoramic image and the third panoramic image; and calculate the second loss function based on the fifth norm loss function and the sixth norm loss function.

[0188] Furthermore, the training unit 11 is specifically used to add a second weight value to the fifth norm loss function and the sixth norm loss function; and generate the second loss function according to the second weight value, the fifth norm loss function and the sixth norm loss function.

[0189] Furthermore, the processing unit 12 is also used to strip the non-sky components in the second training data in the training unit 11 to obtain third training data; before determining the first training data according to the third training data and the fourth training data, strip the non-sky components in the second training data to obtain third training data; and determine the first training data according to the third training data and the fourth training data.

[0190] In the embodiments of the present application, further, Fig.16 The structure of the terminal proposed in the embodiment of the present application is shown in FIG. Figure 2 ,like Fig.16 As shown, the terminal 10 proposed in the embodiment of the present application may also include a processor 14 and a memory 15 storing executable instructions of the processor 14. Further, the terminal 10 may also include a communication interface 16 and a bus 17 for connecting the processor 14, the memory 15 and the communication interface 16.

[0191] Furthermore, in an embodiment of the present application, the processor 14 is used to train an initial autoencoder model and an initial decoding model using first training data to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; optimize the initial autoencoder model and the initial decoding model using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model; train an initial prediction model using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model; perform illumination estimation processing on the input image according to the target prediction model and the target decoding model to obtain a sky light panoramic image corresponding to the input image.

[0192] In an embodiment of the present application, the above-mentioned processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to realize the above-mentioned processor function can also be other, and the embodiment of the present application is not specifically limited. The memory can be connected to the processor, wherein the memory is used to store executable program code, and the program code includes computer operation instructions, and the memory may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0193] In the embodiment of the present application, the bus is used to connect the communication interface, the processor and the memory, as well as the mutual communication between these devices.

[0194] In the embodiment of the present application, the memory is used to store instructions and data.

[0195] In practical applications, the above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (flash memory), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, and provide instructions and data to the processor.

[0196] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0197] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0198] An embodiment of the present application provides a terminal, which uses first training data to train an initial autoencoder model and an initial decoding model to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; uses the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to optimize the initial autoencoder model and the initial decoding model to obtain a target autoencoder model and a target decoding model; uses the second training data, the first latent vector, the second latent vector, and the first panoramic image to train an initial prediction model to obtain a target prediction model; performs illumination estimation processing on an input image according to the target prediction model and the target decoding model to obtain a sky light panoramic image corresponding to the input image. It can be seen that the present application optimizes the initial autoencoder model and the initial decoding model by training the first latent vector, the second latent vector, the first panoramic image and the second panoramic image obtained by the initial autoencoder model and the initial decoding model to obtain the target autoencoder model and the target decoding model, and then uses the first latent vector, the second latent vector and the first panoramic image in the above training process to train the initial prediction model to obtain the target prediction model. Finally, the input image can be decoded by the constructed target prediction model to obtain the latent vector corresponding to the input image, and then the latent vector is decoded by the target decoding model to obtain the sky light panorama corresponding to the latent vector. The sky light panorama can display all the lighting information in the input image, thereby producing a better lighting estimation effect. Therefore, the target prediction model and target decoding model proposed in the present application have better lighting estimation capabilities.

[0199] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the illumination estimation method as described above is implemented.

[0200] Specifically, a program instruction corresponding to a lighting estimation method in this embodiment may be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instruction corresponding to a lighting estimation method in the storage medium is read or executed by the terminal, the following steps are included:

[0201] Using the first training data to train an initial autoencoder model and an initial decoding model, obtaining a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector;

[0202] Optimizing the initial autoencoder model and the initial decoding model using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model;

[0203] Training an initial prediction model using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model;

[0204] The input image is subjected to illumination estimation processing according to the target prediction model and the target decoding model to obtain a sky light panorama corresponding to the input image.

[0205] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0206] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0209] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for estimating illumination, characterized in that: The method comprises: Using the first training data to train an initial autoencoder model and an initial decoding model, a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector are obtained; the first latent vector refers to a normal latent vector obtained after the first training data is input into the initial autoencoder model; the second latent vector refers to a latent vector output after the first training data with color distortion is input into the initial autoencoder model; Optimizing the initial autoencoder model and the initial decoding model using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model; Training an initial prediction model using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model; Performing illumination estimation processing on the input image according to the target prediction model and the target decoding model to obtain a sky light panorama corresponding to the input image; The step of optimizing the initial autoencoder model and the initial decoding model by using the first latent vector, the second latent vector, the first panoramic image, and the second panoramic image to obtain a target autoencoder model and a target decoding model includes: Calculating a first norm loss function corresponding to the first latent vector and the second latent vector; Calculating a second norm loss function corresponding to the first panoramic image, the second panoramic image, and the first training data; Calculating a first loss function according to the first norm loss function and the second norm loss function; The initial autoencoding model and the initial decoding model are optimized according to the first loss function to obtain the target autoencoding model and the target decoding model.

2. The method according to claim 1, characterized in that The calculating the second norm loss function corresponding to the first panoramic image, the second panoramic image, and the first training data includes: Calculating a third norm loss function corresponding to the first panoramic image and the first training data; Calculating a fourth norm loss function corresponding to the second panoramic image and the first training data; The third norm loss function and the fourth norm loss function are added to obtain the second norm loss function.

3. The method according to claim 1, characterized in that The calculating the first loss function according to the first norm loss function and the second norm loss function includes: Adding a first weight value to the first norm loss function and the second norm loss function; The first loss function is generated according to the first weight value, the first norm loss function and the second norm loss function.

4. The method according to claim 1, characterized in that: The step of training an initial prediction model using the second training data, the first latent vector, the second latent vector, and the first panoramic image to obtain a target prediction model includes: Inputting the second training data into the initial prediction model, and outputting a prediction vector corresponding to the second training data; Inputting the prediction vector into the target decoding model, and outputting a third panoramic image corresponding to the prediction vector; Calculate a second loss function corresponding to the initial prediction model according to the prediction vector, the first latent vector, the first panoramic image, and the third panoramic image; The initial prediction model is optimized according to the second loss function to obtain the target prediction model.

5. The method according to claim 4, characterized in that The calculating a second loss function corresponding to the initial prediction model according to the prediction vector, the first latent vector, the first panoramic image, and the third panoramic image includes: Calculating a fifth norm loss function corresponding to the prediction vector and the first latent vector; Calculating the sixth norm loss function corresponding to the first panoramic image and the third panoramic image; The second loss function is calculated according to the fifth norm loss function and the sixth norm loss function.

6. The method according to claim 5, characterized in that The calculating the second loss function according to the fifth norm loss function and the sixth norm loss function comprises: Adding a second weight value to the fifth norm loss function and the sixth norm loss function; The second loss function is generated according to the second weight value, the fifth norm loss function and the sixth norm loss function.

7. The method according to claim 1, characterized in that Before using the first training data to train the initial autoencoder model and the initial decoding model, and obtaining the first latent vector and the second latent vector corresponding to the first training data, the first panoramic image corresponding to the first latent vector, and the second panoramic image corresponding to the second latent vector, the method further includes: performing stripping processing on the non-sky components in the second training data to obtain third training data; The first training data is determined according to the third training data and the fourth training data.

8. A terminal, characterized in that: The terminal comprises a training unit, a processing unit and an estimation unit, The training unit is used to train an initial autoencoder model and an initial decoding model using the first training data to obtain a first latent vector and a second latent vector corresponding to the first training data, a first panoramic image corresponding to the first latent vector, and a second panoramic image corresponding to the second latent vector; and using the second training data, the first latent vector, the second latent vector, and the first panoramic image to train an initial prediction model to obtain a target prediction model; the first latent vector refers to a normal latent vector obtained after the first training data is input into the initial autoencoder model; the second latent vector refers to a latent vector output after the first training data with color distortion is input into the initial autoencoder model; The processing unit is used to calculate the first norm loss function corresponding to the first latent vector and the second latent vector; calculate the second norm loss function corresponding to the first panoramic image, the second panoramic image and the first training data; calculate the first loss function according to the first norm loss function and the second norm loss function; optimize the initial autoencoder model and the initial decoding model according to the first loss function to obtain a target autoencoder model and a target decoding model; The estimation unit is used to perform illumination estimation processing on the input image according to the target prediction model and the target decoding model to obtain a sky light panorama corresponding to the input image.

9. A terminal, characterized in that: The terminal includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a program stored thereon, applied to a terminal, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images