Augmented reality-oriented high and low frequency illumination estimation method and device

Abstract

The invention discloses an augmented reality-oriented high and low frequency illumination estimation method and device, and the method comprises the steps: generating a panoramic HDR image through employing a high frequency illumination estimation method based on a generative adversarial network in an augmented reality scene generation system, and providing high frequency illumination information for a virtual model; through feature extraction of front and rear view angle images, an environmental spherical harmonic coefficient is estimated, and low-frequency illumination information is provided for a virtual model. According to the method, the robustness of illumination estimation of the mobile augmented reality equipment under the complex conditions that the visual angle of a camera is small, the illumination environment changes rapidly and the like is improved, and the long-time and real-time illumination condition estimation capability of the augmented reality equipment is expanded.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality, and in particular to a high- and low-frequency illumination estimation method and device for augmented reality. Background Art

[0002] Lighting conditions in outdoor scenes are unpredictable, varying depending on the environment and time of day. Augmented reality scenarios require instant feedback on real-time lighting conditions to calculate realistic lighting effects for objects. Addressing the uncertainties in light source position and intensity requires an accurate lighting estimation method. Methods that focus on the accuracy of lighting estimation algorithms often rely on additional landmarks, but fail to produce effective results without the necessary marker placement. Lighting estimation on mobile devices presents two challenges. First, due to the limited field of view (FOV) of mobile device cameras, estimating the lighting of the entire scene requires pre-scanning panoramic images, which is unsuitable for outdoor scenes with constantly changing ambient lighting. Second, images captured by mobile devices are all LDR images, which cannot be used in environment maps. Consequently, rendering of metallic objects is suboptimal. Furthermore, due to the lack of high dynamic range (HDR), lighting realism is lacking. In summary, while traditional lighting estimation algorithms can achieve good results in certain scenarios, they cannot fully meet the performance requirements for real-time, high dynamic range (HDR) lighting estimation in complex environments. Summary of the Invention

[0003] The purpose of the present invention is to provide a high- and low-frequency illumination estimation method and device for augmented reality, which improves the illumination consistency of the model under different conditions in the augmented reality system and determines the direction of sunlight.

[0004] The technical solution to achieve the purpose of the present invention is: a high- and low-frequency illumination estimation method for augmented reality, comprising the steps of:

[0005] Get the RGB image and geographic time information of the current environment;

[0006] The RGB image is cropped to obtain the central pixel area of the image. The RGB image is captured by the Hololens2 camera. The RGB image size input to the network is 256 pixels in height and 192 pixels in width.

[0007] The RGB image is used as the input of the StyleGAN network to predict the image panorama;

[0008] Based on the panoramic LDR image, the panoramic HDR image is output through the StyleGAN network;

[0009] Based on the before and after image pairs of the region in the panoramic HDR image, the spherical harmonic coefficients of the current environment are obtained through a spherical harmonic coefficient estimation network composed of a feature extraction module and an illumination estimation module.

[0010] Furthermore, the StyleGAN network includes two parallel discriminators, one for generating panoramic HDR images and the other for generating low FOV panoramas. The network output includes panoramic HDR images and corresponding lighting renderings of different material models. The network is represented as:

[0011]

[0012] Where, Represents the minimization and maximization game process of the network; G represents the generator network, which is used to generate HDR and LDR images; D(x) and D'(x) are used as discriminators for HDR and LDR images respectively; it receives an image x and outputs the probability of whether the image is a true HDR or LDR image; and Respectively represent the real data distribution P data (x) and P′ data (x) Expected loss for discriminating HDR and LDR images; and Denotes the expected loss of the HDR and LDR images generated by the generator G, respectively, when the discriminator D and D' judge them as fake; w denotes the noise or feature vector input to the generator G, usually from the prior distribution P w mid-sampling; ψ(·) and φ(·) are functions used to convert the image generated by the generator G into an HDR image and an LDR image, respectively.

[0013] Furthermore, the different material models include Diffuse, Specular, and Metallic.

[0014] Furthermore, the spherical harmonic coefficient estimation network includes a feature extraction module and an illumination estimation module; the feature extraction module is built based on the ConvNeXt V2 model, and the illumination estimation module completes the spherical harmonic coefficient estimation through a convolutional layer, a RELU layer, and a SA attention layer;

[0015] The SH coefficient loss is used to measure the error between the true SH coefficient and the SH coefficient predicted by the network. The mean square error loss is used to calculate the first four-order SH coefficients. The formula is expressed as:

[0016]

[0017] Where, is the rendering loss function, which represents the difference between the generated image and the image rendered using spherical harmonic coefficients (SH); Is a normalization factor used to average the loss value to each pixel and each channel, where W is the width of the image, H is the height of the image, and C is the number of channels of the image; R(SH, x, y, c) represents the value of channel c at pixel (x, y) in the rendering result using the spherical harmonic coefficient SH; (R generated (SH,x,y,c)-R(SH,x,y,c)) 2 Represents the squared difference between the generated image and the true or target image.

[0018] A high- and low-frequency illumination estimation device for augmented reality is used to implement the above-mentioned method. The device includes an image acquisition module, an image preprocessing module, an illumination estimation module, and an output module. The image acquisition module is used to obtain an RGB image of the current scene, the image preprocessing module is used to crop the RGB image, and the illumination estimation module adopts a StyleGAN network. The RGB image under low FOV conditions is used as the input of the StyleGAN network to estimate the HDR illumination environment panorama; the front and back image pairs of the panoramic HDR image are used as the input of the spherical harmonic coefficient estimation network to estimate the environmental spherical harmonic parameters.

[0019] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0020] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0021] A computer program product comprises a computer program, which implements the steps of the above method when executed by a processor.

[0022] Compared with the existing technology, the present invention has positive effects: the present invention uses RGB images under low FOV conditions to input the neural network algorithm, generates a panoramic HDR image by cropping the LDR image, and provides high-frequency information of the model; at the same time, the panoramic front and rear view images are subjected to feature extraction and estimation to calculate the environmental spherical harmonic coefficients, providing low-frequency lighting information of the model, which improves the robustness of lighting estimation in complex augmented reality environments such as small camera viewing angles and changing lighting environments, and improves the lighting consistency of the model under different conditions in the augmented reality system; at the same time, the method can output the real-time solar azimuth based on geographic time and input image, and determine the outdoor sunlight azimuth. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flowchart of the illumination estimation of the present invention.

[0024] Figure 2This is a diagram of the network structure for the high-frequency illumination estimation LDR / HDR panorama generation method of the present invention.

[0025] Figure 3 This is a diagram of the low-frequency spherical harmonic parameter estimation network structure of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] The high- and low-frequency illumination estimation method for augmented reality in this embodiment relies on the implementation of an augmented reality scene generation system based on a cloud architecture. The main control computer of the system uses the RGB image obtained by Hololens2 to run a real-time illumination estimation algorithm and sends the processing results to the HoloLens2 augmented reality head-mounted display for enhanced display. The illumination estimation algorithm flow chart is shown in FIG. Figure 1 As shown, the method includes the following steps:

[0028] Step 1: The main control computer uses Hololens2 to obtain RGB images in real time;

[0029] Step 2: The main control computer first crops the RGB image and takes the central pixel area of the image; the RGB image input to the StyleGAN network uses a height of 256 pixels and a width of 192 pixels.

[0030] Step 3: Use the RGB image as the input of the StyleGAN network to predict the image panorama and generate a panoramic LDR image;

[0031] Step 4: Based on the panoramic LDR image, the panoramic HDR image is output through the network;

[0032] Step 5: Send the panoramic HDR image estimation results to the HoloLens2 augmented reality head-mounted display as an ambient light map.

[0033] Step 6: The original camera image is cropped into two 224*224 pixel image pairs as the input for the low-frequency illumination estimation part. After passing through the feature extraction network and the parameter estimation network, the current environmental spherical harmonic coefficients are obtained.

[0034] In one embodiment, in the first step of the steps, the main control computer obtains the RGB image of Hololens 2 in real time mainly by opening the main camera address of Hololens 2 and then using the cv2.VideoCapture method.

[0035] In one embodiment, the current brightness level is calculated by calculating the image histogram, and the histogram mapping formula used can be described as:

[0036]

[0037] Among them, s k Refers to the value of the current gray level after being mapped by the cumulative distribution function, n is the sum of the pixels in the image, n j is the number of pixels at the current gray level, and L is the total number of gray levels in the image.

[0038] In one embodiment, the multi-source data ambient light estimation method utilizes the panoramic HDR image result and the real-time image of the HoloLens 2 camera for joint estimation. The specific method is as follows:

[0039] Step 1: The discriminator of the StyleGAN network inputs the low FOV LDR image and the expanded LDR image respectively, shares the same generator, and outputs the estimated panoramic HDR image. The network description is as follows:

[0040]

[0041] Where, represents the minimization and maximization game process of the network. G represents the generator network, which is used to generate HDR and LDR images. D(x) and D'(x) are used as the discriminator for HDR and LDR images, respectively. It receives an image x and outputs the probability of whether the image is a true HDR or LDR image. and Respectively represent the real data distribution P data (x) and P′ data (x), the expected loss for discriminating HDR and LDR images. and Denotes the expected loss of the HDR and LDR images generated by the generator G, respectively, when the discriminator D and D' judges them as fake. w represents the noise or feature vector input to the generator G, usually from the prior distribution P w ψ(·) and φ(·) are functions used to convert the image generated by the generator G into an HDR image and an LDR image, respectively.

[0042] Step 2: Use Hololens 2 as the main camera to calculate the image light level in real time. The current brightness level is calculated through the image histogram. The histogram mapping formula used can be described as:

[0043]

[0044] Among them, s k Refers to the value of the current gray level after being mapped by the cumulative distribution function, n is the sum of the pixels in the image, n jis the number of pixels at the current gray level, and L is the total number of gray levels in the image. The calculation formula can be described as:

[0045]

[0046] Among them, R is the brightness level obtained, S i is the grayscale level of a single pixel, P is the total number of pixels, and M is the maximum grayscale level constant.

[0047] Step 3: The original camera image is cropped into two 224*224 pixel image pairs as the input for the low-frequency illumination estimation part. The illumination estimation network uses the mean square error loss to calculate the first four-order SH coefficients. The formula can be expressed as:

[0048]

[0049] Where, is the rendering loss function, which represents the difference between the generated image and the image rendered using the spherical harmonic coefficients (SH). Is a normalization factor used to average the loss value to each pixel and each channel, where W is the width of the image, H is the height of the image, and C is the number of channels of the image. R(SH, x, y, c) represents the value of channel c at pixel (x, y) in the rendering result using the spherical harmonic coefficient SH. (R generated (SH,x,y,c)-R(SH,x,y,c)) 2 Represents the squared difference between the generated image and the true or target image.

[0050] In one embodiment, the result sent to the HoloLens 2 augmented reality head-mounted display includes a panoramic HDR image and 16 spherical harmonic parameters of the first four orders of the current environment.

[0051] Figure 2 This is the network architecture for generating LDR / HDR panoramic images for high-frequency illumination estimation in this invention. The entire process begins with noise z, which is mapped to a latent space w via a mapping network f consisting of two fully connected layers FC, providing guiding feature information for the subsequent generation process. The generator G receives the latent space w and uses the features therein to construct image content, structure, and details. The generated image undergoes an HDR conversion ψ and an LDR conversion φ, respectively, to obtain a panoramic HDR image and an LDR image at a low FOV. The discriminators D and D' are used to judge the authenticity of the panoramic HDR image and the LDR image, respectively. By comparing them with the real image, they help the network learn how to generate more realistic images, ultimately achieving the goal of simultaneously generating high-quality panoramic HDR images and corresponding LDR images.

[0052] Figure 3This is a diagram of the low-frequency spherical harmonic parameter estimation network structure of the present invention. Figure a is the feature extraction module. The input data is sequentially subjected to preliminary feature extraction and normalization processing through the convolution layer (Conv) and layer normalization (LN), and then multiple ConvNeXtV2 blocks are used to further extract and process features. The ConvNeXtV2 block combines depth convolution and point-by-point convolution to effectively capture the relationship between spatial features and channels, and finally outputs a feature map. Figure b is the spherical harmonic estimation module, which receives the feature map output by the feature extraction module and performs illumination estimation and feature fusion. Through multiple convolution operations, features are gradually extracted and fused, and finally spherical harmonic parameters are output to describe the illumination information of the scene. Figure c shows the structure of the ConvNeXtV2 block. The block first uses 1×7 convolution (d7×7, 96) to capture local spatial features, and then stabilizes the training process through layer normalization (LN). Next, 1×1 convolution is used to reorganize features and adjust dimensions between channels, and then the GELU activation function is used to introduce nonlinearity to enhance the expressive power of the model. Finally, the features are further optimized through GRN, and the entire block realizes feature fusion through residual connection to improve the efficiency and effect of feature extraction.

[0053] Based on the same inventive concept, the present invention also provides a high- and low-frequency illumination estimation device for augmented reality, comprising an image acquisition module, an image preprocessing module, an illumination estimation module, and an output module. The image acquisition module is used to acquire an RGB image of the current scene, the image preprocessing module is used to crop the RGB image, and the illumination estimation module employs a StyleGAN network, using the RGB image under low FOV conditions as input to the StyleGAN network to estimate the HDR illumination environment panorama. The output module calculates the output illumination intensity level and solar azimuth angle based on the uncropped RGB image using a histogram and geographic time. The front and back image pairs of the panoramic HDR image serve as input to the spherical harmonic coefficient estimation network, which estimates the environmental spherical harmonic parameters. High- and low-frequency illumination estimation methods are performed simultaneously to estimate both high- and low-frequency components.

[0054] This invention addresses the current problem of unstable lighting estimation results based on RGB images, which is affected by factors such as low real-time resolution and limited camera viewing range. Panoramic HDR images are more stable and can provide more realistic effects for model rendering in augmented reality.

[0055] The above embodiments are illustrations of specific implementation methods of the present invention, rather than limitations of the present invention. Technicians in the relevant technical fields can make various changes and modifications to obtain corresponding equivalent technical solutions without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions should be included in the scope of patent protection of the present invention.

Claims

1. A high- and low-frequency illumination estimation method for augmented reality, characterized in that: Including steps: Get the RGB image and geographic time information of the current environment; Crop the RGB image and take the center pixel area of the image; The RGB image is used as the input of the StyleGAN network to predict the image panorama; Based on the panoramic LDR image, the panoramic HDR image is output through the StyleGAN network; Based on the before and after image pairs of the region in the panoramic HDR image, the spherical harmonic coefficients of the current environment are obtained through a spherical harmonic coefficient estimation network composed of a feature extraction module and an illumination estimation module.

2. The high- and low-frequency illumination estimation method for augmented reality according to claim 1, characterized in that: The StyleGAN network includes two parallel discriminators, one for generating panoramic HDR images and the other for generating low-FOV panoramas. The network output includes panoramic HDR images and corresponding lighting renderings of different material models. The network is represented as: Where, Represents the minimization and maximization game process of the network; G represents the generator network, which is used to generate HDR and LDR images; D(x) and D'(x) are the discriminators for HDR and LDR images, respectively, which receive an image x and output the probability of whether the image is a true HDR or LDR image; and Respectively represent the real data distribution P data (x) and P′ data (x) Expected loss for discriminating HDR and LDR images; and Denotes the expected loss of the HDR and LDR images generated by the generator G, respectively, when the discriminators D and D' judge them as fake; w denotes the noise or feature vector input to the generator G, from the prior distribution P w mid-sampling; ψ(·) and φ(·) are functions used to convert the image generated by the generator G into an HDR image and an LDR image, respectively.

3. The high- and low-frequency illumination estimation method for augmented reality according to claim 2, characterized in that: The different material models include Diffuse, Specular, and Metallic.

4. The high- and low-frequency illumination estimation method for augmented reality according to claim 1, characterized in that: The spherical harmonic coefficient estimation network includes a feature extraction module and an illumination estimation module; the feature extraction module is built based on the ConvNeXtV2 model, and the illumination estimation module completes the spherical harmonic coefficient estimation through a convolutional layer, a RELU layer, and a SA attention layer; The SH coefficient loss is used to measure the error between the true SH coefficient and the SH coefficient predicted by the network. The mean square error loss is used to calculate the first four-order SH coefficients. The formula is expressed as: Where, is the rendering loss function, which represents the difference between the generated image and the image rendered using the spherical harmonic coefficients; Is a normalization factor used to average the loss value to each pixel and each channel, where W is the width of the image, H is the height of the image, and C is the number of channels of the image; R(SH,x,y,c) represents the value of channel c at pixel (x,y) in the rendering result using the spherical harmonic coefficient SH; (R generated (SH,x,y,c)-R(SH,x,y,c)) 2 Represents the squared difference between the generated image and the true or target image.

5. The high- and low-frequency illumination estimation method for augmented reality according to claim 1, characterized in that: The RGB image is collected by a Hololens2 camera.

6. The high- and low-frequency illumination estimation method for augmented reality according to claim 1, characterized in that: The RGB image size of the StyleGAN network input is 256 pixels in height and 192 pixels in width.

7. A high- and low-frequency illumination estimation device for augmented reality, characterized in that: Used to implement the method described in any one of claims 1 to 6, the device includes an image acquisition module, an image preprocessing module, an illumination estimation module and an output module, the image acquisition module is used to obtain an RGB image of the current scene, the image preprocessing module is used to crop the RGB image, the illumination estimation module adopts a StyleGAN network, takes the RGB image under low FOV conditions as the input of the StyleGAN network, and estimates the HDR illumination environment panorama; the front and back image pairs of the panoramic HDR image are used as the input of the spherical harmonic coefficient estimation network to estimate the environmental spherical harmonic parameters.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images