Illumination estimation method, apparatus, and system

Abstract

The application provides a light estimation method, device and system, relates to the field of augmented reality, and can realize accurate estimation of light information in a scene. The method comprises the following steps: receiving a plurality of first images sent by a first electronic device, the plurality of first images being a plurality of images of a scene where the first electronic device is located, which are captured by the first electronic device at a first time with different exposure parameters; obtaining pose information of the first electronic device; obtaining a plurality of second images matched with the pose information of the first electronic device, the plurality of second images being the same scene as the plurality of first images but with different shooting angles; obtaining a high dynamic range panorama image of the scene according to the plurality of first images and the plurality of second images, the high dynamic range panorama image being used to extract light information of the scene at the first time, and the light information of the scene at the first time being used to render a virtual object displayed on the first electronic device.

Description

Technical Field

[0001] This application relates to the field of augmented reality (AR) technology, and more particularly to a lighting estimation method, apparatus, and system. Background Technology

[0002] With the rapid iteration and widespread adoption of electronic devices such as smartphones and tablets, these devices can not only provide users with functions such as calls, text messages, and video playback, but also AR (Augmented Reality) services. AR cleverly integrates virtual information with the real world, allowing the two types of information to complement each other, thereby "enhancing" the real world.

[0003] Furthermore, with the development of artificial intelligence (AI) technology, AI has acquired advanced capabilities such as strong environmental understanding, virtual-real fusion imaging, and precise positioning. These capabilities can provide users with a "global-scale, constantly evolving, and seamlessly integrated" new digital world—the virtual world. However, placing virtual objects from the virtual world into real-world scenes has always presented significant challenges. To ensure lighting consistency between virtual objects and the real scene, it's necessary to obtain lighting information from the real scene, such as the direction of light sources. Lighting estimation, a fundamental characteristic of augmented reality applications, directly determines the realism of the rendered virtual objects. Inconsistencies between virtual objects and the real scene lead to a poor user experience.

[0004] Therefore, how to accurately estimate the lighting information in real-world scenes to achieve more realistic augmented reality effects has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a lighting estimation method, apparatus, and system, which helps to improve the accuracy of lighting estimation and achieve more realistic augmented reality effects.

[0006] In a first aspect, embodiments of this application provide an illumination estimation method, which may include: receiving multiple frames of first images sent by a first electronic device, wherein the multiple frames of first images are multiple frames of images of a scene in which the first electronic device is located, captured by the first electronic device at different exposure parameters at a first time; obtaining pose information of the first electronic device; obtaining multiple frames of second images that match the pose information of the first electronic device, wherein the multiple frames of second images have the same scene as the multiple frames of first images but different shooting angles; and obtaining a high dynamic range panoramic image of the scene based on the multiple frames of first images and the multiple frames of second images, wherein the high dynamic range panoramic image is used to extract illumination information of the scene at a first time, and the illumination information of the scene at a first time is used to render virtual objects displayed on the first electronic device.

[0007] Therefore, the first aspect of this application can be implemented by generating a high dynamic range panoramic image of the scene where the first electronic device is located, based on multiple first images with different exposure levels and multiple second images of the same scene but different shooting angles as the first images, to achieve accurate estimation of lighting information in the scene. Compared to estimating lighting information in the scene using a single-frame image, the lighting information obtained based on the high dynamic range panoramic image of the scene where the first electronic device is located is more accurate. Rendering virtual objects based on this lighting information can satisfy lighting consistency, making the displayed virtual objects more realistic and achieving seamless integration between virtual objects and the real scene.

[0008] In one possible design, acquiring multiple frames of second images that match the pose information of the first electronic device includes: matching the multiple frames of second images from a preset image library based on the pose information of the first electronic device. The preset image library includes multiple frames of images captured by the first electronic device and / or at least one second electronic device.

[0009] In one possible design, obtaining a high dynamic range (HDR) panoramic image of the scene based on multiple frames of first images and multiple frames of second images includes: generating a HDR local image based on the multiple frames of first images; generating a low dynamic range (LVR) panoramic image based on the multiple frames of second images; and transferring the illumination information from the HDR local image to the LVR panoramic image to obtain the HDR panoramic image of the scene.

[0010] In one possible design, obtaining the pose information of the first electronic device includes: generating a low dynamic range local image based on multiple frames of the first image; and determining the pose information of the first electronic device based on the low dynamic range local image.

[0011] In one possible design, generating a low dynamic range (LVR) panoramic image based on multiple frames of second images includes: transferring the illumination information of the local LVR images to the multiple frames of second images to obtain multiple frames of third images; and generating the LVR panoramic image based on the multiple frames of third images.

[0012] In one possible design, generating a low dynamic range (LVR) panoramic image based on multiple frames of third images includes: obtaining a low dynamic range incomplete image based on the multiple frames of third images; performing image completion processing on the low dynamic range incomplete image to generate the low dynamic range panoramic image.

[0013] In one possible design, image completion processing is performed on a low dynamic range (LVR) incomplete image to generate a LVR panoramic image, including: inputting the LVR incomplete image into a deep learning model and obtaining the LVR panoramic image output by the deep learning model.

[0014] In one possible design, the illumination information of a high dynamic range (HDR) local image is transferred to a low dynamic range (LR) panoramic image to obtain a HDR panoramic image of the scene. This includes: inputting the HDR local image and the LR panoramic image into a style transfer network, and obtaining the HDR panoramic image of the scene output by the style transfer network. The style transfer network is used to transfer the illumination information of the HDR local image to the LR panoramic image to obtain the HDR panoramic image.

[0015] In one possible design, the method further includes at least one of the following: determining the direction of the main light source based on the high dynamic range panoramic image; or, determining the spherical harmonic coefficients based on the high dynamic range panoramic image, and determining the ambient light intensity and the main light source intensity of the scene based on the spherical harmonic coefficients; or, determining an environmental texture map based on the high dynamic range panoramic image, wherein the environmental texture map is used for virtual objects with specular reflective materials to reflect the texture of the scene specularly.

[0016] In one possible design, the method may further include: sending a high dynamic range panoramic image or illumination information of the scene at a first moment to a first electronic device.

[0017] Secondly, embodiments of this application provide a lighting estimation method, the method comprising: detecting a first operation performed by a user; responding to the first operation, a first electronic device capturing multiple frames of first images of a scene in which the first electronic device is located at different exposure parameters at a first time; sending the multiple frames of first images to a server, the multiple frames of first images being used to obtain a high dynamic range panoramic image of the scene, the high dynamic range panoramic image being used to extract lighting information of the scene at the first time, and the lighting information of the scene at the first time being used to render virtual objects displayed on the first electronic device.

[0018] In one possible design, the method further includes: receiving a high dynamic range panoramic image or lighting information of the scene at a first moment sent by a server; and rendering a virtual object displayed on a first electronic device based on the high dynamic range panoramic image or the lighting information of the scene at the first moment.

[0019] Thirdly, embodiments of this application provide an illumination estimation method, which includes: detecting a first operation performed by a user; in response to the first operation, a first electronic device captures multiple frames of first images of a scene in which the first electronic device is located at different exposure parameters at a first time; acquiring multiple frames of second images matching the pose information of the first electronic device from a server, wherein the multiple frames of second images have the same scene as the multiple frames of first images but different shooting angles; and obtaining a high dynamic range panoramic image of the scene based on the multiple frames of first images and the multiple frames of second images, wherein the high dynamic range panoramic image is used to extract the illumination information of the scene at the first time, and the illumination information of the scene at the first time is used to render a virtual object displayed on the first electronic device.

[0020] In one possible design, obtaining a high dynamic range (HDR) panoramic image of the scene based on multiple frames of first images and multiple frames of second images includes: generating a HDR local image based on the multiple frames of first images; generating a low dynamic range (LVR) panoramic image based on the multiple frames of second images; and transferring the illumination information from the HDR local image to the LVR panoramic image to obtain the HDR panoramic image of the scene.

[0021] In one possible design, obtaining multiple frames of second images matching the pose information of a first electronic device from a server includes: obtaining the pose information of the first electronic device; sending the pose information of the first electronic device to the server, the pose information of the first electronic device being used by the server to match multiple frames of second images from a preset image library. The preset image library includes multiple frames of images taken by the first electronic device and / or at least one second electronic device; and receiving the multiple frames of second images sent by the server.

[0022] In one possible design, obtaining the pose information of the first electronic device includes: generating a low dynamic range local image based on multiple frames of the first image; and determining the pose information of the first electronic device based on the low dynamic range local image.

[0023] In one possible design, generating a low dynamic range (LVR) panoramic image based on multiple frames of second images includes: transferring the illumination information of the local LVR images to the multiple frames of second images to obtain multiple frames of third images; and generating the LVR panoramic image based on the multiple frames of third images.

[0024] In one possible design, generating a low dynamic range (LVR) panoramic image based on multiple frames of third images includes: obtaining a low dynamic range incomplete image based on the multiple frames of third images; performing image completion processing on the low dynamic range incomplete image to generate the low dynamic range panoramic image.

[0025] In one possible design, image completion processing is performed on a low dynamic range (LVR) incomplete image to generate a LVR panoramic image, including: inputting the LVR incomplete image into a deep learning model and obtaining the LVR panoramic image output by the deep learning model.

[0026] In one possible design, the illumination information of a high dynamic range (HDR) local image is transferred to a low dynamic range (LR) panoramic image to obtain a HDR panoramic image of the scene. This includes: inputting the HDR local image and the LR panoramic image into a style transfer network, and obtaining the HDR panoramic image of the scene output by the style transfer network. The style transfer network is used to transfer the illumination information of the HDR local image to the LR panoramic image to obtain the HDR panoramic image.

[0027] In one possible design, the method further includes at least one of the following: determining the direction of the main light source based on the high dynamic range panoramic image; or, determining the spherical harmonic coefficients based on the high dynamic range panoramic image, and determining the ambient light intensity and the main light source intensity of the scene based on the spherical harmonic coefficients; or, determining an environmental texture map based on the high dynamic range panoramic image, wherein the environmental texture map is used for virtual objects with specular reflective materials to reflect the texture of the scene specularly.

[0028] Fourthly, this application provides an illumination estimation device, which can be a server, such as a chip or system-on-a-chip in the server, or a functional module in the server for implementing the first aspect or any possible implementation of the first aspect. For example, the illumination estimation device includes: a transceiver module for receiving multiple frames of first images sent by a first electronic device, the multiple frames of first images being multiple frames of images of the scene where the first electronic device is located, taken by the first electronic device at different exposure parameters at a first time; a processing module for acquiring pose information of the first electronic device; the processing module is further configured to acquire multiple frames of second images matching the pose information of the first electronic device, the multiple frames of second images having the same scene as the multiple frames of first images but different shooting angles; the processing module is further configured to obtain a high dynamic range panoramic image of the scene based on the multiple frames of first images and the multiple frames of second images, the high dynamic range panoramic image being used to extract illumination information of the scene at a first time, and the illumination information of the scene at a first time being used to render a virtual object displayed on the first electronic device.

[0029] In one possible design, the processing module is used to: obtain multiple frames of second images from a preset image library based on the pose information of the first electronic device. The preset image library includes multiple frames of images captured by the first electronic device and / or at least one second electronic device.

[0030] In one possible design, the processing module is used to: generate a high dynamic range local image based on the multi-frame first image;

[0031] Based on the multiple frames of the second image, a low dynamic range panoramic image is generated; the illumination information of the high dynamic range local image is transferred to the low dynamic range panoramic image to obtain a high dynamic range panoramic image of the scene.

[0032] In one possible design, the processing module is used to: generate a low dynamic range local image based on multiple frames of the first image; and determine the pose information of the first electronic device based on the low dynamic range local image.

[0033] In one possible design, the processing module is used to: transfer the illumination information of the low dynamic range local image to multiple frames of second images to obtain multiple frames of third images; and generate a low dynamic range panoramic image based on the multiple frames of third images.

[0034] In one possible design, the processing module is used to: obtain a low dynamic range incomplete image based on multiple frames of third images; perform image completion processing on the low dynamic range incomplete image to generate a low dynamic range panoramic image.

[0035] In one possible design, the processing module is used to: input a low dynamic range incomplete image into a deep learning model, and obtain a low dynamic range panoramic image output by the deep learning model.

[0036] In one possible design, the processing module is used to: input a high dynamic range local image and a low dynamic range panoramic image into a style transfer network to obtain a high dynamic range panoramic image of the scene output by the style transfer network; wherein, the style transfer network is used to transfer the illumination information of the high dynamic range local image to the low dynamic range panoramic image to obtain the high dynamic range panoramic image.

[0037] In one possible design, the processing module is also used to perform at least one of the following: determining the direction of the main light source based on the high dynamic range panoramic image; or, determining the spherical harmonic coefficients based on the high dynamic range panoramic image, and determining the ambient light intensity and the main light source intensity of the scene based on the spherical harmonic coefficients; or, determining an environmental texture map based on the high dynamic range panoramic image, wherein the environmental texture map is used for virtual objects with specular reflective materials to reflect the texture of the scene specularly.

[0038] In one possible design, the transceiver module is also used to: send a high dynamic range panoramic image or the scene's illumination information to the first electronic device at a specific moment.

[0039] Fifthly, this application provides a lighting estimation device, which can be a first electronic device, such as a chip or system-on-a-chip in the first electronic device, or a functional module in the first electronic device for implementing the second aspect or any possible implementation of the second aspect. For example, the lighting estimation device includes: a processing module for detecting a first operation performed by a user; the processing module is further configured to, in response to the first operation, capture multiple frames of first images of the scene where the first electronic device is located at different exposure parameters at a first time; and a transceiver module for sending the multiple frames of first images to a server, the multiple frames of first images being used to obtain a high dynamic range panoramic image of the scene, the high dynamic range panoramic image being used to extract lighting information of the scene at the first time, and the lighting information of the scene at the first time being used to render a virtual object displayed on the lighting estimation device.

[0040] In one possible design, the transceiver module is further configured to: receive a high dynamic range panoramic image or the scene's illumination information at a first moment from the server; the processing module is further configured to render a virtual object displayed on the illumination estimation device based on the high dynamic range panoramic image or the scene's illumination information at that first moment.

[0041] In a sixth aspect, this application provides an illumination estimation device, which can be a first electronic device, such as a chip or system-on-a-chip in the first electronic device, or a functional module in the first electronic device for implementing the second aspect or any possible implementation thereof. For example, the illumination estimation device includes: a processing module configured to detect a first operation performed by a user, and in response to the first operation, to capture multiple frames of first images of the scene where the first electronic device is located at different exposure parameters at a first time. The processing module is further configured to acquire multiple frames of second images matching the pose information of the first electronic device from a server via a transceiver module, wherein the multiple frames of second images have the same scene as the multiple frames of first images but different shooting angles. The processing module is further configured to obtain a high dynamic range panoramic image of the scene based on the multiple frames of first images and the multiple frames of second images, wherein the high dynamic range panoramic image is used to extract illumination information of the scene at a first time, and the illumination information of the scene at a first time is used to render a virtual object displayed on the first electronic device.

[0042] In one possible design, the processing module is used to: generate a high dynamic range (HDR) local image based on multiple frames of the first image; generate a low dynamic range (LVR) panoramic image based on multiple frames of the second image; and transfer the illumination information from the HDR local image to the LVR panoramic image to obtain the HDR panoramic image of the scene.

[0043] In one possible design, the processing module is used to acquire the pose information of the first electronic device. The transceiver module is used to send the pose information of the first electronic device to the server, which is used by the server to match multiple frames of second images from a preset image library. The preset image library includes multiple frames of images taken by the first electronic device and / or at least one second electronic device. The transceiver module is also used to receive the multiple frames of second images sent by the server.

[0044] In one possible design, the processing module is used to: generate low dynamic range local images based on multiple frames of the first image; and determine the pose information of the first electronic device based on the low dynamic range local images.

[0045] In one possible design, the processing module is used to: transfer the illumination information of the low dynamic range local image to multiple frames of second images to obtain multiple frames of third images. Based on the multiple frames of third images, a low dynamic range panoramic image is generated.

[0046] In one possible design, the processing module is used to: obtain a low dynamic range (LVR) incomplete image based on multiple frames of third images; perform image completion processing on the LVR incomplete image to generate a LVR panoramic image.

[0047] In one possible design, the processing module is used to: input the low dynamic range incomplete image into the deep learning model, and obtain the low dynamic range panoramic image output by the deep learning model.

[0048] In one possible design, the processing module is used to: input a high dynamic range (HDR) local image and a low dynamic range (LVR) panoramic image into a style transfer network, and obtain a HDR panoramic image of the scene output by the style transfer network. Specifically, the style transfer network is used to transfer the illumination information from the HDR local image to the LVR panoramic image to obtain the HDR panoramic image.

[0049] In one possible design, the processing module is also used to perform at least one of the following: determining the direction of the main light source based on the high dynamic range panoramic image; or, determining the spherical harmonic coefficients based on the high dynamic range panoramic image, and determining the ambient light intensity and the main light source intensity of the scene based on the spherical harmonic coefficients; or, determining an environmental texture map based on the high dynamic range panoramic image, wherein the environmental texture map is used for virtual objects with specular reflective materials to reflect the texture of the scene specularly.

[0050] In a seventh aspect, embodiments of this application provide an illumination estimation apparatus, comprising:

[0051] One or more processors;

[0052] Memory, used to store one or more programs;

[0053] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect or any possible design of the first aspect.

[0054] Eighthly, embodiments of this application provide an illumination estimation apparatus, comprising:

[0055] One or more processors;

[0056] Memory, used to store one or more programs;

[0057] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the second aspect or any possible design of the second aspect, or the one or more processors implement the method as described in the third aspect or any possible design of the third aspect.

[0058] Ninthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on a computer, cause the computer to perform the illumination estimation method as described in the first aspect or any possible design of the first aspect, or cause the computer to perform the illumination estimation method as described in the second aspect or any possible design of the second aspect, or cause the computer to perform the illumination estimation method as described in the third aspect or any possible design of the third aspect.

[0059] In a tenth aspect, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the illumination estimation method as described in the first aspect or any possible design of the first aspect, or causes the computer to perform the illumination estimation method as described in the second aspect or any possible design of the second aspect, or causes the computer to perform the illumination estimation method as described in the third aspect or any possible design of the third aspect.

[0060] Eleventhly, embodiments of this application provide a lighting estimation system, which includes an electronic device and a server. The electronic device establishes a communication connection with the server. The server is used to execute the lighting estimation method as described in the first aspect or any possible design of the first aspect. The electronic device is used to execute the lighting estimation method as described in the second aspect or any possible design of the second aspect.

[0061] In a twelfth aspect, embodiments of this application provide an illumination estimation system, which includes an electronic device and a server. The electronic device establishes a communication connection with the server and is used to perform the illumination estimation method as described in the third aspect or any possible design of the third aspect.

[0062] In a thirteenth aspect, embodiments of this application provide an electronic device, which serves as a first electronic device, including a display component; a camera; one or more processors; and a memory; the memory being used to store computer program code, the computer program code including computer instructions, which, when read from the memory by the processor, cause the first electronic device to perform the method described in the third aspect or any possible design of the third aspect.

[0063] In this application, the illumination estimation device, computer storage medium, computer program product, or illumination estimation system provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0064] Figure 1 A schematic diagram of an illumination estimation system provided in an embodiment of this application;

[0065] Figure 2 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application;

[0066] Figure 3 This is a schematic diagram of the structure of a server 30 provided in an embodiment of this application;

[0067] Figure 4 A structural block diagram of an illumination estimation system provided in an embodiment of this application;

[0068] Figure 5 A schematic flowchart illustrating an illumination estimation method provided in an embodiment of this application;

[0069] Figure 6 A schematic flowchart illustrating an illumination estimation method provided in an embodiment of this application;

[0070] Figure 7 A schematic flowchart illustrating an illumination estimation method provided in an embodiment of this application;

[0071] Figure 8 A schematic diagram illustrating the processing steps of an illumination estimation method provided in an embodiment of this application;

[0072] Figure 9 A schematic diagram of a user interface provided for an embodiment of this application;

[0073] Figure 10 A schematic diagram of a style transfer network provided in an embodiment of this application;

[0074] Figure 11 This is a schematic diagram of the training structure of the style transfer network provided in an embodiment of this application. Detailed Implementation

[0075] The illumination estimation method, apparatus, and system provided in this application will now be described in detail with reference to the accompanying drawings.

[0076] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0077] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0078] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0079] In the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone.

[0080] First, a brief introduction to the relevant concepts and technologies involved in the embodiments of this application will be given.

[0081] Lighting consistency refers to ensuring that virtual objects have the same lighting effects as real-world scenes. The goal of lighting consistency is to make the lighting conditions of virtual objects identical to those in the real-world scene, ensuring consistent brightness, shadows, and other effects to enhance the realism of the virtual objects. The key to solving the lighting consistency problem is obtaining accurate lighting information from the real-world scene. Accurate lighting information enables more realistic augmented reality effects and improves the user experience.

[0082] Dynamic range, in many fields, represents the ratio of the maximum to the minimum value of a variable. In digital imaging, dynamic range represents the ratio between the maximum and minimum brightness values ​​within the displayable range of an image. For real-world natural scenes, the dynamic range is typically around 10. -3 Up to 10 6 Within this range. Because this is a very large dynamic range, it is usually called high dynamic range (HDR). In contrast to high dynamic range, the dynamic range of a regular image is called low dynamic range (LDR) or standard dynamic range (SDR). Therefore, it can be understood that the imaging process of a digital camera is actually a mapping from the high dynamic range of the real world to the low dynamic range of the photograph. The larger the dynamic range of an image, the more scene details it displays, the richer the brightness levels, and the more realistic the visual effect. Traditional digital images typically use one byte, or 8 bits, to store a pixel value, while high dynamic range uses multiple bytes of floating-point numbers to store a pixel value, thus enabling the representation of the high dynamic range of natural scenes.

[0083] Pose information can include position and attitude. Position can include (x, y, z) coordinates, and attitude can include angular deflections around the three coordinate axes: yaw, pitch, and roll. A pose including (x, y, z) coordinates and yaw, pitch, and roll angles can also be called a 6-degree-of-freedom (DoF) pose.

[0084] Lighting information for a specific location in a real-world scene can be obtained primarily through the following three methods:

[0085] Method 1: Obtaining illumination information at specific locations by adding optical probe spheres. Specifically, mirror sphere probes and highlight sphere probes are placed at specific locations in a real-world scene. These probes are used to acquire illumination information with different dynamic ranges and frequencies. Based on illuminance principles, image processing techniques are used to extract the direction and intensity of the main light source, ambient light information, and environmental map information from the optical probe spheres, thereby obtaining the illumination information for that specific location. However, this method requires optical probe spheres, which causes significant inconvenience in actual operation and results in a poor user experience.

[0086] Method 2: Acquire an image from the current field of view using an electronic device, and then use image processing methods to extract the location of light sources and calculate their intensity. However, this method can lead to significant errors in the extracted location and intensity information when the light source is not clearly visible in a certain area of ​​the image. Furthermore, the lighting information extracted in this way is LDR lighting information, which cannot guarantee realistic lighting rendering effects for virtual objects.

[0087] Method 3: Acquire images from the current field of view using electronic devices, and then use a deep learning network to estimate the lighting information of the current scene. However, since images with a limited field of view are insufficient to solve for global illumination in the spatial scene, the extracted lighting information will have large errors and poor robustness.

[0088] In summary, it is difficult to accurately estimate lighting information in a real scene using only a single image. Therefore, this application provides a lighting estimation method as described below to improve the accuracy of lighting estimation, thereby improving the lighting rendering effect of virtual objects, i.e., a more realistic lighting rendering effect.

[0089] Figure 1 This is a schematic diagram of an illumination estimation system provided in an embodiment of this application, as shown below. Figure 1 As shown, the illumination estimation method provided in this application embodiment can be applied to this system. Specifically, the system may include multiple electronic devices and a server.

[0090] The multiple electronic devices may include a first electronic device and one or more second electronic devices. Figure 1 (Taking two second electronic devices as an example), the one or more second electronic devices are several electronic devices other than the first electronic device. Each electronic device can communicate with the server. For example, any electronic device can communicate with the server via wireless-fidelity (WiFi), Bluetooth, or 2G / 3G / 4G / 5G cellular communication. It should be understood that other communication methods, including future communication methods such as 6G, can also be used between the server and the electronic devices. This application embodiment does not specifically limit this. It should be noted that "one or more second electronic devices" in this application embodiment is only used to refer to electronic devices other than the first electronic device, but does not limit whether the multiple electronic devices are of the same type.

[0091] The aforementioned electronic device (e.g., the first electronic device or the second electronic device) can be various types of devices equipped with a camera and display components. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, video recorder, or other terminal device. Figure 1(Taking a mobile phone as an example, electronic devices can also be devices used for virtual scene interaction, including VR glasses, AR devices, MR interactive devices, etc. Electronic devices can also be wearable electronic devices such as smartwatches and smart bracelets, and can be devices mounted on vehicles, autonomous vehicles, drones, industrial robots, etc. Electronic devices can also be smart home devices such as smart screens. This application does not specifically limit the specific form of the electronic device.)

[0092] In addition, the aforementioned electronic equipment may also be referred to as user equipment (UE), subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, terminal equipment, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handheld device, user agent, mobile client, client, or any other suitable term.

[0093] The aforementioned server can be one or more physical servers. Figure 1 (Taking a physical server as an example) It can also be a computer cluster, a virtual machine or cloud server in a cloud computing scenario, and so on.

[0094] In this embodiment, the electronic device can install virtual scene applications (APPs) such as VR, AR, or MR applications, and can run VR, AR, or MR applications based on user operations (e.g., clicking, touching, swiping, shaking, voice control, etc.). The electronic device can collect visual information of any object in the environment through sensors, and then display virtual objects on the display component based on the collected visual information. These virtual objects can be virtual objects in VR, AR, or MR scenes (i.e., objects in the virtual environment).

[0095] In this embodiment, the electronic device can install navigation, detection, control, and game interaction applications, and run the corresponding applications based on user control or preset programs. The electronic device can perform path planning, object detection, vehicle control, and other applications based on its own pose information and other state information in the current environment.

[0096] It should be noted that, in the embodiments of this application, the virtual scene application in the electronic device can be an application built into the electronic device itself, or an application provided by a third-party service provider that is installed by the user, and there is no specific limitation on this.

[0097] In this embodiment, the electronic device may also be configured with a simultaneous localization and mapping (SLAM) system. The SLAM system can create a map in a completely unknown environment and use the map for localization, pose (position and attitude) determination, navigation, etc. In this embodiment, the map created by the SLAM system is referred to as a SLAM map. A SLAM map can be understood as a map drawn by the SLAM system based on environmental information collected by the acquisition device. The acquisition device may include a visual information acquisition device and an inertial measurement unit (IMU) in the electronic device. The visual information acquisition device may include, for example, a camera, a depth camera, etc., and the IMU may include, for example, sensors such as a gyroscope, an accelerometer, etc.

[0098] The first electronic device in this application embodiment can capture multiple frames of first images of the scene where the first electronic device is located at different exposure parameters in a first instant, and send the multiple frames of first images to a server. The server executes the illumination estimation method of this application embodiment to process the multiple frames of first images to obtain a high dynamic range panoramic image of the scene where the first electronic device is located. This high dynamic range panoramic image is used to extract the illumination information of the scene where the first electronic device is located at the first instant, and this illumination information is used to render virtual objects displayed on the first electronic device. This is beneficial to enhancing the realism of virtual object rendering.

[0099] In some embodiments, the aforementioned plurality of second electronic devices and the first electronic device can share data in the cloud. For example, any second electronic device or the first electronic device can capture one or more frames of images and store them in a preset image library on a server. These images can serve as prior information for illumination estimation, thereby improving the accuracy of the illumination estimation. Optionally, the one or more frames of images can be images under conditions of a small field of view (i.e., non-panoramic).

[0100] It should be noted that the first electronic device can also process multiple frames of the first image to obtain a high dynamic range panoramic image of the scene where the first electronic device is located.

[0101] Figure 2 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application, such as... Figure 2 As shown, electronic device 20 can be Figure 1 The illustrated embodiment includes at least one of a first electronic device and one or more second electronic devices. It should be understood that... Figure 2 The structure shown does not constitute a specific limitation on the electronic device 20. In other embodiments of this application, the electronic device 20 may include... Figure 2The structure shown may have more or fewer components, or combine some components, or split some components, or have different component arrangements. Figure 2 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0102] Electronic device 20 may include: a chip 21, a memory 22 (one or more computer-readable storage media), a user interface 23, a display component 24, a camera 25, a sensor 26, a positioning module 27 for device positioning, and a transceiver 28 for communication. These components may communicate with each other via one or more buses 29.

[0103] Chip 21 may integrate one or more processors 211, a clock module 212, and a power management module 213. The clock module 212 integrated in chip 21 primarily provides the timer required for data transmission and timing control of processor 211, enabling clock functions for these operations. Processor 211 can perform calculations and generate operation control signals based on instruction opcodes and timing signals, thus controlling instruction fetching and execution. The power management module 213 integrated in chip 21 primarily provides a stable and highly accurate voltage for chip 21 and other components of electronic device 20.

[0104] Processor 211, also known as a central processing unit (CPU), may specifically include one or more processing units. For example, processor 211 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0105] In one possible implementation, the processor 211 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0106] The memory 22 can be connected to the processor 211 via bus 29, or coupled to the processor 311, for storing various software programs and / or multiple sets of instructions. The memory 22 may include high-speed random access memory (e.g., cache memory) or non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 22 can store operating systems, such as embedded operating systems like Android, iOS, Windows, or Linux. The memory 22 can also store data, such as image data, point cloud data, 3D map data, pose data, coordinate system transformation information, map update information, etc. The memory 22 can also store computer-executable program code, including instructions, such as communication program instructions, SLAM system-related program instructions, etc. The memory 22 can also store one or more application programs, such as AR / VR / MR virtual scene applications, 3D map applications, image management applications, navigation and control applications, etc. The memory 22 can also store user interface programs, which can display the content of the application, such as virtual objects in virtual scenes such as AR / VR / MR, in a realistic way through a graphical user interface and present it through the display component 24, and realize the user's control operation on the application through input controls such as menus, dialog boxes and buttons.

[0107] User interface 23 may be, for example, a touch panel that can detect user operation commands on it, or user interface 23 may be, for example, a keypad, physical buttons or a mouse.

[0108] Electronic device 20 may include one or more display components 24. Electronic device 20 can achieve display functions through the display components 24, the graphics processing unit (GPU) in chip 21, and the application processor (AP). The GPU is a microprocessor that performs image processing; it connects the display components 24 and the application processor, and performs mathematical and geometric calculations for graphics rendering. The display components 24 can display the interface content output by electronic device 20, such as images and videos from virtual scenes like AR / VR / MR. The interface content may include the interface of a running application and system-level menus, and can specifically consist of the following interface elements: input interface elements, such as buttons, text input boxes, scroll bars, and menus; and output interface elements, such as windows, labels, images, videos, and animations.

[0109] Display component 24 can be a display panel, lenses (e.g., VR glasses), projection screen, etc. The display panel can also be called a display screen, for example, it can be a touch screen, flexible screen, curved screen, etc., or other optical components. It should be understood that the display screen of the electronic device in the embodiments of this application can be a touch screen, flexible screen, curved screen, or other form of screen; that is, the display screen of the electronic device has the function of displaying images, and the specific material and shape of the display screen are not specifically limited.

[0110] For example, when the display component 24 includes a display panel, the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. Furthermore, in one possible implementation, the touch panel in the user interface 23 and the display panel in the display component 24 can be coupled together. For example, the touch panel can be located below the display panel, and the touch panel is used to detect the touch pressure applied to the display panel when the user inputs a touch operation (e.g., click, swipe, touch, etc.) through the display panel. The display panel is used for content display.

[0111] Camera 25 can be a monocular camera, a binocular camera, or a depth camera, used to capture / record the environment to obtain images / videos. The images / videos captured by camera 25 can be used as input data for a SLAM system, or displayed as images / videos via display component 24.

[0112] In one possible implementation, the camera 25 can also be regarded as a sensor. The images captured by the camera 25 can be in IMG format or other format types, and this application embodiment does not specifically limit them.

[0113] Sensor 26 can be used to collect data related to changes in the state of electronic device 20 (e.g., rotation, oscillation, movement, jitter, etc.). Sensor 26 may include one or more sensors, such as an inertial measurement unit (IMU), a time-of-flight (TOF) sensor, etc. The IMU may include sensors such as a gyroscope and an accelerometer. The gyroscope is used to measure the angular velocity of the electronic device during movement, and the accelerometer is used to measure the acceleration of the electronic device during movement. The TOF sensor may include a light emitter and a light receiver. The light emitter is used to emit light outward, such as laser light, infrared light, radar waves, etc., and the light receiver is used to detect reflected light, such as reflected laser light, infrared light, radar waves, etc.

[0114] It should be noted that sensor 26 may also include other sensors, such as inertial sensors, barometers, magnetometers, wheel speedometers, etc., but this application embodiment does not specifically limit them.

[0115] The positioning module 27 is used to achieve physical positioning of the electronic device 20, for example, to obtain the initial position of the electronic device 20. The positioning module 27 may include one or more of a WiFi positioning module, a Bluetooth positioning module, a base station positioning module, and a satellite positioning module. The satellite positioning module may be equipped with a Global Navigation Satellite System (GNSS) to assist positioning; GNSS is not limited to the BeiDou system, the Global Positioning System (GPS) system, the GLONASS system, or the Galileo satellite navigation system.

[0116] Transceiver 28 is used to enable communication between electronic device 20 and other devices (e.g., servers, other electronic devices, etc.). Transceiver 28 integrates a transmitter and a receiver for transmitting and receiving radio frequency (RF) signals, respectively. In specific implementations, transceiver 28 includes, but is not limited to: an antenna system, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec (CODEC) chip, a subscriber identification module (SIM) card, and storage media, etc. In one possible implementation, transceiver 28 can also be implemented on a separate chip. Transceiver 28 supports at least one data network communication method among 2G / 3G / 4G / 5G, and / or supports at least one of the following short-range wireless communication methods: Bluetooth (BT) communication, Wireless Fidelity (WiFi) communication, Near Field Communication (NFC), Infrared (IR) wireless communication, Ultra Wide Bandwidth (UWB) communication, and ZigBee communication.

[0117] In this embodiment, the processor 211 executes various functional applications and data processing of the electronic device 20 by running program code stored in the memory 22. For example, it executes... Figure 5 The steps in the method shown in the embodiment, or the execution of such... Figure 7 Functions on the electronic device side in the embodiment.

[0118] Figure 3 This is a schematic diagram of a server 30 provided in an embodiment of this application, such as... Figure 3 As shown, server 30 can be Figure 1 The server in the illustrated embodiment. Server 30 includes a processor 301, a memory 302 (one or more computer-readable storage media), and a transceiver 303. These components can communicate with each other via one or more buses 304.

[0119] Processor 301 can be one or more CPUs. If processor 301 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0120] The memory 302 can be connected to the processor 301 via bus 304, or it can be coupled together with the processor 301 to store various program codes and / or multiple sets of instructions, as well as data (e.g., map data, pose data, etc.). In specific implementations, the memory 302 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), etc.

[0121] The transceiver 303 mainly integrates a receiver and a transmitter, wherein the receiver is used to receive data (e.g., requests, images, etc.) sent by electronic devices, and the transmitter is used to send data (e.g., map data, pose data, etc.) to electronic devices.

[0122] It should be understood that Figure 3 The server 30 shown is only an example provided by the embodiments of this application. The server 30 may also have more components than shown, and the embodiments of this application do not specifically limit it.

[0123] In this embodiment, the processor 301 executes various functional applications and data processing of the server 30 by running program code stored in the memory 302. For example, it executes... Figure 6 The steps in the method shown in the embodiment, or the execution Figure 7 The server-side functionality in this embodiment.

[0124] The term "coupled" as used in the embodiments of this application means a direct connection or a connection through one or more intermediary components or circuits.

[0125] It should be understood that the server 30 described above is only one example provided in the embodiments of this application. In specific implementations, the server 30 may have more components than shown in the figure.

[0126] Figure 4 This is a structural block diagram of an illumination estimation system provided in an embodiment of this application. See also... Figure 4 As shown, the system may include an electronic device 100 and a server 200, wherein the electronic device 100 may be the aforementioned Figure 1 The first electronic device in the system. The electronic device 100 is configured with an image acquisition module 101, a network transmission module 102, a computing module 103, a rendering module 104, and a display module 105. The image acquisition module 101, network transmission module 102, computing module 103, rendering module 104, and display module 105 can exist in the form of software code. In one specific embodiment, the data / programs of these functional modules can be stored in the memory of the electronic device and can run on the processor of the electronic device.

[0127] Image acquisition module 101 is configured to utilize, for example Figure 2 The camera 25 shown is used to obtain multiple first images of the current scene. The exposure parameters of the multiple first images are different, and the multiple first images are transmitted to the network transmission module 102.

[0128] Network transmission module 102 can utilize, for example Figure 2 The transceiver 28 shown enables communication with the server 200. Specifically, the network transmission module 102 is configured to upload images to the server 200, download images from the server 200, and encode and decode images to ensure efficient communication between the electronic device 100 and the server 200.

[0129] The calculation module 103 and the rendering module 104 are configured to utilize, for example Figure 2 The processor 211 shown performs network inference, solves for lighting parameters, and performs rendering operations based on the material and geometric information of the 3D model. Among these, network inference includes, but is not limited to, inference using deep learning models or style transfer networks.

[0130] The display module 105 is configured to utilize, for example Figure 2 The user interface 23 shown implements the detection and acquisition of user operations, and utilizes, for example, Figure 2 The display component 24 shown displays images / videos / virtual objects, such as AR / VR / MR application content. For example, the rendering results generated by the calculation module 103 and the rendering module 104 can be presented to the user.

[0131] In a specific embodiment, the functional modules in the electronic device 100 can cooperate with each other to perform, for example... Figure 5The steps in the method shown in the embodiment, or the execution of such... Figure 7 Functions on the electronic device side in the embodiment.

[0132] Configure a network transmission module 201, a storage module 202, and an AI computing module 203 in server 200. These modules can exist in the form of software code. In a specific implementation, the data / programs of these functional modules can be stored in, for example... Figure 3 The memory 302 shown can operate as follows: Figure 3 The processor 301 shown.

[0133] Network transmission module 201 can utilize, for example Figure 3 The transceiver 303 shown enables communication with the electronic device 100. For example, the network transmission module 201 can receive images from the electronic device 100, such as multiple frames of a first image. The network transmission module 201 can also provide high dynamic range panoramic images to the electronic device 100.

[0134] Storage module 202 can be used to store, maintain, and update a preset image library.

[0135] The AI ​​computing module 203 can be configured to perform illumination estimation based on multiple frames of the first image to improve the accuracy of illumination estimation.

[0136] In a specific embodiment, the functional modules in server 200 can cooperate with each other to perform, for example... Figure 6 The steps in the method shown in the embodiment, or the execution Figure 7 The server-side functionality in this embodiment.

[0137] Based on the above description, the following are some lighting estimation methods provided by embodiments of this application, which are used to provide more realistic virtual object display effects. For convenience, the various method embodiments described below are all described as a combination of a series of action steps. However, those skilled in the art should understand that the specific implementation of the technical solution of this application is not limited by the order of the described series of action steps.

[0138] Figure 5 This is a flowchart illustrating an illumination estimation method provided in an embodiment of this application. In some possible embodiments, the method can be applied to a first electronic device having a display component and a camera. In other words, the method can be... Figure 1 or Figure 2 or Figure 4 The method may be executed by the first electronic device shown, or by an application built into the first electronic device for displaying virtual objects; however, this application embodiment does not specifically limit this. The method includes, but is not limited to, the following steps:

[0139] Step 501: The first operation performed by the user is detected.

[0140] In this embodiment, the user inputs a trigger operation on the first electronic device to display a virtual object. The trigger operation can be any operation, such as opening an AR application or activating the AR function of an application; this embodiment does not specifically limit this. The specific form of the trigger operation can be clicking, touching, swiping, shaking, voice control, etc.

[0141] Step 502: In response to the first operation performed by the user, capture multiple first images of the scene where the first electronic device is located at different exposure parameters at the first moment.

[0142] The "first time" can be a time period starting from the time the first operation is detected, and the length of this time period can be reasonably set according to requirements, such as 0.5s, 1s, etc. For example, the time length can be related to the acquisition frequency of the image acquisition module. In this embodiment, the first electronic device can acquire images of the current scene with different exposure parameters through the image acquisition module to obtain multiple frames of first images. The current scene can be the scene where the first electronic device is located at the first time. The exposure parameters corresponding to each frame of the first image are different. Different exposure parameters mean that each frame of the first image is acquired with different exposure parameters. The exposure of each frame of the first image is different, resulting in different levels of brightness.

[0143] The number of frames in the first multi-frame image can be 2, 3, 4 or more, and can be set reasonably according to needs.

[0144] In one implementation, the multiple first images may include one or more underexposed images, one or more correctly exposed images, and one or more overexposed images. For example, using three first images as an example, these three images may include one underexposed image, one correctly exposed image, and one overexposed image. Each of the three first images corresponds to different exposure parameters. The underexposed image will be generally darker and is mainly used to capture the bright parts of the current scene to ensure accurate exposure of those bright parts. The correctly exposed image will highlight the subject of the image to ensure clear texture of the main subject, but it may not highlight details in overly bright or dark areas of the current scene. The overexposed image will be generally brighter and is mainly used to capture details in the darker areas of the current scene to ensure correct exposure of those darker areas.

[0145] For example, exposure parameters may include one or more of aperture, shutter speed, or ISO.

[0146] In one example, in response to the first operation, on the one hand, the first electronic device can capture multiple frames of first images of the scene in which the first electronic device is located at different exposure parameters in a single moment; on the other hand, the first electronic device can display an application interface on a display component. For example, the application interface can present a correctly exposed image. This application could be an AR application installed on the first electronic device.

[0147] Step 503: Send multiple frames of the first image to the server. The multiple frames of the first image are used to obtain a high dynamic range panoramic image of the scene. The high dynamic range panoramic image is used to extract the lighting information of the scene at the first moment. The lighting information of the scene at the first moment is used to render the virtual object displayed on the first electronic device.

[0148] The different exposure levels of multiple first-frame images result in varying degrees of brightness within each image. These multiple first-frame images with different exposure levels can be used to generate a high dynamic range panoramic image (HDR) of the scene. The field of view of an HDR panoramic image can be 360 ​​degrees. Based on the HDR panoramic image, the lighting information of the scene where the first electronic device is located at a given moment can be extracted. This lighting information represents the panoramic lighting information of the scene. Rendering virtual objects based on this lighting information makes the displayed virtual objects more realistic, achieving seamless integration between virtual objects and the real scene.

[0149] In one implementation, a first electronic device can receive an HDR panoramic image sent by a server. The first electronic device can then extract the real-time lighting information of the scene in which it is located based on the HDR panoramic image to render virtual objects.

[0150] Another possible approach is for the server to extract the lighting information of the scene where the first electronic device is located at a specific time based on the HDR panoramic image, and send the lighting information of the scene where the first electronic device is located at that specific time to the first electronic device so that the first electronic device can render virtual objects based on this information.

[0151] Because the field of view of the camera of the first electronic device is limited, the field of view of each frame of the first image captured by the camera can be a small field of view, i.e., a non-panoramic image. The illumination estimation method of this application embodiment can generate a high dynamic range panoramic image of the scene based on multiple first images with different exposures, so as to achieve accurate estimation of the illumination information of the scene.

[0152] For example, a first electronic device or server can generate HDR local images and LDR local images based on multiple first images with different exposure levels. The field of view of the HDR local images is the same as that of the first images with different exposure levels. The field of view of the LDR local images is the same as that of the first images with different exposure levels. Multiple frames of second images matching the pose information of the first electronic device are obtained from the server. These second frames depict the same scene as the first images but are shot from different angles. Then, based on the multiple frames of second images and the LDR local images, an LDR panoramic image is obtained. Furthermore, the illumination information of the HDR local images is transferred to the LDR panoramic image to obtain the HDR panoramic image.

[0153] The multiple frames of second images can originate from the first electronic device or from one or more second electronic devices. When the multiple frames of second images include images from the second electronic devices, the embodiments of this application can implement a multi-device cloud collaborative illumination estimation method to improve the accuracy of illumination estimation.

[0154] In this embodiment, by detecting a first operation performed by the user, and in response to the user's first operation, multiple first frames of images of the scene where the first electronic device is located are captured at different exposure parameters in a first-time manner. These multiple first frames are then sent to the server. These multiple first frames are used to obtain a high dynamic range (HVR) panoramic image of the scene. This HVR panoramic image is used to extract the lighting information of the scene at the first-time exposure. This lighting information is then used to render virtual objects displayed on the first electronic device. This embodiment can generate a HVR panoramic image of the scene where the first electronic device is located based on multiple first images with different exposure levels, thereby achieving accurate estimation of the lighting information in the scene. Compared to estimating the lighting information in the scene using a single-frame image, the lighting information obtained based on the HVR panoramic image of the scene where the first electronic device is located is more accurate. Rendering virtual objects based on this lighting information can satisfy lighting consistency, making the displayed virtual objects more realistic and achieving seamless integration between virtual objects and the real scene.

[0155] Figure 6 This is a flowchart illustrating an illumination estimation method provided in an embodiment of this application. In some possible embodiments, the method can be applied to a server. In other words, the method can be... Figure 1 or Figure 3 or Figure 4 The server shown in the example executes the method, but this embodiment does not specifically limit the scope of the method. The method includes, but is not limited to, the following steps:

[0156] Step 601: Receive multiple first images sent by the first electronic device. The multiple first images are multiple images of the scene where the first electronic device is located, taken by the first electronic device at different exposure parameters at the first time.

[0157] For example, the server can receive multiple frames of the first image sent by the first electronic device through the aforementioned network transmission module. For a detailed explanation of the multiple frames of the first image, please refer to [link to relevant documentation]. Figure 5 The explanation of the multiple first images in step 502 of the illustrated embodiment will not be repeated here.

[0158] Step 602: Obtain the pose information of the first electronic device.

[0159] The pose information of the first electronic device can be the 6DoF pose of the first electronic device in the current scene.

[0160] In one implementation, the server can receive pose information sent by the first electronic device. For example, the first electronic device can obtain its own pose information using visual positioning technology and send it to the server.

[0161] Another possible implementation is that the server can obtain the pose information of the first electronic device based on the aforementioned multiple frames of the first image using visual positioning technology. For example, the server can generate a low dynamic range local image, i.e., an LDR local image, based on the multiple frames of the first image. Then, based on the LDR local image, the pose information of the first electronic device is determined using visual positioning technology. Of course, it is understood that the server can also determine the pose information of the first electronic device based on a single frame of the multiple frames of the first image, for example, a correctly exposed first image, using visual positioning technology. The embodiments of this application do not limit the method of obtaining the pose information of the first electronic device to the above exemplary description.

[0162] Step 603: Obtain multiple frames of second images that match the pose information of the first electronic device. The multiple frames of second images have the same scene as the multiple frames of first images but different shooting angles.

[0163] The exposure parameters for each frame of the first image are different, but the shooting angle is the same. Each frame of the second image has the same scene as any of the first images, but the shooting angle is different. These multiple second images are used to generate a panoramic image of the scene where the first electronic device is located. In some cases, the lighting conditions corresponding to each of the multiple second images are different from the lighting conditions corresponding to the first image. The lighting conditions corresponding to each of the multiple second images refer to the lighting conditions at the time each frame of the second image was captured.

[0164] For example, the server is equipped with a preset image library, which may include multiple frames of images taken by a first electronic device and / or at least one second electronic device. The server can match multiple frames of second images from the preset image library based on the pose information of the first electronic device.

[0165] Step 604: Based on multiple frames of the first image and multiple frames of the second image, a high dynamic range panoramic image of the scene is obtained. This high dynamic range panoramic image is used to extract the lighting information of the scene at the first moment. The lighting information of the scene at the first moment is used to render virtual objects displayed on electronic devices.

[0166] HDR panoramic images can be used to extract the real-time lighting information of the scene where the first electronic device is located. This lighting information is the panoramic lighting information of the scene. Rendering virtual objects based on this lighting information can make the displayed virtual objects more realistic and achieve seamless integration between virtual objects and the real scene.

[0167] For example, the server can generate an HDR local image based on multiple frames of the first image. Based on multiple frames of the second image, it can generate an LDR panoramic image. The illumination information from the HDR local image is then transferred to the LDR panoramic image to obtain an HDR panoramic image of the scene where the first electronic device is located.

[0168] The multi-frame first image can include one or more underexposed images, one or more correctly exposed images, and one or more overexposed images. The underexposed image is generally darker and is primarily used to capture the bright parts of the scene to ensure accurate exposure of those areas. The correctly exposed image highlights the subject, ensuring clear textures. The overexposed image is generally brighter and is primarily used to capture details in the darker parts of the scene to ensure proper exposure of those areas. By fusing multiple first images from the same shooting angle within the same scene, an HDR local image is obtained, thus restoring the brightness information of the real scene. This HDR local image can be an image in HDR format.

[0169] One possible approach is to convert the HDR local image into an 8-bit single-channel image using tone mapping after obtaining the HDR local image, thus obtaining an LDR local image. The lighting information of the LDR local image is then transferred to multiple frames of second images to obtain multiple frames of third images. Based on these multiple frames of third images, an LDR panoramic image is generated. The multiple frames of second images depict the same scene as the multiple frames of first images but differ in shooting angle and lighting conditions. By transferring the lighting information of the LDR local images to the multiple frames of second images to obtain multiple frames of third images, the lighting conditions of the multiple frames of third images are unified to those of the LDR local images. Finally, the multiple frames of third images are stitched together, or stitched together and completed, to generate the LDR panoramic image.

[0170] Because the field of view of the camera of the first electronic device is limited, the field of view of each frame of the first image captured by the camera can be a small field of view, i.e., a non-panoramic image. The illumination estimation method of this application embodiment can generate HDR local images based on multiple first images with different exposure levels. An LDR panoramic image is generated based on multiple frames of second images. The illumination information of the HDR local images is transferred to the LDR panoramic image to obtain an HDR panoramic image of the scene where the first electronic device is located, thereby achieving accurate estimation of the illumination information of that scene.

[0171] Optionally, the server sends an HDR panoramic image to the first electronic device. The first electronic device can then extract the real-time lighting information of the scene in which it is located based on the HDR panoramic image to render virtual objects.

[0172] Optionally, the server can extract the lighting information of the scene where the first electronic device is located at a real time based on the HDR panoramic image, and send the lighting information of the scene where the first electronic device is located at a real time to the first electronic device so that the first electronic device can render virtual objects based on it.

[0173] In this embodiment, multiple frames of first images sent by a first electronic device are received. These first frames are images of the scene where the first electronic device is located, captured by the first electronic device at different exposure parameters at a single moment. Multiple frames of second images matching the pose information of the first electronic device are then acquired. These second frames depict the same scene as the first frames but are captured from different angles. Based on the first and second frames, a high dynamic range (HDL) panoramic image of the scene is obtained. This HDL panoramic image is used to extract the lighting information of the scene at the first moment. This lighting information is then used to render virtual objects displayed on the electronic device. This embodiment can generate a HDL panoramic image of the scene where the first electronic device is located based on multiple first images with different exposure levels, thereby achieving accurate estimation of lighting information in the scene. Compared to estimating lighting information in a scene using a single frame image, the lighting information obtained from the HDL panoramic image of the scene where the first electronic device is located is more accurate. Rendering virtual objects based on this lighting information can satisfy lighting consistency, making the displayed virtual objects more realistic and achieving seamless integration between virtual objects and the real scene.

[0174] Figure 7 This is a flowchart illustrating an illumination estimation method provided in an embodiment of this application. Figure 8 This is a schematic diagram illustrating the processing steps of an illumination estimation method provided in an embodiment of this application. Figure 9This is a schematic diagram of a user interface provided in an embodiment of this application. In some implementations, this method can be applied to a server and a first electronic device having a display component and a camera, thereby achieving end-to-cloud (i.e., the first electronic device side) collaborative illumination estimation to improve the accuracy of illumination estimation. Combined with... Figure 7 and Figure 8 As shown, the method includes, but is not limited to, the following steps:

[0175] Step 701: The first electronic device detects the first operation performed by the user.

[0176] Step 702: In response to the first operation, the first electronic device captures multiple first images of the scene where the first electronic device is located at different exposure parameters at the first moment.

[0177] For an explanation of steps 701 and 702, please refer to [link to relevant documentation]. Figure 5 Steps 501 and 502 of the illustrated embodiment.

[0178] Step 703: The first electronic device sends multiple frames of the first image to the server.

[0179] Optionally, the first electronic device may also display a prompt message to the user that multiple frames of images with different exposure parameters in the current scene will be captured and uploaded to the cloud.

[0180] One example, combined Figure 9 As shown in (a), an exemplary user interface of a first electronic device is provided. This user interface may include a current scene view 901, a prompt message 902, a confirmation button 903, and a denial button 904. The current scene view 901 may be a view of the current scene captured by the first electronic device through its own camera. For example, the current scene view 901 may specifically be a correctly exposed image. The prompt message 902 prompts the user to collect multiple frames of images with different exposure parameters in the current scene and upload the collected multiple frames to the cloud. The confirmation button 903 receives confirmation from the user allowing the collection of multiple frames of images with different exposure parameters in the current scene and the uploading of the collected multiple frames to the cloud. The denial button 904 receives confirmation from the user disallowing the collection of multiple frames of images with different exposure parameters in the current scene and the uploading of the collected multiple frames to the cloud. When the first electronic device detects that the user has clicked the confirmation button 903, it can execute step 703 to upload the multiple frames of images with different exposure parameters to the server. It then performs illumination estimation through the steps of the following embodiments, and renders virtual objects based on this estimation to display as shown. Figure 9 The user interface shown in (b) may include the current scene screen 901 and virtual object 905.

[0181] It should be noted that, Figure 9 The user interface shown may also include other interface elements, and the embodiments of this application are not limited to the above examples. For example, the first electronic device may set up a monitoring mechanism to detect the network connection status at a certain period of time. When an abnormal network connection or unstable network signal is detected, a pop-up window may be displayed on the user interface, indicating to the user that the current network signal is unstable and that the estimation of the current scene's lighting conditions may have some deviation.

[0182] Let's take the first image of the three frames as an example to illustrate this, combined with... Figure 8 As shown, the first electronic device sends three first images to the server, which are overexposed, correctly exposed, and underexposed first images, respectively.

[0183] Step 704: The server generates HDR local images and LDR local images based on the multiple frames of the first image.

[0184] The embodiments of this application can synthesize HDR local images and LDR local images based on multiple frames of first images with different exposure parameters.

[0185] Because ordinary cameras capture images with only 8 bits per color channel, the limited number of channels means that the pixel value range for each channel is between 0 and 255, thus limiting the dynamic range of the captured images. The dynamic range in the real world is much larger. To ensure that virtual objects match the lighting in the real world, it is necessary to estimate the high dynamic range lighting information of the real scene. This is achieved by adjusting the camera's exposure parameters and sequentially capturing an underexposed first image, a correctly exposed first image, and an overexposed first image. Then, a multi-exposure image fusion algorithm is used to generate an HDR local image. Optionally, to reduce artifacts during HDR local image synthesis, the AlignMTB algorithm can be used to align multiple frames of the first images with different exposure parameters. Then, the camera's response function is extracted, and the image intensity is made linear to synthesize the multiple frames of the first images, generating an HDR local image in HDR format. By synthesizing HDR local images from multiple frames of the first images with different exposure parameters, the brightness information of the real scene can be recovered. The HDR local image is then converted into an 8-bit single-channel image using a tone mapping method, thus generating an LDR local image. The LDR local image can retain as much detail as possible.

[0186] For example, let's take the first three frames as an example for further illustration, such as... Figure 8 As shown, the server can synthesize multiple frames of the first image into an HDR local image and an LDR local image.

[0187] Step 705: The server obtains the pose information of the first electronic device based on the LDR local image.

[0188] The server can use the visual positioning system (VPS) visual positioning technology to obtain the 6DOF pose of the first electronic device in the current scene based on the synthesized LDR local image. The positioning accuracy of the 6DOF pose can be at the centimeter level and does not rely on the GPS system.

[0189] Step 706: The server obtains multiple frames of second images from a preset image library based on the pose information of the first electronic device.

[0190] The server can search for the best matching N frames of images in a preset image library based on the pose information of the first electronic device, where N is any positive integer greater than 1. For example, it can obtain images such as... Figure 8 The second image 1, the second image 2, ..., the second image N are shown.

[0191] Optionally, the server can also update the preset image library, that is, store the LDR partial image or the first correctly exposed image into the preset image library.

[0192] Step 707: The server transfers the illumination information of the LDR local image to multiple frames of the second image to obtain multiple frames of the third image.

[0193] The server can use a style transfer network to transfer the illumination information of a local LDR image to multiple frames of second images, resulting in multiple frames of third images. For example, the local LDR image and one frame of second image can be output to the style transfer network to obtain one frame of third image output by the network. Using the same method, each frame of second image is sequentially input into the style transfer network to obtain each frame of third image. For example, it can obtain images like... Figure 8 The third image 1, third image 2, ..., third image N are shown.

[0194] A style transfer network is used to unify the illumination information of multiple frames of the second image to the illumination conditions in the LDR local image.

[0195] For example, Figure 10 A schematic diagram of a style transfer network provided in an embodiment of this application is shown below. Figure 10As shown, the style transfer network may include a generator (Generator_1). The input of the generator (Generator_1) is a second image and an LDR local image. The output of the generator (Generator_1) is a third image. The generator (Generator_1) is used to transfer the illumination information of the LDR local image to the second image to obtain the third image.

[0196] Step 708: The server stitches together multiple frames of the third image to obtain the LDR incomplete image.

[0197] The server can use image stitching technology to obtain incomplete panoramic images, i.e., LDR (Low-Resolution Image) incomplete images. For example, to obtain images like... Figure 8 The image shown is an incomplete LDR image.

[0198] Step 709: The server performs image completion processing on the incomplete LDR image to generate an LDR panoramic image.

[0199] The server can complete missing data in incomplete LDR images to obtain LDR panoramic images with clear textures and reasonable content. For example, it can obtain images like... Figure 8 The LDR panoramic image shown.

[0200] For example, the server can input an incomplete LDR image into a deep learning model to obtain an LDR panoramic image output by the deep learning model. This deep learning model has the ability to complete missing data. This deep learning model is trained using training data that includes the incomplete image and its corresponding complete image.

[0201] Step 710: The server transfers the illumination information of the HDR local image to the LDR panoramic image to obtain the HDR panoramic image.

[0202] The server can use a style transfer network to transfer lighting information from an HDR local image to an LDR panoramic image. For example, both the HDR local image and the LDR panoramic image can be output to the style transfer network to obtain the HDR panoramic image output by the network. For instance, it can obtain... Figure 8 The HDR panoramic image shown.

[0203] Style transfer networks are used to unify the lighting information of LDR panoramic images with clear textures and reasonable content to the lighting conditions in HDR local images.

[0204] It should be noted that the style transfer network used in step 710 and the style transfer network used in step 707 can be the same network, for example, such as Figure 10 The generator shown can also be a different network, which can be configured appropriately according to requirements.

[0205] Step 711: The server sends an HDR panoramic image to the first electronic device.

[0206] HDR panoramic images can more accurately reflect the lighting information in a real scene.

[0207] It should be noted that, in some embodiments, the server can extract lighting information from the HDR panoramic image and send the lighting information to the first electronic device, enabling the first electronic device to render virtual objects based on the lighting information. This application uses the example of the server sending an HDR panoramic image to the first electronic device for illustration.

[0208] Step 712: The first electronic device renders a virtual object based on the HDR panoramic image and displays the rendered virtual object.

[0209] Taking the extraction of illumination information by the first electronic device as an example, the first electronic device can perform at least one of the following:

[0210] (1) Extract the direction of the main light source in the HDR panoramic image. This direction of the main light source can be used to project directional light onto virtual objects to produce rendering effects such as shadows and highlights.

[0211] (2) Extract the spherical harmonic coefficients from the HDR panoramic image. These spherical harmonic coefficients are used to project ambient light onto virtual objects, which can make the surface texture details of virtual objects more layered and improve the fusion effect between virtual objects and the environment.

[0212] (3) Extract the environment texture map from the HDR panoramic image. This environment texture map is used to make the virtual object with specular reflective material reflect the texture of the current scene.

[0213] Real-world lighting conditions are extremely complex. This application's embodiments utilize spherical harmonic lighting to extract lighting information from HDR panoramic images, thereby producing high-quality real-time rendering and shadow effects. Spherical harmonic lighting is a rendering technique used to achieve hyper-realistic lighting; it is based on a simplification of complex lighting equations using spherical harmonic basis theory.

[0214] L(p,ω o )=Le(p,ω o )+∫f r (p,ω i ,ω o )L i (p,ω i )G(p,p')V(p,p')dω i (1)

[0215] The classic global illumination formula is described, as shown in formula (1) above. The illumination equation is considered as the sum of the self-emission at a point and the integral on the sphere near that point. From this formula, it can be seen that the brightness of the reflected light at a point on the surface of an object is determined by the self-emission at that point, the energy of other incident rays during ray tracing, the bidirectional reflectance distribution function (BRDF), occlusion relationships, and geometric relationships. The detailed meanings of the formula symbols are as follows:

[0216] L(p,ω o ) represents the reflection from point p on the surface of the object to ω. o Light intensity in a given direction.

[0217] Le(p,ω o ) indicates that this point is at ω o Directional self-illumination.

[0218] f r (p,ω i ,ω o The BRDF at that point represents the propagation distribution of light rays after they pass through the incident direction.

[0219] L i (p,ω i ) indicates that during ray tracing, the ray travels from other positions along ω i The intensity of light reflected to that point.

[0220] G(p,p') represents the intersection of point L and point L. i (p,ω i The geometric relationship function between the two determines the amount of energy transferred.

[0221] V(p,p') represents the visibility function between p and p', which determines the occlusion relationship.

[0222] The brightness of a point in space is determined by the incident light from all directions in space. Due to the complexity of the lighting equation, the current hardware conditions at the edge are insufficient to calculate the lighting information of a specific point in space in real time according to the formula. Instead, we can use spherical harmonic transformation and use the spherical harmonic basis as the basis function to fit the equation and obtain the spherical harmonic coefficients to achieve an approximate calculation of the lighting information in the scene. The formula for calculating the spherical harmonic coefficients (2) is as follows:

[0223]

[0224] The Monte Carlo integration method is used to calculate the expression for the spherical harmonic coefficients, where c i f(s) represents the i-th component in the spherical harmonic coefficients, N represents the number of sampling points, and f(s) represents the i-th component in the spherical harmonic coefficients. j) represents the intensity value of a random sampling point, y i These are spherical harmonic basis functions. The spherical harmonic coefficients of the corresponding basis functions are obtained by projecting the original function. For example, i can be any positive integer less than or equal to 9 or 17.

[0225] By using spherical harmonic transformation, the lighting information contained in HDR panoramic images is represented as spherical harmonic coefficients. Based on the spherical harmonic coefficients, the intensity of the main light source and the intensity of the ambient light in the scene are extracted. The lighting information in the real scene is then superimposed onto the virtual objects to enhance the realism of the virtual objects.

[0226] In this embodiment, LDR and HDR local images are generated based on multiple frames of first images with different exposure parameters. The pose information of the first electronic device is obtained from the LDR local images. Based on the pose information, the best-matching multi-frame second images are searched from a preset image library. The illumination information from the LDR local images is transferred to the multi-frame second images to obtain multi-frame third images. Image stitching technology is used to stitch the multi-frame third images together to obtain an incomplete panoramic image, i.e., the LDR incomplete image. The LDR incomplete image is then completed to obtain an LDR panoramic image with clear texture and reasonable content. The illumination information from the HDR local images is transferred to the LDR panoramic images to obtain an HDR panoramic image. Compared to estimating the illumination information in a scene using a single-frame image, the illumination information obtained from the high dynamic range panoramic image of the scene where the first electronic device is located is more accurate in this embodiment. Rendering virtual objects based on this illumination information can satisfy illumination consistency, making the displayed virtual objects more realistic and achieving seamless integration between virtual objects and the real scene.

[0227] It should be noted that the above embodiment uses the example of a first electronic device sending multiple frames of first images to a server, and the server processing the multiple frames of first images to obtain an HDR panoramic image of the scene where the first electronic device is located. This application provides other possible illumination estimation methods. For example, the first electronic device detects a first operation performed by a user and, in response to the user's first operation, captures multiple frames of first images of the scene where the first electronic device is located at different exposure parameters in a first instant. The first electronic device can then obtain multiple frames of second images from the server that match the pose information of the first electronic device, and further obtain an HDR panoramic image of the scene where the first electronic device is located based on the multiple frames of first images and the multiple frames of second images. In other words, the above... Figure 7The steps performed by the server in the illustrated embodiment can also be performed by the first electronic device, such as using a style transfer network to transfer lighting information or using a deep learning model for image completion. This also allows for accurate estimation of lighting information in the scene. Compared to estimating lighting information in a scene using a single-frame image, the lighting information obtained from a high dynamic range panoramic image of the scene where the first electronic device is located is more accurate. Rendering virtual objects based on this lighting information can satisfy lighting consistency, making the displayed virtual objects more realistic and achieving seamless integration between virtual objects and the real scene.

[0228] The style transfer network involved in the above embodiments can be trained using the methods described in the following embodiments.

[0229] First, the training dataset for the style transfer network will be explained.

[0230] The training dataset for a style transfer network can include multiple sets of training images from different scenes. Each set of training images can consist of multiple frames of training images from the same scene under different lighting conditions. The field of view of multiple frames of training images from the same scene under different lighting conditions can be the same or different. The shooting angle of multiple frames of training images from the same scene under the same lighting conditions can be different.

[0231] In this embodiment, a motion control 001 is added to a scene and a camera is bound to it. The camera's exposure parameters can be controlled as needed. It is mainly used to randomly walk within the scene and capture multiple frames of images with different exposure parameters. A control 002 is added to the scene and a panoramic camera is bound to it. Its movement trajectory is constrained by the control 001. The panoramic camera is mainly responsible for acquiring HDR panoramic images at specified locations within the scene. After the motion control 001 randomly walks within the scene a certain number of times, it changes the lighting in the scene in a somewhat random manner, such as randomly changing the position, intensity, and number of light sources. This method can acquire training images under different lighting conditions in the same scene. After acquiring a certain amount of training images, the above steps are repeated continuously by changing the scene to generate more training images. The HDR panoramic images acquired by the panoramic camera can be used as training data for a style transfer network that transfers the lighting information of HDR local images to LDR panoramic images; that is, it serves as the ground truth corresponding to the HDR panoramic images output by the style transfer network during training.

[0232] The style transfer network can then be trained using the training dataset.

[0233] The style transfer network is responsible for transferring lighting information from a local LDR or HDR image to a specified image (e.g., a second image, an LDR panoramic image). This embodiment uses a recurrent generative adversarial network (GAN) to train the style transfer network.

[0234] The formula for the lighting shading model can be expressed as follows (3).

[0235]

[0236] Where X1 represents the brightness information of a point in the scene, X represents the texture image of a point in the scene as X, and the subscript 1 indicates that the current lighting condition is Light1. material This represents the material properties of a point in the scene. The brightness of a point in the scene is determined by both its material properties and lighting parameters. The shooting angle of X1 is different from that of Y1. Based on AI technology, a data-driven approach is used to model the relationship between the brightness of a point in the scene and the material properties of objects and lighting conditions. For example, ... Figure 11 The training structure shown is used to achieve style transfer under lighting conditions through training.

[0237] Figure 11 This is a schematic diagram of the training structure of the style transfer network provided in the embodiments of this application, as shown below. Figure 11 As shown, the algorithm includes two generators, Generator_1 and Generator_2, and two discriminators, Discriminator_1 and Discriminator_2. Generator_1 takes x1 and y2 as inputs and its function is to transfer the lighting information light1 from x1 to y to obtain output y1', and to transfer the lighting information light2 from y2 to x to obtain output x2'. Generator_2 takes x2' and y1' as inputs and its function is to transfer the lighting information light2' from x2' to y to obtain output y2', and to transfer the lighting information light1' from y1' to x to obtain output x1'.

[0238] Generators Generator_1 and Generator_2 are passed through a cycle-consistent loss function (e.g., ... Figure 11 The training of the two networks is constrained by the Cycle_Consit_loss to make the output of Generator_2 as close as possible to the input of Generator_1, by calculating the L1 loss function (e.g., ...). Figure 11 The L1_loss in the dataset supervises the outputs of Generator_1 and Generator_2.

[0239] The discriminator_1 takes x2 and x2' or y1 and y1' as input and is responsible for distinguishing the output generated by the generator_1 from its corresponding ground truth (x2 or y1). The loss function is a binary cross-entropy loss function (e.g., ...). Figure 11 The distance between two distributions is calculated using `gan_loss` and `disc_loss`. The discriminator `Discriminator_2` takes y2 and y2' or x1 and x1' as input and is responsible for distinguishing the output of the generator `Generator_2` from its corresponding ground truth (y2 or x1). The discriminator and generator work together as a game to improve their capabilities; the generator aims to produce data that closely approximates real lighting conditions, while the discriminator strives to enhance its ability to differentiate between real and fake data.

[0240] Based on the above training dataset, during training, training images under different lighting conditions at the same field of view in a certain scene, training images under the same lighting conditions at different field of view, and training images under different lighting conditions at different field of view are randomly selected and used alternately to train the network, so that the style transfer network can achieve style transfer under different lighting conditions.

[0241] This application embodiment can divide the first electronic device or server into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into a processing module. The integrated module can be implemented in hardware or as a software functional module. For example, the first electronic device can be divided into a transceiver module and a processing module for executing... Figure 5 The illustrated embodiment or Figure 7 The steps performed by the first electronic device in the illustrated embodiment. The server can be divided into a transceiver module and a processing module for performing... Figure 6 The illustrated embodiment or Figure 7 The steps performed by the server in the illustrated embodiment.

[0242] It should be noted that the module division in this embodiment is illustrative and is only a logical functional division. In actual implementation, there may be other division methods.

[0243] This application also provides a computer-readable storage medium storing computer software instructions. When the computer software instructions are run in a light estimation device, the light estimation device can execute the relevant method steps in the above embodiments to implement the methods in the above embodiments.

[0244] This application also provides a computer program product that, when run on a computer, causes the computer to execute the relevant method steps in the above embodiments to implement the methods in the above embodiments.

[0245] In this application, the first electronic device, server, computer storage medium, or computer program product provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0246] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0247] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. For example, the vehicle terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of modules or units may be electrical, mechanical, or other forms.

[0248] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0249] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0250] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program instructions, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0251] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for estimating illumination, characterized in that, The method includes: Receive multiple frames of first images sent by a first electronic device, wherein the multiple frames of first images are multiple frames of images of the scene where the first electronic device is located, taken by the first electronic device at different exposure parameters at a first time; Obtain the pose information of the first electronic device; Acquire multiple frames of second images that match the pose information of the first electronic device. The multiple frames of second images have the same scene as the multiple frames of first images but different shooting angles. Based on the multiple first images and the multiple second images, a high dynamic range panoramic image of the scene is obtained. The high dynamic range panoramic image is used to extract the lighting information of the scene at the first time. The lighting information of the scene at the first time is used to render virtual objects displayed on the first electronic device. The step of obtaining a high dynamic range panoramic image of the scene based on the multiple first images and the multiple second images includes: Based on the multiple frames of the first image, generate a high dynamic range local image and a low dynamic range local image; The illumination information of the low dynamic range local image is transferred to the multiple frames of the second image to obtain multiple frames of the third image; Based on the multiple frames of the third image, a low dynamic range incomplete image is obtained; The low dynamic range incomplete image is processed to generate a low dynamic range panoramic image; The illumination information of the high dynamic range local image is transferred to the low dynamic range panoramic image to obtain the high dynamic range panoramic image of the scene.

2. The method according to claim 1, characterized in that, The step of acquiring multiple frames of second images that match the pose information of the first electronic device includes: Based on the pose information of the first electronic device, the multiple frames of the second image are matched from a preset image library; The preset image library includes multiple frames of images taken by a first electronic device and / or at least one second electronic device.

3. The method according to claim 1, characterized in that, The step of obtaining the pose information of the first electronic device includes: The pose information of the first electronic device is determined based on the low dynamic range local image.

4. The method according to claim 1, characterized in that, The step of performing image completion processing on the low dynamic range incomplete image to generate the low dynamic range panoramic image includes: The low dynamic range incomplete image is input into a deep learning model to obtain the low dynamic range panoramic image output by the deep learning model.

5. The method according to any one of claims 1 to 4, characterized in that, The step of transferring the illumination information of the high dynamic range local image to the low dynamic range panoramic image to obtain the high dynamic range panoramic image of the scene includes: The high dynamic range local image and the low dynamic range panoramic image are input into a style transfer network to obtain the high dynamic range panoramic image of the scene output by the style transfer network. The style transfer network is used to transfer the illumination information of the high dynamic range local image to the low dynamic range panoramic image to obtain the high dynamic range panoramic image.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes at least one of the following: The direction of the main light source is determined based on the high dynamic range panoramic image; or, Based on the high dynamic range panoramic image, determine the spherical harmonic coefficients, and then determine the ambient light intensity and main light source intensity of the scene based on the spherical harmonic coefficients; or... Based on the high dynamic range panoramic image, an environmental texture map is determined. This environmental texture map is used to reflect the texture of the scene using a virtual object with a specular reflective material.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The high dynamic range panoramic image or the lighting information of the scene at the first time is sent to the first electronic device.

8. A method for estimating illumination, characterized in that, The method includes: The first action performed by the user has been detected; In response to the first operation, the first electronic device captures multiple frames of first images of the scene where the first electronic device is located at different exposure parameters in a first instant; The first multi-frame image is sent to the server. The first multi-frame image is used to obtain a high dynamic range panoramic image of the scene. The high dynamic range panoramic image is used to extract the lighting information of the scene at the first time. The lighting information of the scene at the first time is used to render a virtual object displayed on the first electronic device. The high dynamic range panoramic image of the scene is obtained by generating high dynamic range local images and low dynamic range local images based on the multiple first images; transferring the illumination information of the low dynamic range local images to multiple second images to obtain multiple third images, the multiple second images being matched with the pose information of the first electronic device, the multiple second images having the same scene as the multiple first images but different shooting angles; obtaining a low dynamic range incomplete image based on the multiple third images; performing image completion processing on the low dynamic range incomplete image to generate a low dynamic range panoramic image; and transferring the illumination information of the high dynamic range local images to the low dynamic range panoramic image to obtain the high dynamic range panoramic image of the scene.

9. The method according to claim 8, characterized in that, The method further includes: Receive the high dynamic range panoramic image or the scene's illumination information at the first time sent by the server; The virtual object is rendered and displayed on the first electronic device based on the high dynamic range panoramic image or the lighting information of the scene at the first time.

10. A light estimation device, characterized in that, The device includes: The transceiver module is used to receive multiple frames of first images sent by the first electronic device. The multiple frames of first images are multiple frames of images of the scene where the first electronic device is located, taken by the first electronic device at different exposure parameters at a first time. The processing module is used to acquire the pose information of the first electronic device; The processing module is further configured to acquire multiple frames of second images that match the pose information of the first electronic device, wherein the multiple frames of second images have the same scene as the multiple frames of first images but different shooting angles; The processing module is further configured to obtain a high dynamic range panoramic image of the scene based on the multiple first images and the multiple second images, wherein the high dynamic range panoramic image is used to extract the lighting information of the scene at the first time, and the lighting information of the scene at the first time is used to render virtual objects displayed on the first electronic device. The processing module is further configured to: Based on the multiple frames of the first image, generate a high dynamic range local image and a low dynamic range local image; The illumination information of the low dynamic range local image is transferred to the multiple frames of the second image to obtain multiple frames of the third image; Based on the multiple frames of the third image, a low dynamic range incomplete image is obtained; The low dynamic range incomplete image is processed to generate a low dynamic range panoramic image; The illumination information of the high dynamic range local image is transferred to the low dynamic range panoramic image to obtain the high dynamic range panoramic image of the scene.

11. The apparatus according to claim 10, characterized in that, The processing module is used for: Based on the pose information of the first electronic device, the multiple frames of the second image are matched from a preset image library; The preset image library includes multiple frames of images taken by a first electronic device and / or at least one second electronic device.

12. The apparatus according to claim 10, characterized in that, The processing module is used for: The pose information of the first electronic device is determined based on the low dynamic range local image.

13. The apparatus according to claim 10, characterized in that, The processing module is used for: The low dynamic range incomplete image is input into a deep learning model to obtain the low dynamic range panoramic image output by the deep learning model.

14. The apparatus according to any one of claims 10 to 13, characterized in that, The processing module is used for: The high dynamic range local image and the low dynamic range panoramic image are input into a style transfer network to obtain the high dynamic range panoramic image of the scene output by the style transfer network. The style transfer network is used to transfer the illumination information of the high dynamic range local image to the low dynamic range panoramic image to obtain the high dynamic range panoramic image.

15. The apparatus according to any one of claims 10 to 13, characterized in that, The processing module is also configured to perform at least one of the following: The direction of the main light source is determined based on the high dynamic range panoramic image; or, Based on the high dynamic range panoramic image, determine the spherical harmonic coefficients, and then determine the ambient light intensity and main light source intensity of the scene based on the spherical harmonic coefficients; or... Based on the high dynamic range panoramic image, an environmental texture map is determined. This environmental texture map is used to reflect the texture of the scene using a virtual object with a specular reflective material.

16. The apparatus according to any one of claims 10 to 13, characterized in that, The transceiver module is also used for: The high dynamic range panoramic image or the lighting information of the scene at the first time is sent to the first electronic device.

17. A light estimation device, characterized in that, The device includes: The processing module is used to detect the first operation performed by the user; The processing module is also configured to, in response to the first operation, capture multiple frames of first images of the scene where the first electronic device is located at different exposure parameters at a first time. The transceiver module is used to send the multi-frame first image to the server. The multi-frame first image is used to obtain a high dynamic range panoramic image of the scene. The high dynamic range panoramic image is used to extract the lighting information of the scene at the first time. The lighting information of the scene at the first time is used to render a virtual object displayed on the first electronic device. The high dynamic range panoramic image of the scene is obtained by generating high dynamic range local images and low dynamic range local images based on the multiple first images; transferring the illumination information of the low dynamic range local images to multiple second images to obtain multiple third images, the multiple second images being matched with the pose information of the first electronic device, the multiple second images having the same scene as the multiple first images but different shooting angles; obtaining a low dynamic range incomplete image based on the multiple third images; performing image completion processing on the low dynamic range incomplete image to generate a low dynamic range panoramic image; and transferring the illumination information of the high dynamic range local images to the low dynamic range panoramic image to obtain the high dynamic range panoramic image of the scene.

18. The apparatus according to claim 17, characterized in that, The transceiver module is further configured to: receive the high dynamic range panoramic image or the scene's illumination information at the first time sent by the server; The processing module is also used to render virtual objects displayed on the first electronic device based on the high dynamic range panoramic image or the lighting information of the scene at the first time.

19. A light estimation device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

20. A light estimation device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in claim 8 or 9.

21. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on a computer, cause the computer to perform the illumination estimation method as described in any one of claims 1-9.

22. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the illumination estimation method as described in any one of claims 1-9.

23. An illumination estimation system, characterized in that, The illumination estimation system includes an electronic device and a server. The electronic device establishes a communication connection with the server. The server is used to execute the illumination estimation method as described in any one of claims 1-7. The electronic device is used to execute the illumination estimation method as described in claim 8 or 9.

Images