Method and apparatus for displaying virtual objects based on light intensity

Abstract

This application relates to the field of AR technology, providing a method and device for displaying virtual objects based on illumination intensity. For each frame of color images of the real environment captured by a camera, corresponding local illumination maps are generated. Based on the spherical coordinates of the pixels contained in each frame of the local illumination map and the acquired camera pose data, the local illumination maps of each frame are uniformly mapped onto a spherical carrier to obtain a panoramic texture map. Since the panoramic texture map is generated based on each frame of color images captured in real time, it has high realism. Furthermore, different color images correspond to different camera pose data. Thus, the panoramic texture map contains the texture data of the real environment seen from each user's perspective. When illumination intensity is extracted from the panoramic texture map, the accuracy of the illumination intensity is ensured. Therefore, when displaying virtual objects based on illumination intensity, the surface brightness and shadows of the virtual objects are consistent with the illumination of the real environment, improving the realism of virtual-real fusion.

Description

Technical Field

[0001] This application relates to the field of augmented reality (AR) technology, and more particularly to a method and device for displaying virtual objects based on light intensity. Background Technology

[0002] AR technology overlays computer-generated objects such as virtual 3D objects, videos, text, and images onto real-world scenes in real time, achieving natural human-computer interaction through virtual-real fusion. It has broad application prospects in industries such as healthcare, military simulation, industrial maintenance, education, and entertainment.

[0003] The realism of AR effects is mainly reflected in geometric consistency, temporal consistency, and lighting consistency. Geometric consistency refers to the accuracy of the position, perspective, and occlusion relationships of computer-generated virtual objects in the real environment; temporal consistency refers to the coordination between the motion states of virtual objects and objects in the real environment; and lighting consistency refers to the matching of the light and shadow relationships between virtual objects and real objects.

[0004] In AR experiences, users have a subtle sense of the lighting in the environment. When the displayed virtual object has no light or shadow or the light and shadow do not reflect the light intensity in the real environment, the user will feel that the virtual object does not belong to the real environment, which reduces the AR experience.

[0005] Therefore, enabling the displayed virtual objects to match the lighting of the real environment is crucial for an immersive and realistic AR experience. Summary of the Invention

[0006] This application provides a method and device for displaying virtual objects based on light intensity, which can improve the realism of the integration of virtual objects with the real environment.

[0007] On one hand, embodiments of this application provide a method for displaying virtual objects based on light intensity, including:

[0008] Acquire each frame of color images of the real environment captured by the camera, and the attitude data of the camera corresponding to each frame of color images captured by the inertial measurement unit (IMU);

[0009] For each frame of color image, a local illumination map corresponding to the color image is generated, and the spherical coordinates of each pixel in the local illumination map are determined according to the resolution of the local illumination map and the field of view of the camera.

[0010] Based on the pose data corresponding to each frame of color image and the spherical coordinates of the pixels contained in each frame of local illumination map, the local illumination maps of each frame are uniformly mapped onto the spherical carrier to obtain the panoramic texture map.

[0011] Based on the panoramic texture map, the light intensity of the real environment is extracted, and virtual objects are displayed according to the extracted light intensity.

[0012] On the other hand, embodiments of this application provide a display device that supports AR technology, including a camera, an inertial measurement unit (IMU), a memory, a display, and a processor. The camera, the IMU, the memory, the display, and the processor are connected via a bus.

[0013] The camera is used to capture color images of the real environment;

[0014] The IMU is used to measure the attitude data of the camera;

[0015] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0016] Acquire each frame of color image of the real environment captured by the camera, and the pose data of the camera corresponding to each frame of color image captured by the IMU;

[0017] For each frame of color image, a local illumination map corresponding to the color image is generated, and the spherical coordinates of each pixel in the local illumination map are determined according to the resolution of the local illumination map and the field of view of the camera.

[0018] Based on the pose data corresponding to each frame of color image and the spherical coordinates of the pixels contained in each frame of local illumination map, the local illumination maps of each frame are uniformly mapped onto the spherical carrier to obtain the panoramic texture map.

[0019] Based on the panoramic texture map, the light intensity of the real environment is extracted, and virtual objects are displayed on the display according to the extracted light intensity.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a method for displaying virtual objects based on light intensity.

[0021] In the above embodiments of this application, each frame of color image of the real environment captured by the camera is acquired, and a local illumination map corresponding to each frame of color image is generated. Based on the resolution of each frame of local illumination map and the field of view of the camera, the spherical coordinates of each pixel in the corresponding local illumination map are determined. Based on the pose data of the camera corresponding to each frame of color image and the spherical coordinates of the pixels contained in each frame of local illumination map, each frame of local illumination map is uniformly mapped onto a spherical carrier to obtain a panoramic texture map. Since the panoramic texture map is generated based on each frame of color image of the real environment captured in real time, it has high realism. Moreover, the pose data of the camera corresponding to different color images are different, and different pose data reflect different user perspectives. In this way, the panoramic texture map contains the texture data of the real environment seen by the user from each perspective. When the light intensity is extracted from the panoramic texture map, the accuracy of the light intensity is ensured. Thus, when displaying virtual objects based on the light intensity, the surface brightness and shadow of the virtual objects can reflect the lighting of the real environment, improving the realism of the integration of virtual objects with the real environment. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 An exemplary flowchart illustrates a method for displaying virtual objects based on light intensity provided in an embodiment of this application;

[0024] Figure 2 An exemplary comparison diagram of different numbers of SG provided in the embodiments of this application is shown;

[0025] Figure 3 An exemplary diagram illustrating the AR application capturing real-world images provided in this application embodiment is shown.

[0026] Figure 4 An exemplary illustration shows a schematic diagram of an AR application provided in this application adding virtual objects to a real environment;

[0027] Figure 5 An exemplary diagram shows the effect of virtual objects being integrated with the real environment according to an embodiment of this application;

[0028] Figure 6 An exemplary structural diagram of a display device provided in an embodiment of this application is shown. Detailed Implementation

[0029] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0030] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0031] The terms “including” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0032] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0033] In AR scenarios, simulated virtual objects can be placed on real-world scenes, allowing the two types of information to complement each other, thereby "enhancing" the real environment.

[0034] In practical applications, the ambient light intensity of the real environment has a significant impact on whether virtual objects can be realistically integrated into the real scene. In darker environments, the surface brightness of virtual objects should be slightly lower; in brighter environments, the surface brightness should be slightly higher; and when the material of the virtual object has specular reflective properties, its surface should reflect the image of the real environment. Therefore, AR-enabled display devices need to have the ability to sense ambient light conditions.

[0035] Currently, when virtual objects are overlaid on real-world images, their texture mapping mostly employs machine learning methods. This involves extracting lighting intensity information from real-world images captured by a camera, and then adjusting the virtual object's lighting and surface brightness based on this information, thus achieving a natural blend between the virtual object and the real environment. However, when rendering virtual objects within panoramic images, the angles not captured by the camera require estimation using deep learning. This leads to a significant discrepancy between the estimated texture and the real-world environment, resulting in inaccurate extracted lighting intensity information and ultimately reducing the realism of the virtual object's integration with the real environment.

[0036] In view of this, embodiments of this application provide a method and device for displaying virtual objects based on illumination intensity. By collecting multiple frames of color images of the real environment from different user perspectives, a panoramic texture map of the real environment is generated. Compared with using machine learning methods to predict the texture map of the real environment, embodiments of this application ensure the consistency between the panoramic texture map and the real environment. When extracting the illumination intensity of the real environment from the panoramic texture map, the extracted illumination intensity is more accurate. Therefore, when displaying virtual objects based on accurate illumination intensity, the surface brightness and shadows of the virtual objects are consistent with the real environment, improving the realism of the integration of virtual objects and the real environment, and thus enhancing the user's AR experience.

[0037] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0038] like Figure 1 The diagram shown is a flowchart of a method for displaying virtual objects based on light intensity according to an embodiment of this application. This process is executed by a display device that supports AR technology and mainly includes the following steps:

[0039] S101: Acquire each frame of color image of the real environment captured by the camera, and the camera's pose data corresponding to each frame of color image captured by the IMU.

[0040] In the embodiments of this application, the display device is equipped with a camera and an inertial measurement unit (IMU). After the display device is powered on, the camera and IMU synchronously acquire data. The camera acquires color images of the real environment in real time, while the IMU measures the camera's attitude data in real time, which is the attitude data of the display device, denoted as (AngleX, AngleY, AngleZ). In practical applications, users can change the attitude data of the display device by rotating it, thereby changing the image of the real environment captured by the camera. In other words, by rotating the device, the camera can capture images of the real environment from different user perspectives.

[0041] In S101, the display device receives in real time each frame of color images of the real environment captured by the camera, as well as the camera's pose data corresponding to each frame of color images captured by the IMU. The color images are used to extract the light intensity of the real environment, and the pose data is used to unify the images from each user's perspective into a coordinate system.

[0042] It should be noted that the resolution and field of view of the camera of the display device in this embodiment can be preset according to actual needs. After being set, the resolution of each frame of color image is the same, which is equal to the resolution of the camera.

[0043] S102: For each frame of color image, generate a local illumination map corresponding to the color image, and determine the spherical coordinates of each pixel in the local illumination map based on the resolution of the local illumination map and the field of view of the camera.

[0044] The following uses a single color image as an example to describe the specific implementation process of S102.

[0045] In the embodiments of this application, the Spherical Gaussian (SG) parameter represents the use of N spherical Gaussian distributions to fit N light sources in the environment when performing light mapping on an image. The SG parameter includes the center direction of the light source, light source brightness information, and bandwidth. Based on the SG parameter, a local illumination map of the color image can be generated.

[0046] Specifically, after acquiring a color image, the display device determines the spherical Gaussian parameters of that color image and generates a local illumination map corresponding to the color image based on these parameters. The resolution of the local illumination map is the same as the resolution of the corresponding color image. The formula for calculating the spherical Gaussian parameters is as follows:

[0047] G(η;α,λ,μ)=αexp(λ(μ·η-1)) Formula 1

[0048] Where μ represents the direction vector from the center of the sphere to the center of the spherical Gaussian, λ∈(0,+∞) is the bandwidth of the spherical Gaussian, used to control the attenuation degree of the spherical Gaussian from the center to the surroundings, and α is the amplitude of the spherical Gaussian (i.e., RGB color weight). Inputting a point at a distance η from the center of the sphere, the value of the spherical Gaussian at that point can be calculated using Formula 1. Optionally, η can be set according to actual needs, for example, it can be 8 or 16.

[0049] For any color image, it can be fitted with N spherical Gaussians. The smaller N is, the weaker the fitting ability, and it can only represent rough ambient lighting. As N increases, the fitting ability improves, it can fit more details, and the represented ambient lighting is closer to the true value. Comparison of the same color image with different numbers of SG: Figure 2 As shown.

[0050] Furthermore, after obtaining the local illumination map corresponding to the color image, the spherical coordinates of each pixel in the local illumination map are determined based on the resolution of the local illumination map and the field of view of the camera.

[0051] Taking a single pixel as an example, in practice, the spherical longitude corresponding to the pixel is determined based on the camera's horizontal field of view, the lateral resolution of the local illumination map, and the pixel's horizontal coordinate, using the following formula:

[0052]

[0053] Furthermore, based on the camera's vertical field of view, the vertical resolution of the local illumination map, and the ordinate of the pixel, the spherical latitude corresponding to that pixel is determined using the following formula:

[0054]

[0055] Then, based on the radius, longitude, and latitude / longitude of the spherical carrier, the spherical coordinates of the pixel are determined using the following formula:

[0056] x w =r*cos(longitude)*cos(latitude) Formula 4

[0057] y W =r*cos(longitude)*sin(latitude) Formula 5

[0058] z w =r*sin(latitude) Formula 6

[0059] In formula 2-6, u represents the x-coordinate of the pixel, v represents the y-coordinate of the pixel, and FOV H The field of view (FOV) of a camera indicates its horizontal field of view. V The vertical field of view of the camera is represented by , W represents the horizontal resolution of the local illumination map where the pixel is located, H represents the vertical resolution of the local illumination map where the pixel is located, r represents the radius of the spherical carrier, longitude represents the spherical longitude corresponding to the pixel, latitude represents the spherical latitude corresponding to the pixel, (x w y W , z w ) represents the spherical coordinates of the pixel.

[0060] In the embodiments of this application, the spherical carrier is used to play panoramic images.

[0061] S103: Based on the pose data corresponding to each frame of color image and the spherical coordinates of the pixels contained in each frame of local illumination map, map each frame of local illumination map onto the spherical carrier to obtain a panoramic texture map.

[0062] The following uses a local lighting map as an example to describe the texture mapping process.

[0063] In practice, the display device determines the offset of the color image relative to the spherical carrier, including the rotation matrix R and translation vector t, based on the posture data of the display device (i.e., the camera's posture data) when acquiring the color image corresponding to the local illumination image. Further, the display device determines the offset of the color image relative to the spherical carrier based on the spherical coordinates (x, y, t) of the pixels contained in the local illumination image. w y W , z w The local illumination map is uniformly mapped onto the spherical carrier using the following formula:

[0064]

[0065] in,

[0066] In this embodiment of the application, the influence of the translation vector on the mapping result is not considered. Therefore, in actual calculation, the rotation matrix R can be determined according to the posture data of the display device, and the translation vector t can be set to 0.

[0067] In S103, as the user's head rotates, the display device can obtain multiple frames of color images from different viewing angles, and assign local illumination to each frame of color image. Figure 1 Once mapped onto a spherical carrier, the texture on the spherical carrier becomes increasingly rich, thus obtaining a panoramic texture map.

[0068] S104: Extract the lighting intensity of the real environment based on the panoramic texture map, and display virtual objects based on the extracted lighting intensity.

[0069] In the embodiments of this application, since the panoramic texture map is generated based on the real-time captured color images of each frame of the real environment, it has high realism. Moreover, the camera pose data corresponding to different color images are different, and different pose data reflect different user perspectives. Thus, the panoramic texture map contains the texture data of the real environment seen by the user from each perspective. When the light intensity is extracted from the panoramic texture map, the accuracy of the light intensity is ensured. Therefore, when displaying virtual objects based on the light intensity, the surface brightness and shadow of the virtual objects can reflect the lighting of the real environment, improve the realism of the integration of virtual objects with the real environment, and thus enhance the user experience.

[0070] In one scenario, the camera continuously captures color images of the real environment as the display device is turned on. If a duplicate color image is captured, the display device will replace the texture of the previously captured color image with the texture of the most recently captured color image.

[0071] In another scenario, in response to a light extraction stop command, the display device sends a stop data acquisition message to the camera and IMU. Upon receiving this message, the camera stops acquiring color images of the real environment, and simultaneously, the IMU, upon receiving the message, stops measuring the display device's pose data. Thus, the display device ceases acquiring color images of the real environment captured by the camera, as well as the camera's pose data.

[0072] In one alternative implementation, the light extraction stop command can be user-triggered. For example, the user clicks the "Stop" option in the user interface provided by the display device, sending a light extraction stop command to the display device.

[0073] In another alternative implementation, the system can be automatically triggered by the display device. For example, after obtaining a panoramic texture map with a horizontal 360° and vertical 180° view, a lighting extraction stop command can be sent to the display device.

[0074] The method for displaying virtual objects based on light intensity provided in this application embodiment can be applied to products that support AR technology, such as display devices like smartphones, smart TVs, laptops, and AR glasses.

[0075] For example, such as Figure 3 As shown, the user opens an AR application on their smartphone, activates the camera and IMU, and through the user interface, sees a color image of the real scene captured by the camera from the current viewpoint. Further, as... Figure 4 As shown, by long-pressing the user interface, the AR application brings up a virtual object addition interface, where the user selects the virtual objects to be added to the real-world scene. Once the AR application receives the selected virtual objects, it extracts the ambient light intensity from the panoramic texture map generated using the method provided in this embodiment. Based on the extracted light intensity, the added virtual objects are displayed, ensuring that the surface brightness and shadows of the virtual objects match the real environment, thus improving the realism of the integration between virtual objects and the real environment.

[0076] like Figure 5 The image shown is an AR effect obtained according to the method provided in the embodiments of this application. The teapot is an added virtual object. The surface brightness and light and shadow of the teapot are consistent with the light intensity of the real environment, which improves the realism of the image.

[0077] In some embodiments, after the AR application receives the virtual object selected by the user, it can directly display the virtual object and then extract the light intensity of the real environment from the panoramic texture map generated by the method provided in this application embodiment. Based on the extracted light intensity, the light intensity of the displayed virtual object is adjusted so that the surface brightness and shadow of the virtual object are consistent with the real environment, thereby improving the realism of the fusion between the virtual object and the real environment.

[0078] Based on the same technical concept, this application provides a display device that supports AR technology. The display device can execute the method flow of displaying virtual objects based on light intensity provided in this application and achieve the same technical effect, which will not be repeated here.

[0079] See Figure 6 The display device includes a camera 601, an IMU 602, a memory 603, a display 604, and a processor 605. The camera 601, IMU 602, memory 603, display 604, and processor 605 are connected via a bus 606.

[0080] Camera 601 is used to capture color images of the real environment;

[0081] The IMU602 is used to measure the camera's attitude data;

[0082] The memory 603 stores a computer program, and the processor 605 performs the following operations according to the computer program stored in the memory 603:

[0083] Acquire each frame of color images of the real environment captured by camera 601, and the camera pose data corresponding to each frame of color image captured by IMU 602;

[0084] For each frame of color image, a local illumination map corresponding to the color image is generated, and the spherical coordinates of each pixel in the local illumination map are determined according to the resolution of the local illumination map and the field of view of the camera.

[0085] Based on the pose data corresponding to each frame of color image and the spherical coordinates of the pixels contained in each frame of local illumination map, the local illumination maps of each frame are uniformly mapped onto the spherical carrier to obtain the panoramic texture map.

[0086] Based on the panoramic texture map, the light intensity of the real environment is extracted, and virtual objects are displayed on the monitor 604 according to the extracted light intensity.

[0087] Optionally, the processor 605 generates a local illumination map corresponding to the color image, specifically as follows:

[0088] Determine the spherical Gaussian parameters of the color image;

[0089] Based on the spherical Gaussian parameters, a local illumination map corresponding to the color image is generated.

[0090] Optionally, the processor 605 determines the spherical coordinates of each pixel in the local illumination map based on the resolution of the local illumination map and the field of view of the camera. Specifically, the operation is as follows:

[0091] For each pixel, perform the following operations:

[0092] The spherical longitude corresponding to the pixel is determined based on the horizontal field of view of the camera, the lateral resolution of the local illumination map, and the horizontal coordinate of the pixel.

[0093] The spherical latitude corresponding to the pixel is determined based on the camera's vertical field of view, the vertical resolution of the local illumination map, and the vertical coordinate of the pixel.

[0094] The spherical coordinates of the pixel are determined based on the radius, longitude, and latitude / longitude of the spherical carrier.

[0095] Optionally, the formula for determining the spherical coordinates of each pixel is as follows:

[0096]

[0097]

[0098] x w =r*cos(longitude)*cos(latitude)

[0099] y W =r*cos(longitude)*sin(latitude)

[0100] z w = r*sin(latitude)

[0101] Where u represents the x-coordinate of the pixel, v represents the y-coordinate of the pixel, and FOV H The field of view (FOV) of a camera indicates its horizontal field of view. V The vertical field of view of the camera is represented by , W represents the horizontal resolution of the local illumination map, H represents the vertical resolution of the local illumination map, r represents the radius of the spherical carrier, longitude represents the spherical longitude corresponding to the pixel, latitude represents the spherical latitude corresponding to the pixel, (x w y W , z w ) represents the spherical coordinates of a pixel.

[0102] Optionally, the processor 605 also performs:

[0103] In response to the light extraction stop command, the acquisition of color images of the real environment captured by the camera, as well as the camera's pose data, is stopped.

[0104] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform the methods of the foregoing embodiments.

[0105] This application also provides a computer program product for storing a computer program that performs the methods described in the foregoing embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0107] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A method for displaying virtual objects based on light intensity, characterized in that, Applications in augmented reality (AR) scenarios include: Acquire each frame of color images of the real environment captured by the camera, and the attitude data of the camera corresponding to each frame of color images captured by the inertial measurement unit (IMU). For each frame of color image, a local illumination map corresponding to the color image is generated, and the spherical coordinates of each pixel in the local illumination map are determined according to the resolution of the local illumination map and the field of view of the camera. Based on the pose data corresponding to each frame of color image and the spherical coordinates of the pixels contained in each frame of local illumination map, the local illumination maps of each frame are uniformly mapped onto the spherical carrier to obtain the panoramic texture map. Based on the panoramic texture map, the light intensity of the real environment is extracted, and virtual objects are displayed according to the extracted light intensity; The step of determining the spherical coordinates of each pixel in the local illumination map based on the resolution of the local illumination map and the field of view of the camera includes: For each pixel, perform the following operations: The spherical longitude corresponding to the pixel is determined based on the horizontal field of view of the camera, the lateral resolution of the local illumination map, and the horizontal coordinate of the pixel. The spherical latitude corresponding to the pixel is determined based on the vertical field of view of the camera, the vertical resolution of the local illumination map, and the vertical coordinate of the pixel. The spherical coordinates of the pixel are determined based on the radius of the spherical carrier, the longitude of the spherical surface, and the latitude and longitude of the spherical surface. The formula for determining the spherical coordinates of each pixel is as follows: Where u represents the x-coordinate of the pixel, v represents the y-coordinate of the pixel, and FOV H The field of view (FOV) of the camera represents the horizontal field of view. V The vertical field of view of the camera is represented by , W represents the horizontal resolution of the local illumination map, H represents the vertical resolution of the local illumination map, r represents the radius of the spherical carrier, longitude represents the spherical longitude corresponding to the pixel, latitude represents the spherical latitude corresponding to the pixel, (x w y W , z w ) represents the spherical coordinates of a pixel.

2. The method as described in claim 1, characterized in that, The generation of the local illumination map corresponding to the color image includes: Determine the spherical Gaussian parameters of the color image; Based on the spherical Gaussian parameters, a local illumination map corresponding to the color image is generated.

3. The method according to any one of claims 1-2, characterized in that, The method further includes: In response to a light extraction stop command, the acquisition of color images of the real environment captured by the camera, as well as the camera's pose data, is stopped.

4. A display device, characterized in that, The display device supports AR technology and includes a camera, an inertial measurement unit (IMU), a memory, a display, and a processor. The camera, the IMU, the memory, the display, and the processor are connected via a bus. The camera is used to capture color images of the real environment; The IMU is used to measure the attitude data of the camera; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquire each frame of color image of the real environment captured by the camera, and the pose data of the camera corresponding to each frame of color image captured by the IMU. For each frame of color image, a local illumination map corresponding to the color image is generated, and the spherical coordinates of each pixel in the local illumination map are determined according to the resolution of the local illumination map and the field of view of the camera. Based on the pose data corresponding to each frame of color image and the spherical coordinates of the pixels contained in each frame of local illumination map, the local illumination maps of each frame are uniformly mapped onto the spherical carrier to obtain the panoramic texture map. Based on the panoramic texture map, the light intensity of the real environment is extracted, and virtual objects are displayed on the display according to the extracted light intensity; The processor determines the spherical coordinates of each pixel in the local illumination map based on the resolution of the local illumination map and the field of view of the camera. Specifically, the operation is as follows: For each pixel, perform the following operations: The spherical longitude corresponding to the pixel is determined based on the horizontal field of view of the camera, the lateral resolution of the local illumination map, and the horizontal coordinate of the pixel. The spherical latitude corresponding to the pixel is determined based on the vertical field of view of the camera, the vertical resolution of the local illumination map, and the vertical coordinate of the pixel. The spherical coordinates of the pixel are determined based on the radius of the spherical carrier, the longitude of the spherical surface, and the latitude and longitude of the spherical surface. The formula for determining the spherical coordinates of each pixel is as follows: Where u represents the x-coordinate of the pixel, v represents the y-coordinate of the pixel, and FOV H The field of view (FOV) of the camera represents the horizontal field of view. V The vertical field of view of the camera is represented by , W represents the horizontal resolution of the local illumination map, H represents the vertical resolution of the local illumination map, r represents the radius of the spherical carrier, longitude represents the spherical longitude corresponding to the pixel, latitude represents the spherical latitude corresponding to the pixel, (x w y W , z w ) represents the spherical coordinates of a pixel.

5. The display device as described in claim 4, characterized in that, The processor generates a local illumination map corresponding to the color image, specifically by: Determine the spherical Gaussian parameters of the color image; generate a local illumination map corresponding to the color image based on the spherical Gaussian parameters.

6. The display device as described in any one of claims 4-5, characterized in that, The processor also performs: In response to a light extraction stop command, the acquisition of color images of the real environment captured by the camera, as well as the camera's pose data, is stopped.

Images