Rendering of augmented reality content
By combining the information of virtual light sources and physical light sources, the lighting effects are aggregated, and the challenges of AR devices in lighting estimation and re-illumination are solved, efficient and realistic AR video image generation is achieved, and the immersion and rendering quality of the user experience is improved.
Patent Information
- Application Number
- CN202380030565.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-07
- Filing Date
- 2023-02-17
- Publication Date
- 2025-01-10
AI Technical Summary
Existing AR devices have challenges in achieving efficient lighting estimation and re-illumination, especially on resource-limited devices, resulting in poor user experience and lighting estimation does not always provide optimal results, affecting the immersion of immersive scenes.
By combining information from virtual light sources and physical light sources, aggregate virtual re-lighting and mixed re-lighting effects are used to generate high-quality AR video images to ensure the realistic integration of virtual objects in real-world environments.
It realizes efficient and realistic lighting reconstruction on various AR devices, improves the immersion and rendering quality of the user experience, ensures the optimal rendering effect of AR content, and can achieve accurate real-time even on devices with limited resources. Light estimate.
Smart Images
Figure CN119301645A_ABST
Abstract
Claims
1. A method (400; 500) for performing AR content rendering, the method comprising: - obtaining (410; S14) a virtual relighting effect (RL1) based on virtual lighting information (DT6) representing at least one virtual light source (SC2); - obtaining (412; S16) a mixed relighting effect (RL2) based on the virtual lighting information (DT6) and physical lighting information (DT4) representing at least one physical light source (SC1); and - generating (408; S18) an AR video picture (PR3) by aggregating a real world scene relighted based on the virtual relighting effect (RL1) and at least one virtual object relighted by the hybrid relighting effect (RL2); and - Rendering (S20) the AR video picture (PR3).
2. The method according to claim 1, wherein generating an AR video screen comprises: - obtaining (410; S14) a re-lit real world scene by applying said virtual re-lighting effect (RL1) based on a video picture representing said real world scene; - obtaining (412; S18) at least one re-lit virtual object (OB2) by applying said hybrid re-lighting effect (RL2) based on three-dimensional (3D) information representing said at least one virtual object; and - Aggregating the re-lit real world scene and the at least one re-lit virtual object into the AR video frame.
3. The method according to claim 1 or 2, wherein the at least one physical light source and the at least one virtual light source are defined by the physical lighting information (DT4) and the virtual lighting information (DT6) as an ambient light source and / or a directional light source of any one of the following types: - Directional light source; -Point light source; - Spotlight source.
4. The method according to any one of claims 1 to 3, further comprising: - obtaining a scene description document, the scene description document comprising first lighting information associated with a first indicator (IT1); - identifying the first lighting information as the physical lighting information based on the first indicator (DT4); and - Obtaining the virtual lighting information via an API of an AR content rendering application (DT6).
5. The method according to any one of claims 1 to 3, further comprising: - obtaining a scene description document, the scene description document comprising at least one of first lighting information associated with the first indicator (IT1) and second lighting information associated with the second indicator (IT2); Wherein the method further comprises at least one of the following: - determining, based on the first indicator (IT1), that the first lighting information is to be used as the physical lighting information (DT4); and - determining, based on the second indicator (IT2), that the second lighting information is to be used as the virtual lighting information (DT6).
6. The method according to any one of claims 1 to 3, further comprising: - obtain a scene description document including a descriptor; as well as - accessing the virtual lighting information (DT4) and the physical lighting information (DT6) contained in the data stream according to the descriptor.
7. The method according to any one of claims 1 to 6, further comprising: - based on at least one indicator (IT1, IT2) included in the received bitstream, detecting that the received bitstream (BT2) carries the virtual lighting information and the physical lighting information as metadata; and - extracting the virtual lighting information (DT6) and the physical lighting information (DT4) from the received bitstream.
8. The method according to any one of claims 1 to 7, further comprising: - respectively obtaining 3D coordinates associated with the virtual lighting information (DT6) and the physical lighting information (DT4); as well as - determining, based on the 3D coordinates, the spatial positions to which the virtual lighting information and the physical lighting information are applicable relative to the real-world scene; The virtual relighting effect (RL1) and the mixed relighting effect (RL2) are determined based on the spatial position of the virtual lighting information and the spatial position of the physical lighting information.
9. A method (540) for implementing AR content rendering, the method comprising: - obtaining (S44) physical lighting information (DT4) representing at least one physical light source (SC1) by performing a first lighting estimation based on a video picture (PR1) of a real-world scene; - obtaining (S46) virtual lighting information (DT6) representing at least one virtual light source (SC2) emitting virtual light in the real world scene; as well as - Transmitting (S48) the physical lighting information and the virtual lighting information to achieve AR rendering of the AR video picture (PR3).
10. The method according to claim 9, wherein the physical lighting information (DT4) and the virtual lighting information (DT6) are transmitted in a scene description document to achieve AR rendering of the AR video picture (PR3), The scene description document comprises at least one syntax element representing the virtual lighting information and the physical lighting information and at least one indicator (IT1, IT2) indicating the existence of the virtual lighting information and / or the physical lighting information within the scene description document.
11. A scene description document (SD1) obtained according to the method of claim 10, formatted to include at least one syntax element representing virtual lighting information (DT6) and physical lighting information (DT4) and at least one indicator (IT1, IT2) indicating the existence of the virtual lighting information and / or the physical lighting information within the scene description document.
12. An AR device (DV1) for AR content rendering, the device comprising means for performing one of the methods of any one of claims 1 to 8.
13. A processing device (DV2) for implementing AR content rendering by an AR device (DV2), the processing device comprising means for executing the method according to claim 9 or 10.
14. A computer program product comprising instructions which, when the program is executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 10.
15. A non-transitory storage medium carrying program code instructions for executing the method according to any one of claims 1 to 10.