Method and system for reconstructing a multi-dimensional extended reality scene

Abstract

Embodiments herein disclose methods and systems for reconstructing a multi-dimensional extended reality (XR) scene. A method performed by an XR device can include operations of obtaining, using sensor data, information corresponding to at least one of: semantics of the scene, geometry of the scene, or acoustics of the scene; determining, using a trained model based on the obtained information, a scene type and at least one capture threshold for capturing the multi-dimensional scene; and providing real-time feedback and active guidance to a user. The method generates user guidance for assisting scene capture using the scene type and the at least one capture threshold to capture the multi-dimensional scene using the generated user guidance.

Description

Technical Field

[0001] The embodiments disclosed herein relate to extended reality (XR) systems, and more specifically, to methods and systems for guiding users to capture scenes for reconstruction using an XR device. Background Technology

[0002] Extended reality (XR) is rapidly evolving, enabling users in different locations to share an immersive experience within the same physical space through scene reconstruction. 3D XR experiences involve spatial, acoustic, lighting, and appearance information components, requiring advanced technologies to help users capture rich data for high-quality reconstruction.

[0003] Modeling scene lighting is crucial for enhancing the realism and immersion of XR experiences. It encompasses various elements that contribute to creating more convincing and engaging environments, such as, but not limited to, shadows, depth perception, virtual objects, and ambient lighting. For example, in a virtual reality (VR) party, dynamic lighting that changes based on the music being played can significantly enhance the atmosphere and make the experience more enjoyable and immersive for participants. Properly modeled lighting helps create believable and realistic virtual scenes that highly mimic the interaction of light with the real world, thereby improving the overall XR experience.

[0004] Similarly, modeling scene acoustics is crucial for achieving spatial audio, allowing users to identify the direction and distance of sounds within an XR environment, thus enhancing realism. Spatial audio adds depth to the experience, making it feel more authentic and immersive. For example, accurately modeled acoustics allow users to pinpoint the exact source of a sound (e.g., a conversation in a virtual meeting or the direction of footsteps in a VR game). This auditory information complements visual cues, creating a more cohesive and realistic experience.

[0005] Visual Perspective (VST) technology represents a significant advancement in extending computing into the 3D realm. VST offers unprecedented opportunities for creating and sharing media and experiences in a new dimension. Equipped with a large number of sensors for environmental awareness, VST opens up possibilities previously unattainable. These sensors allow for detailed capture and interaction with the 3D environment, enabling advanced applications such as, but not limited to, multi-dimensional XR scene reconstruction. This, in turn, necessitates sophisticated user-guided methods to optimally capture and utilize data from these diverse sensors, ensuring a seamless and rich XR experience. Summary of the Invention

[0006] Technical solution Therefore, embodiments of this document provide a method and system for reconstructing a multidimensional extended reality (XR) scene, the method comprising: obtaining, by an XR device, information corresponding to at least one of the following: semantics of the scene, geometry of the scene, or acoustics of the scene, using sensor data while a user moves within the scene during scene capture; determining, by the XR device, a scene type and at least one capture threshold for capturing the multidimensional scene based on the obtained information using a trained model, wherein the multidimensional scene includes at least one of the following: texture characteristics of the scene, spatial characteristics of the scene, acoustic characteristics of the scene, and lighting characteristics of the scene; generating, by the XR device, user guidance for assisting scene capture using the scene type and at least one capture threshold, wherein the user guidance includes at least one of the following: a path to be followed by the user or at least one action to be performed by the user along the path; and capturing, by the XR device, a multidimensional scene representing at least one of the following: spatial characteristics of the scene, lighting characteristics of the scene, or acoustic characteristics of the scene during assisted scene capture using the generated user guidance.

[0007] Accordingly, embodiments of this document provide a method and system for obtaining information corresponding to at least one of the following: the semantics of a scene, the geometry of a scene, or the acoustics of a scene, including: receiving sensor data from at least one sensor in an XR device when a user of an XR device begins capturing a scene, and using the sensor data to obtain information corresponding to at least one of the following: the semantics of a scene, the geometry of a scene, and the acoustics of a scene.

[0008] Accordingly, embodiments of this document provide a method that performs at least one of the following: indicating a gaze point for a user to focus an image device, indicating a walking speed based on at least one detail in a portion of a scene, and enhancing captured data for at least one of optimized texture, optimized acoustics, optimized lighting, and optimized material reconstruction in the scene.

[0009] Accordingly, embodiments of this article provide a method that performs at least one of the following: a listening experience at a given location in a multidimensional XR scene.

[0010] Accordingly, embodiments of this document provide a method for capturing a multidimensional scene representing at least one of the following during assisted scene capture using generated user guidance: spatial characteristics of the scene, lighting characteristics of the scene, and acoustic characteristics of the scene. The method includes: processing image-associated labels, image depth, microphone response signals, scene-associated material classifications, sound generation time, lighting information, and acoustic information; and having an XR device capture a multidimensional scene representing at least one of the following based on the processing: spatial characteristics of the scene, lighting characteristics of the scene, and acoustic characteristics of the scene.

[0011] Accordingly, embodiments of this document provide a method and system for determining a multidimensional scene by: generating a map at the start of traversing the environment; marking areas and purposes in which the user plans to traverse the environment on the map; estimating the ground plane, user height, and pose regions in the environment; and estimating occlusion blind spots by ray propagation from pose regions in the environment.

[0012] Accordingly, embodiments of this document provide a method and system in which user guidance includes at least one of the following: modifying lighting in a scene to capture a model of scene lighting by different light sources by at least one of the following: turning on lights, turning off lights, adjusting the brightness of lights, and changing the color of lights.

[0013] These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. However, it should be understood that while the following description indicates at least one embodiment and its many specific details, it is given by way of illustration rather than limitation. Many changes and modifications can be made within the scope of the embodiments herein without departing from the scope of these embodiments, and the embodiments herein encompass all such modifications. Attached Figure Description

[0014] Embodiments of the invention are illustrated in the accompanying drawings, throughout which the same reference numerals indicate corresponding portions in the various figures. The embodiments of the invention will be better understood from the following description and with reference to the following illustrative drawings. Embodiments of the invention are illustrated by way of example in the accompanying drawings, wherein: Figure 1 A scene reconstruction method based on existing technologies is described; Figures 2A to 2E A flowchart is depicted for providing real-time guidance to a user to capture data for performing joint spatial-acoustic-lighting for XR scene reconstruction, according to embodiments disclosed herein; Figures 3A to 3G An example user journey according to embodiments disclosed herein is described, wherein a user uses joint spatial-acoustic-lighting for XR scene reconstruction; Figure 4 Hardware components of an XR device according to embodiments disclosed herein are described; Figure 5 An example user scenario of a virtual experience according to embodiments disclosed herein is described; Figure 6 Example user scenarios of social media according to embodiments disclosed herein are described; Figure 7 An example user scenario depicting the productivity of embodiments as disclosed herein is described; Figure 8Example user scenarios are described based on the architecture and design of the embodiments disclosed herein; Figures 9A to 9B The differences between example spatial video according to embodiments disclosed herein and video captured using joint spatial-acoustic-lighting XR scene reconstruction are depicted; Figure 10 An example is depicted of real-time guidance and feedback for capturing media reconstructed using a joint spatial-acoustic-lighting XR scene, according to embodiments disclosed herein; Figures 11A to 11B Example lighting and acoustics of a reconstructed scene, which can be modified using joint spatial-acoustic-lighting XR scene reconstruction according to embodiments disclosed herein, are described. Figures 12A to 12B An example flowchart with real-time user assistance is depicted according to embodiments disclosed herein; Figures 13A to 13B An example flowchart depicting XR scene reconstruction according to embodiments disclosed herein is provided. Figure 14A and Figure 14B Example surfaces with high texture richness and surfaces with low texture richness are depicted according to embodiments disclosed herein; Figure 15 Examples of re-creation and high-fidelity spatial reconstruction with spatial and orientation coverage are depicted according to embodiments disclosed herein; Figure 16 An example view depicting the orientation of a light source for modeling and recreating a lighting experience according to embodiments disclosed herein; Figure 17 This is a flowchart depicting a method for providing user guidance for reconstructing a multidimensional XR scene according to embodiments disclosed herein; and Figure 18 This is a flowchart depicting a method for reconstructing a multidimensional XR scene according to embodiments disclosed herein. Detailed Implementation

[0015] The embodiments described herein, along with their various features and advantageous details, are explained more fully with reference to the non-limiting embodiments illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques have been omitted to avoid unnecessarily obscuring the embodiments herein. The examples used herein are intended only to help understand how the embodiments herein can be implemented and to further enable those skilled in the art to implement the embodiments herein. Therefore, the examples should not be construed as limiting the scope of the embodiments herein.

[0016] For the purposes of interpreting this specification, definitions (as defined herein) will be applied, and terms used in the singular will include the plural forms where appropriate, and vice versa. It will be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Unless otherwise stated, the terms “comprising,” “having,” and “including” should be interpreted as open-ended terms.

[0017] The words / phrases “exemplary,” “example,” “illustration,” “in instance,” “etc.,” “for example,” and “i.e.” are used herein only to mean “used as an example, instance, or illustration.” Any embodiment or implementation of the subject matter described herein using the words / phrases “exemplary,” “example,” “illustration,” “in instance,” “etc.,” “for example,” and “i.e.” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0018] The embodiments described herein may be described and illustrated in terms of blocks that perform one or more of the described functions. These blocks, which may be referred to herein as managers, units, modules, hardware components, etc., are physically implemented by analog and / or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuitry, etc.) and may optionally be driven by firmware. For example, the circuitry may be embodied in one or more semiconductor chips or on a substrate support (such as a printed circuit board). The circuitry constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware for performing some functions of the block and a processor for performing other functions of the block. Each block of an embodiment may be physically divided into two or more interacting and discrete blocks without departing from the scope of this disclosure. Similarly, the blocks of an embodiment may be physically combined into more complex blocks without departing from the scope of this disclosure.

[0019] It should be noted that the elements in the accompanying drawings are shown for illustrative and ease-of-understanding purposes and may not necessarily be drawn to scale. For example, flowcharts / sequence diagrams illustrate methods according to the steps required to understand aspects of the embodiments disclosed herein. Furthermore, regarding the construction of the apparatus, one or more components of the apparatus may have been represented by conventional symbols in the drawings, and the drawings may only show specific details relevant to understanding this embodiment, so as not to obscure the drawings by including details that will be readily understood by those skilled in the art upon benefiting from this description. Similarly, regarding the system, one or more components / modules constituting the system may have been represented by conventional symbols in the drawings, and the drawings may only show specific details relevant to understanding this embodiment, so as not to obscure the drawings by including details that will be readily understood by those skilled in the art upon benefiting from this description.

[0020] The accompanying drawings are provided to aid in the easy understanding of the various technical features, and it should be understood that the embodiments presented herein are not limited to the drawings. Therefore, this disclosure should be construed as extending to any modifications, equivalents, and alternatives other than those specifically set forth in the drawings and corresponding descriptions. The use of terms such as first, second, third, etc., to describe components / elements / steps is for illustrative purposes and should not be construed as indicating a sequential order / placement / occurrence unless otherwise stated.

[0021] According to embodiments of this disclosure, methods and systems for guiding users to capture scenes for reconstruction (e.g., multidimensional extended reality (XR) scenes, etc.) using extended reality (XR) devices may be disclosed.

[0022] Alternatively or additionally, embodiments of this disclosure may provide real-time feedback and proactive guidance to the user (while the user is capturing data) to ensure accurate data collection for capturing multidimensional scene information, while providing high-quality scene reconstruction in a shorter time.

[0023] Alternatively or alternatively, embodiments of this disclosure may provide methods and systems for capturing multidimensional scene information (such as, but not limited to, spatial, acoustic, appearance, and lighting information) within a single frame.

[0024] Additionally or alternatively, embodiments of this disclosure may provide methods and systems for customizing a scene through one or more of scene relighting and acoustic reshaping.

[0025] Alternatively or additionally, embodiments of this disclosure may use a scene type and at least one capture threshold to generate user guidance for assisting scene capture. User guidance includes at least one of the following: a path the user is to follow or at least one action the user is to perform along the path.

[0026] Alternatively or additionally, embodiments of this disclosure may be used to capture a multidimensional scene representing at least one of the following during assisted scene capture: spatial characteristics of the scene, lighting characteristics of the scene, and acoustic characteristics of the scene.

[0027] These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. However, it should be understood that while the following description indicates at least one embodiment and its many specific details, it is given by way of illustration rather than limitation. Many changes and modifications can be made within the scope of the embodiments herein without departing from the scope of these embodiments, and the embodiments herein encompass all such modifications.

[0028] The embodiments described herein provide user guidance for reconstructing multidimensional XR scenes using an XR device, wherein the XR device provides real-time feedback and proactive guidance to the user to obtain a high-quality XR experience, wherein high-dimensional capture allows spatial richness, pose richness, acoustic richness, and lighting richness to be captured in a single frame.

[0029] Now refer to the accompanying drawings, and more specifically to... Figures 1 to 18 This paper provides a method and system for reconstructing multidimensional extended reality (XR) scenes, in which similar reference numerals are consistently used to denote corresponding features throughout the accompanying drawings.

[0030] Figure 1 An exemplary scene reconstruction method is described. In step 102, an input frame is received from an extended reality (XR) scene. In step 104, an initial capture of the existing scene is performed. In step 106, the initial capture of the existing scene is converted into a 3D structure. In step 108, when the user views the captured structure, the quality of the converted 3D structure is analyzed. In step 110, if the quality of the converted 3D structure is good, the captured frame is exported as a 3D scene. In step 112, if the quality of the converted 3D structure is poor, receiving input frames begins again.

[0031] According to embodiments of this disclosure, a method for reconstructing (or capturing) an XR scene may include: obtaining real-time feedback from the user, and / or a verification tool, profiling tool, or proxy metric for evaluating the quality of the data capture prior to scene reconstruction. The presence of a proxy metric results in a time-saving process for the user. Additionally or optionally, the method adequately addresses the reconstruction of acoustic properties, lighting, and material details to achieve optimal results in these areas. Additionally or optionally, a method for capturing an XR scene may include: obtaining user feedback that allows the user to easily measure the richness of the captured data or the expected quality of the reconstruction. This improves the experience for non-experts capturing high-quality XR scenes. For example, it eliminates the need to wait for post-processing to determine if the reconstruction is satisfactory. Even with poor quality, the user does not need to recapture the scene, saving time and reducing costs. Additionally or optionally, the method captures various aspects of the scene to enhance the immersive experience. For example, information about lighting and acoustic elements can be used to convincingly customize the reconstructed scene.

[0032] Figures 2A to 2E A flowchart is depicted for providing real-time user guidance to enhance data capture for joint spatial-acoustic-lighting XR scene reconstruction. Figure 2AThe process of providing XR input to a user-guided phase is described, involving the use of various extended reality (XR) inputs, such as data from camera 206, inertial measurement unit (IMU) 201, time-of-flight (ToF) sensor 202, microphone 203, and eye-tracking camera 205, to verify and refine marked capture areas in the scene. For example, camera 206 can provide processed high dynamic range (HDR) images. IMU 201 can provide filtered IMU readings. ToF sensor 202 can provide processed dense depth. Microphone 203 can provide conditioned speech signals. Eye-tracking camera 206 can provide information for eye-tracking.

[0033] XR input can be processed and used to verify (or identify or determine) a capture area marker. If the capture area is verified as marked (“true”), the user-marked capture area module 236 can activate the capture area. Processed input data can be transmitted to the user-marked capture area module 236. If the capture area is not marked or cannot be verified (“false”), the user-marked capture area module 236 can not activate the capture area. Optionally or additionally, the user 232 can provide the user-marked capture area to the user-guided capture phase 242.

[0034] The method according to embodiments of the present disclosure includes: using sensor data to obtain information corresponding to at least one of the following: semantics of a scene, geometry of a scene, and acoustics of a scene. The method according to embodiments of the present disclosure includes: receiving sensor data from at least one sensor in an XR device when a user of an XR device begins capturing a scene. The method according to embodiments of the present disclosure includes: using sensor data to obtain information corresponding to at least one of the following: semantics of a scene, geometry of a scene, and acoustics of a scene. The method according to embodiments of the present disclosure includes: performing at least one of the following: a listening experience at a given location in a multidimensional XR scene.

[0035] Alternatively or additionally, the method may include: capturing a multidimensional scene representing at least one of the following during assisted scene capture using generated user guidance: spatial characteristics of the scene, lighting characteristics of the scene, or acoustic characteristics of the scene. Alternatively or additionally, the method may include: processing at least one of the following: image-associated labels, image depth, microphone response signals, scene-associated material classification, sound generation time, lighting information, or acoustic information. Alternatively or additionally, the method may include: capturing a multidimensional scene representing at least one of the following based on said processing: spatial characteristics of the scene, lighting characteristics of the scene, or acoustic characteristics of the scene. Scene understanding module 238 processes these inputs to estimate geometry and semantics, thereby creating a coarse mesh of the scene. The user then selects specific capture settings (such as texture, acoustics, lighting, and scene type), combining lighting and material property modeling to realistically simulate lighting effects and material textures. These settings guide a user-guided capture phase 242, which sends relevant parameters for multidimensional scene reconstruction to... Figure 2B The high-dimensional joint scene reconstruction module 240 shown ensures accurate and immersive representation of the environment. A multi-dimensional scene essentially includes at least one of the following: 3D scene information, scene reconstruction, scene segmentation, lighting details of the scene including light sources, room color, room reflectivity or bidirectional reflectivity distribution function (BRDF) characteristics, room acoustics, acoustic properties of objects in the scene, or the scene's acoustic response function. According to embodiments of this disclosure, real-time user assistance for data capture can be provided to ensure that users can effectively acquire and integrate data during their interactions, while using quality and richness metrics to evaluate the effectiveness and fidelity of the XR data capture process. For example, a user-guided capture phase 242 can provide the user 232 with optimal paths and speeds for capture, recommended gaze points, and / or recommended motion planning.

[0036] Figure 2B The process of performing multi-dimensional joint scene reconstruction is described. Figure 2BThis paper describes the creation of a detailed and accurate 3D model of a scene by integrating various data types and addressing key aspects of scene representation, acoustic accuracy, and user interaction in a dynamic and realistic virtual environment. The process begins by incorporating pose richness for capturing precise position and movement within the scene, along with detailed lighting and material properties, and validating marked capture areas using inputs from devices such as cameras, IMUs, ToF sensors, microphones, and eye-tracking cameras. The multidimensional scene is determined by: generating a map at the start of environment traversal and marking areas and purposes the user plans to traverse within the environment on the map; estimating the ground plane, user height, and pose regions within the environment; and estimating occlusion blind spots through ray propagation from pose regions within the environment. The semantics of the scene are obtained using a trained object detection and segmentation module, and the geometry of the scene is obtained by developing a coarse mesh depiction of the scene using Simultaneous Localization and Mapping (SLAM) techniques, where SLAM is used to understand the 3D scene to build a map and simultaneously localize a cluster of cameras / sensors within the map. The XR device guides objects in the scene, triggers the audio device associated with the XR device to emit acoustic signals and receive acoustic responses, and activates the light source associated with the XR device to capture the scene and its objects under changing lighting conditions. It guides the user's head posture by instructing the user to point their visual center to a moving overlay visual cue within the XR device. Then, Figure 2A The scene understanding module 238, shown in the diagram, estimates the geometry and meaning of the scene to form a basic mesh. The user selects capture settings (such as texture, acoustics, and lighting) for the guided capture phase. The acquired data (including HDR images with pose markers, depth images, microphone responses, and material classification) is sent to the high-dimensional joint scene reconstruction module 240. The high-dimensional joint scene module 240 combines spatial, acoustic, lighting, and material properties to construct a comprehensive 3D representation of the scene on a device or in the cloud.

[0037] Figure 2CThe process of providing XR input to a user-guided phase is described. This process involves using data from various sources, such as camera 206, IMU 201, ToF sensor 202, microphone 203, and eye-tracking camera 205, to accurately label and understand the capture area. This data helps a scene understanding module (e.g., user-labeled capture area module 236) estimate the geometry and semantics of the scene, thereby creating a coarse 3D model or coarse mesh. The user then selects a specific user-labeled capture area module 236 (such as Simultaneous Localization and Mapping (SLAM) module 231, segmentation module 222, classification module 212, and coarse mesh generator 233). The user-labeled capture area module 236 uses learned user profiles / user-defined custom settings 235 to label the capture area and accurately reconstruct the spatial, acoustic, and material properties of the scene on the device or in the cloud, ensuring realistic and detailed scene reconstruction. In embodiments, SLAM module 231, segmentation module 222, classification module 212, and coarse mesh generator 233 may be included in... Figure 2A The scene understanding module 238 is shown in the diagram. Classification module 212 performs object detection. Classification module 212 and segmentation module 222 are trained modules for identifying and locating objects in an image and / or scene, and for instance segmentation or object segmentation in pixel space. SLAM is a method for 3D scene understanding that constructs a map and simultaneously locates cameras and sensors within the map. SLAM module 231 allows for mapping the environment. The training model is a neural network trained on labeled data using various techniques in machine learning to predict outputs for different new, unseen inputs, minimizing prediction errors on known datasets, and for testing accuracy on unseen data. Alternatively or optionally, the trained model can also be a deterministic algorithm, such as SLAM module 231, which is a series of steps to generate a 3D reconstructed map and locate a cluster of cameras / sensors on the map. Capture thresholds include at least one of the following: spatial quality threshold, acoustic threshold, lighting threshold, or material appearance threshold. Each threshold included in the capture thresholds defines the quality of a reconstruction metric for the scene, such that if the reconstruction quality score from the scene reconstruction algorithm exceeds the threshold, the scene is considered accurately reconstructed according to the user's desired accuracy level. The capture thresholds include predefined criteria using a pre-trained model, which are optimal for capturing the scene's texture, spatial, acoustic, and lighting characteristics. Alternatively or additionally, thresholds are determined based on the scene's semantics and geometry using received sensor data. Scene geometry includes at least one of the scene's three-dimensional structures. Alternatively or additionally, the three-dimensional structure includes at least one of the following: the shape of objects in the scene, the distance between objects in the scene, or the relative position of objects in the scene. The scene's texture characteristics include the surface details of objects in the scene. The scene's spatial characteristics include the arrangement of objects in the scene.

[0038] Alternatively or additionally, the acoustic characteristics of the scene include sound properties (including echoes and / or ambient noise during scene capture). Alternatively or additionally, the lighting characteristics of the scene include scene capture lighting conditions. The training model is a neural network trained on labeled data using various techniques in machine learning to predict outputs for different new, unseen inputs, minimizing prediction errors on known datasets, and for testing accuracy on unseen data. Alternatively or additionally, the trained model can also be a deterministic algorithm, such as SLAM module 231, which is a series of steps for generating a 3D reconstructed map and locating a cluster of cameras / sensors on the map to achieve various metrics for specific and guided actions. Deterministic algorithms can effectively create an abstraction layer that simplifies the process of achieving high-quality XR experiences, even for individuals without specialized knowledge. The methods according to this disclosure enable applications to meticulously capture and reconstruct physical spaces and objects with high fidelity and multidimensional detail. For example, this method can provide real-time capture guidance by tracking and analyzing one or more key metrics, such as, but not limited to, pose coverage (which ensures accurate spatial location); audio pulse coverage (which measures the effectiveness of sound capture); and texture detail score (which assesses the sharpness and resolution of visual elements)).

[0039] Figure 2D The process of a user-guided capture phase based on user-marked capture regions is described. The XR system estimates the geometry and key details of the scene to create a coarse 3D model (e.g., a coarse mechanism of the scene). The user selects capture settings (e.g., texture, acoustics, lighting) to guide the next phase. The selected capture settings or user-marked capture regions can be provided to the user-guided capture phase 242. SLAM module 231 provides scene understanding to the user-guided capture phase 242. Segmentation module 222 and classification module 212 provide scene geometry and semantics to the user-guided capture phase 242. Coarse mesh generator 233 provides a coarse mesh of the scene to the user-guided capture phase 242. The model used to determine the capture settings provides one or more capture thresholds (e.g., texture threshold, acoustic threshold, lighting threshold, and / or accuracy threshold) to the user-guided capture phase 242.

[0040] The user-guided capture phase 242 may include a user motion planner, a user gaze planner, and / or a path and velocity planner. The user motion planner may recommend motion plans to the user 232. The user gaze planner may recommend one or more gaze points to the user 232. The path and velocity planner may provide the user 232 with the optimal path and velocity for capture. The guided capture phase 242 may provide the user 232 with guidance including at least one of the following: recommended motion plans, recommended gaze points, or the optimal path and velocity for capture.

[0041] The user-guided capture phase 242 may suggest or recommend one or more actions to the user 232. These actions may include, but are not limited to, placing audio sources and / or audio receivers (e.g., telephones, headphones, and / or speakers) at indicated locations along the guided path for richer acoustic reconstruction. These actions may also include, but are not limited to, modifying the lighting in the scene by turning one or more light sources on or off, modifying the brightness and / or color of the light sources for better capture, and / or modeling how each light source affects the lighting of the scene.

[0042] Figure 2E The process of performing multidimensional joint scene reconstruction is described. The method according to this disclosure involves using data from various sources (such as...) Figure 2A The XR input of the camera 206, IMU 201, ToF sensor 202, microphone 203, and eye-tracking camera 205 (shown in the diagram) is used to verify the capture area marking. If the area is to be marked, the scene understanding module (e.g., Figure 2A The scene understanding module 238 estimates the geometry and semantics of the scene to create a coarse mesh. Coarse mesh rendering is a representation of the 3D scene that provides a good understanding of the surfaces. The mesh density, or the number of points in the mesh, is low, thus providing a large-scale understanding. While coarse mesh rendering involves a low mesh density or the number of points in the mesh and may miss minute details, it is generated quickly and utilizes less sensor data due to the lower mesh resolution used for dynamic tasks such as obstacle avoidance and path planning.

[0043] The method according to this disclosure includes determining one or more capture settings (such as, but not limited to, texture, acoustics, lighting accuracy, and scene type), which are selected by the user and sent to a user-guided capture phase 242. A typical listening experience (i.e., acoustics) at a given location within a site may be determined by the distance between the audio source and the ear (or microphone); and / or by the dimensions of the location and the sound absorption coefficients of the walls and floor of the location (if the given audio scene corresponds to the location). The user-guided capture phase 242 may send one or more parameters to a high-dimensional joint scene reconstruction module 240, which performs spatial, acoustic, lighting, and material property reconstruction on a device or in the cloud.

[0044] Examples of parameters can be, but are not limited to, pose-tagged HDR images, depth images, microphone responses, material classification, sound generation time, etc. Pose-tagged HDR images capture high dynamic range vision and camera position, aiding in 3D scene reconstruction, while depth images map object distances, essential for 3D modeling. Depth information is crucial for constructing a 3D model of the scene. Audio data captured by the microphone (e.g., microphone responses) includes sound characteristics used to recreate the acoustic environment. Material classification involves identifying and classifying materials present in the scene, contributing to a more accurate scene reconstruction. The specific time at which sound is generated within the scene (e.g., signature sound generation time) helps synchronize sound with visual and spatial data, ensuring the reconstructed scene accurately reflects the original environment. The various actions mentioned above can be performed in the order of presentation, in a different order, or simultaneously. Additionally or optionally, the method provides real-time guidance during the capture process. This allows users to make informed adjustments on the fly, enhancing the overall quality and accuracy of the XR experience.

[0045] The embodiments described herein can assess the quality of input data (before reconstruction), thereby enhancing the accuracy and relevance of captured data. By using learned user profiles and scene-specific quality settings, the embodiments described herein provide dynamic, real-time guidance tailored to the user's environment during the capture process.

[0046] The embodiments described herein employ personalized feedback mechanisms to ensure that input data is optimized for the specific needs of various user roles, such as those involved in social media, architecture, music, etc. The embodiments described herein can utilize default and custom profiles to address the unique requirements of different users, thereby enhancing the versatility and applicability of the method across different domains and applications. The embodiments described herein can improve the user experience by delivering context-aware, actionable insights during data capture. Furthermore, in some embodiments, [the following may be omitted]. Figures 2A to 2E Some of the actions listed in the document.

[0047] Users can enhance their audio experience by strategically placing audio sources and receivers (such as, but not limited to, phones, headphones, and speakers) at one or more designated locations along a suggested path to achieve richer acoustic reconstruction. Additionally, users can optimize scene lighting by adjusting existing light sources (such as by turning them on or off, or by changing their brightness and color settings) to accurately capture and model the impact of each light source on the overall scene lighting. These adjustments will significantly improve the auditory and visual quality of the scene, creating a more immersive and detailed environment.

[0048] The embodiments described in this paper enhance the comprehensiveness and quality of the collected data, thereby facilitating superior XR reconstruction of various scene attributes, such as, but not limited to, geometry, texture, acoustics, and lighting. By improving the fidelity and detail of the captured data, users can achieve a more accurate and realistic representation of the physical environment in XR, resulting in a more immersive and believable user experience. This comprehensive data augmentation ensures that every aspect of the scene—from spatial arrangement and material properties to sound dynamics and lighting conditions—is meticulously reconstructed to accurately reflect real-world conditions.

[0049] Figures 3A to 3G An example user journey is depicted using the proposed joint spatial-acoustic-lighting XR scene reconstruction method. Figures 3A to 3G In the embodiment shown, user 30 uses XR device 300. Although XR device 300 is shown as a pair of handheld controllers, the configuration of XR device 300 is not limited to this. For example, XR device 300 may include a display device (e.g., a head-mounted display, augmented reality (AR) / XR helmet, or AR / VR glasses) that provides a user interface to user 30. In embodiments, XR device 300 may utilize one or more algorithms to track one or more gestures of user 30 without any handheld controllers.

[0050] Figure 3A An example image is shown indicating that a user 30 of the XR device 300 has accessed an application for XR scene reconstruction. When the user 30 uses the XR device 300, the XR device 300 can provide a user interface 302 including one or more icons representing corresponding functions. For example, the user interface 302 may include at least one of the following: an icon representing the current time, an icon representing the user 30's face (or image), an icon representing one or more wireless connections of the XR device 300, an icon corresponding to a function representing a list of applications installed on the XR device 300, an icon corresponding to a function representing a list of contacts stored on the XR device 300, an icon corresponding to a function representing a list of notifications occurring in the XR device 300, an icon corresponding to a function for sharing one or more XR scenes generated by the XR device 300, or an icon corresponding to a function for modifying the settings of the XR device 300.

[0051] In response to user 30's input of icons corresponding to functions representing a list of applications installed on XR device 300, XR device 300 can provide application list user interface 304. Application list user interface 304 may include one or more icons corresponding to one or more applications of XR device 300, respectively. XR device may also display line user interface 306 indicating user 30's tracking intent. Line user interface 306 may be displayed based on gestures of user 30 tracked by XR device 300. In response to input of icon 308, an application for joint spatial-acoustic-lighting XR scene reconstruction can be opened. Furthermore, in the depicted example, when user 30 opens the application for joint spatial-acoustic-lighting XR scene reconstruction, an interface prompting user 30 to begin the process may be provided.

[0052] Figure 3B Example images are shown of an application for XR scene reconstruction that provides instructions via a guided user interface 310. The application can provide a guided user interface 310 that guides the user 30 to walk around the scene and mark the capture area. For example, via the guided user interface 310, the application instructs the user 30 to walk around the environment and mark the boundaries of the capture area.

[0053] Figure 3C An example image is shown, indicating that a user 30 wearing an XR device 300 confirms that the capture area is correctly marked on the ground plane. Figure 3C In the embodiments shown, the XR device 300 may include a head-mounted wearable device 300c having an imaging device (e.g., a camera). Figure 3D An example image is shown indicating an estimated path 312 for optimal capture, which has been highlighted for user 30 to follow. The XR device may display a user interface 314 that prompts user 30 to confirm the estimated path 312 (e.g., the capture area). In response to input from user 30 to a “back” icon, the application may re-estimate the capture area. In embodiments of this disclosure, a gaze point is indicated to the user to focus the imaging device (e.g., the camera of XR device 300). Additionally or alternatively, walking speed may be indicated based on the detail in different parts of the scene. Other actions are recommended to enhance the richness of the captured data for better texture, acoustics, lighting, and material reconstruction.

[0054] For example, on the left side of the image, the viewer is asked to look at the marked point to capture more details about the light source. Figure 3E and Figure 3F Example images are shown that instruct the user 30 to walk along a highlighted path 316 and perform actions to optimize image capture. For example, in Figure 3FIn this context, the XR device 300c can guide the user 30 to focus on specific fixed points (e.g., point 320) in the environment as they walk along a highlighted path 326. As the user 30 moves through the space, the application continuously captures spatial data while simultaneously collecting information about the scene's acoustic characteristics and lighting conditions. The application can then reconstruct the XR scene based on the collected information and, through the user interface 322 (e.g., ...), display the scene. Figure 3G The example depicted in the illustration indicates that the user 30's data associated with the scene has been successfully captured. As the user 30 walks along the highlighted path 316, the user 30 is guided to perform specific actions (including focusing the camera on a designated gaze point and adjusting the walking speed based on scene details) to ensure optimal data capture.

[0055] The application can recommend one or more additional actions (such as, but not limited to, focusing on a point to capture detailed lighting information) to enhance data richness. For example, it can prompt the user to focus on a specific point (e.g., Figure 3F Point 320 (shown in the diagram) is used to capture more light source details. Additionally, the application can instruct the user 30 that the capture area has been correctly marked on the ground plane. As the user 30 completes a lap, the application analyzes the captured data and extracts detailed scene semantic and geometric information. Subsequently, the scene semantic and geometric information of the marked area is collected, and the scene type and capture accuracy thresholds (e.g., one or more thresholds for texture, acoustic, and / or lighting accuracy) are determined. The application then determines the scene type and sets specific capture accuracy thresholds for texture, acoustic, and lighting. The captured data (including HDR images with pose markings, depth images, microphone response signals, material classification, signature sound generation times, and other lighting and acoustic information) can then be processed to create a reconstruction. This processing can be performed offline (e.g., by the XR device 300) or in the cloud, ensuring comprehensive and high-quality data for accurate scene reconstruction.

[0056] This comprehensive dataset ensures that the reconstructed XR scenes accurately reflect the real-world environment, thus providing a highly immersive and realistic experience.

[0057] Figure 4 The hardware components of an XR device 230 are depicted, including a processor 230a, a scene reconstruction controller 230b, and a memory 230c. In some embodiments, the XR device 230 may not include at least one of these components, or may include at least one other component.

[0058] Processor 230a includes one or more processing means or circuits, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In some embodiments, processor 120 includes one or more of a central processing unit (CPU), application processor (AP), communication processor (CP), or graphics processing unit (GPU). Processor 230a is capable of performing control over at least one of the other components of XR device 230.

[0059] The scene reconstruction controller (230b) is combined with a processor (230a) and a memory (230c). The scene reconstruction controller (230b) is configured to: obtain information corresponding to at least one of the semantics, geometry, or acoustics of the scene using sensor data when a user moves within the scene during scene capture. The scene reconstruction controller 230b is configured to: activate a user tag for the capture area if the capture area is verified; determine the scene type and at least one capture threshold for capturing the multidimensional scene using a trained model based on the obtained information; generate user guidance for assisting scene capture using the scene type and at least one capture threshold; and capture a multidimensional scene representing at least one of the spatial characteristics, lighting characteristics, or acoustic characteristics of the scene during assisted scene capture using the generated user guidance, wherein the user guidance includes at least one of the following: a path to be followed by the user or at least one action to be performed by the user along the path.

[0060] Memory 230c may store commands or data associated with at least one other component of XR device 230. Memory 230c may include volatile memory (e.g., random access memory (RAM)) and / or non-volatile memory (e.g., flash memory or solid-state drive (SSD)). According to embodiments of this disclosure, memory 230c may include one or more storage media storing one or more instructions. When executed individually or jointly by processor 230a and / or scene reconstruction controller 230b, one or more instructions may cause XR device 230 to perform any combination of the operations described herein.

[0061] Figure 5An example virtual experience is described. In the example scenario, consider a user saying, “I wish I could attend this concert, but I can’t get a ticket?” The embodiments described herein provide users with an immersive virtual concert experience that highly simulates the feeling of attending a live event in person. The embodiments described herein reproduce the intricate details of the concert’s lighting and sound systems, ensuring that users can experience the atmosphere and environment of a live performance with remarkable fidelity. The embodiments described herein capture the visual and auditory nuances of a concert and also extend the experience beyond geographical limitations, allowing friends and loved ones to join in and enjoy the event together, regardless of their location. This enhanced social interaction amplifies the fun and creates a shared, memorable experience, bridging the gap between physical presence and virtual participation.

[0062] Figure 6 An example virtual experience is described. In the example scenario, consider a user saying, "I want to share my experience of this museum exhibition with my friends in an immersive way?" The embodiments described herein capture a high-fidelity reconstruction of a 3D scene. The user can then share it with friends who can view the exhibition immersively from any viewpoint.

[0063] Figure 7 An example virtual experience is described. In the example scenario, consider a user saying, “I want to have a meeting, but my team works in a hybrid mode across multiple offices, and it’s difficult to communicate effectively using only video conferencing.” The embodiments described in this paper offer the possibility of creating a high-dimensional reconstruction of a meeting room, meticulously modeling not only the physical layout of the meeting room but also its acoustics and lighting, ensuring a highly immersive virtual environment. The embodiments accurately capture and reproduce face-to-face interactions, including subtle aspects such as the tone of voice, gestures, and body language of participants in the meeting, all presented at a 1:1 scale. By integrating these elements into the virtual reconstruction, the embodiments effectively bridge the communication gap, allowing for a high degree of mimicry of face-to-face meeting interactions. This level of detail enhances the realism of virtual communication, making it more effective and engaging compared to traditional virtual communication platforms.

[0064] Figure 8An example virtual experience is described. In the example scenario, consider a user saying, "When designing or renovating a house, designing with computer software is time-consuming because I can't clearly visualize the size and structure of the building in my mind. Therefore, I need to be on-site at any time to check on the construction progress." The embodiments described herein can help architects obtain appropriately scaled 3D reconstructions, which aids in design and additionally in acoustic and lighting modeling. This allows architects to experiment with different materials, layouts, and styles, and experience the design from any location within the reconstructed scene. Furthermore, when the architect is not on-site, workers on-site can share the project's current progress in the 3D reconstruction. Moreover, the embodiments described herein significantly enhance the architectural design process by enabling precisely scaled 3D reconstructions, which are crucial for accurate design, acoustic analysis, and lighting modeling. The embodiments described herein allow architects to experiment with various materials and design styles in a virtual environment, providing a comprehensive view from any vantage point within the reconstructed scene. Additionally, the embodiments described herein facilitate real-time collaboration by allowing on-site workers to remotely update and share the project's current progress with the architect, ensuring continuous and effective communication even when the architect is not on-site. The seamless integration of this technology not only optimizes design accuracy and efficiency, but also enhances the overall management and execution of building projects.

[0065] Figures 9A to 9B This is an example scene depicting the difference between spatial video and the proposed joint spatial-acoustic-lighting XR scene reconstruction. Figure 9A The spatial video frame viewpoint is depicted, which is limited to the capture position of the stereo camera. Figure 9B Video frames depicting a scene reconstructed using the proposed joint space-acoustic-lighting XR scene are presented, in which the scene can be viewed from any viewpoint at any distance.

[0066] Figure 10 An example scenario depicting real-time guidance and feedback for scene capture is described. To reconstruct the XR scene, the XR device can begin capturing data from its surroundings. The XR device can suggest a path to follow and one or more actions to perform (e.g., capturing one or more nearby points with the XR device). As the user follows the suggestions, the XR device can collect data from its surroundings and reconstruct the scene based on the collected data.

[0067] Figures 11A to 11B The lighting and acoustics of a reconstructed scene, which can be modified using the proposed joint spatial-acoustic-lighting XR scene reconstruction, are depicted. According to... Figure 11A The environment shown can reconstruct the joint XR experience, and as... Figure 11BThe combined XR experience may include one or more spatial characteristics (e.g., one or more objects in the environment), one or more lighting characteristics (e.g., one or more lights 1102), and one or more acoustic characteristics (e.g., one or more sound elements 1104). One or more characteristics included in the combined XR experience can be customized. For example, one or more lights 1102 and / or one or more sound elements 1104 may be customized (or adjusted) by the user.

[0068] In spatial video, viewers may be limited by the perspective from which the video was initially recorded, typically constrained by the position of the original camera. Therefore, users may be confined to a fixed viewpoint, limiting their ability to dynamically explore the scene. Spatial video technology offers users the possibility to explore a scene from various angles and distances, providing a more immersive and flexible experience. In embodiments, this dynamic exploration is facilitated by a calibrated stereo camera setup that records the scene from multiple viewpoints. The method according to embodiments of this disclosure revolutionizes this approach by eliminating the need for such complex equipment. The method of embodiments of this disclosure allows spatial video to be captured with only a single camera, simplifying the recording process while still enabling users to interact with the scene and view it from different perspectives. This advancement opens up new possibilities for more accessible and versatile spatial video applications.

[0069] The method according to embodiments of this disclosure delivers real-time feedback and proactive guidance to users, enhancing their ability to obtain high-quality XR experiences. By integrating high-dimensional capture within a single frame, users are able to record spatial richness (including pose, acoustic, and lighting elements). Accurate audio reconstruction is crucial for creating realistic and immersive XR experiences because it enhances spatial awareness, fosters social interaction, promotes emotional connection, and provides an understanding of the nuances of the environment. Similarly, accurate scene lighting modeling is essential for realistic rendering, influencing the perception of depth, shape, and texture, thereby ensuring a believable experience. This approach allows users to dynamically control the scene, making applications such as XR design and digital twins more practical and efficient.

[0070] Figure 12A and Figure 12B A flowchart with a real-time user assistance process is described. Figure 12A and Figure 12B A system 200 is described for providing user guidance for reconstructing a multidimensional XR scene using a server connected to an XR device 230. The XR device 230 may include... Figure 12A and Figure 12B One or more modules are shown. The XR device 230 is configured to: (when the user begins capturing the scene) from Figure 2AMultiple sensors 201-205 receive sensor data, use the received data to derive semantic and geometric information about the scene, and (when the region is successfully verified) activate... Figure 2A The XR device 230 uses a user-labeled capture region module 236. This module creates a coarse semantic scene mesh based on the derived information, optionally allowing users to label capture regions. The XR device 230 uses a pre-trained model to determine the scene type and capture accuracy threshold, where the pre-trained model uses texture, spatial, acoustic, and / or lighting properties from the metric semantic localization & mapping module 210, the material type classifier module 212, and the shadow region classifier & reconstruction module 214. The XR device 230, using the pre-trained model, generates user guidance for a second round of scene capture, including paths and actions for the user. During this second round, the user follows the provided guidance to capture multidimensional scene data.

[0071] Figure 13A and Figure 13B A system 400 is described in detail with a flowchart illustrating an XR scene reconstruction process. An XR device 230 may include... Figure 13A and Figure 13B One or more modules are shown in the diagram. The XR device 230 is configured to: create a coarse semantic scene mesh using derived information, determine the scene type and capture threshold for multidimensional data, determine whether the capture area is marked on the ground plane, and highlight the optimal capture path. The capture threshold defines a threshold for the quality of the reconstruction metric for the scene. If the metric or reconstruction quality score from the scene reconstruction algorithm improves above the threshold, the scene is considered to have reconstructed the environment well with the accuracy expected by the user. The threshold is set at the start based on user history, environment, user input, and user preferences. The reconstruction quality of each of these modalities or dimensions depends on these thresholds. The capture path guides the user along the path, providing real-time visual feedback for reconstructing the scene (which includes spatial, acoustic, lighting, and material appearance characteristics), and providing user assistance for creating multidimensional XR scenes (which integrate texture, spatial, audio, visual, and lighting experiences). The XR device 230 is configured to: evaluate data quality, provide guidance based on user profiles and scenes, and / or support default and custom user roles tailored to users and / or applications. Scene segmentation module 222 creates a coarse scene mesh by estimating the semantics and geometry of the scene while the user labels the capture regions. A pre-trained model determines the scene type and sets capture thresholds for multi-dimensional scene data, including texture, spatial, acoustic, and lighting properties. A ground plane verification module verifies that the capture regions are correctly labeled on the ground plane. A path highlighting module (e.g., ...) Figure 12B The path planner module 218 highlights the best capture path for the user. Figure 12BThe user guidance module 220 instructs the user to follow the path and perform necessary actions. The data processing module processes the captured multidimensional scene data to reconstruct the XR scene, while the real-time feedback module provides visual feedback for scene reconstruction. The real-time user assistance module provides real-time support for creating multidimensional XR scenes. The evaluation and guidance module measures the quality of the reconstruction input data and provides real-time guidance based on the user profile and scene. Figure 12A User role support module 211 adapts to different user needs Figure 12A Default and custom user roles 213.

[0072] XR device 230 can use a direction of arrival (DOA) algorithm combined with visual cues extracted from recorded video or images to locate audio sources within a room. XR device 230 can also estimate one or more location-related parameters (such as, but not limited to, dimension and absorption values) from the recorded video or images. XR device 230 can perform object segmentation for identifying material types and their properties, which can be used as initial conditions. Subsequently, XR device 230 can fine-tune these parameters by analyzing the recorded audio response spectrum. XR device 230 can derive the dimension and geometry of a location by combining a sparse simultaneous localization and mapping (SLAM) map with a preliminary dense viewpoint synthesis technique using neural radiation fields (NERF) or symbolic distance fields (SDF). XR device 230 can use a source separation algorithm to separate audio information, and XR device 230 can apply a dereverberation algorithm to these separated signals to mitigate the influence of location acoustics on each component.

[0073] The operations used to model and recreate the listening experience using the proposed method include: estimating the room impulse response (RIR), and user guidance for recording the RIR. RIR is the transfer function between the sound source and the microphone. T60 and the spatial dimension can be used to estimate the RIR, where T60 is the time it takes for the sound to decay by 60 dB. T60 can vary at different locations within the space. The XR device 230 can provide guidance to the user to record the RIR at possible locations within the listener's space (e.g., in front of a TV, chair, etc.).

[0074] The XR device 230 can generate an RIR for each audio source based on estimated source location and room parameters. The XR device 230 can apply these RIRs to the individual audio sources and combine the results to recreate the audio experience at a new listening point, thereby effectively simulating the sound as if heard from this new location.

[0075] Recreating high-fidelity spatial reconstruction involves media capture techniques that ensure accurate representation of textures and occlusions in 3D models, including verifying texture richness and effectively managing occlusions.

[0076] Figure 14A and Figure 14B Examples of surfaces with varying levels of texture richness are depicted, where one surface exhibits a higher level of detail compared to another surface with lower texture richness. Therefore, embodiments of this paper are able to identify regions with different texture richness, which can be achieved by analyzing the texture granularity in the captured data, calculated using a histogram of edge data in the media. To capture media with enhanced texture richness, embodiments of this paper can proactively guide the user during the capture process, directing them to areas / surfaces requiring closer inspection or additional attention. By doing so, the XR device 230 can provide effective guidance to improve the fidelity of spatial reconstruction.

[0077] To detect occluded real-world objects within a scene, XR device 230 can guide the user to a full 360-degree view and perform a brief walk through the space. This initial exploration helps XR device 230 initialize and estimate how objects occlude each other, if any. In embodiments described herein, XR device 230 can use a ray hit test to evaluate occlusion coverage, which involves calculating the intersection of rays with objects and identifying triangles within the field of view occluded by one or more other objects. XR device 230 can estimate occlusion at individual points on a 3D mesh based on the user's viewing direction. By integrating these occlusion estimates with additional matrices, XR device 230 can create and visualize a detailed representation of the occlusion. Embodiments described herein ensure that occlusion analysis captures the complexity of objects in a scene occluding each other from different angles.

[0078] like Figure 15As shown, the spatial reconstruction process involves creating a detailed and accurate 3D representation of the space using a mesh system that records the coverage quality at each point. Embodiments herein also disclose a user interface (UI) to aid in visualizing coverage within the 3D space using a micromap. In embodiments herein, the micromap is characterized by a heatmap that visually represents coverage density. In embodiments herein, the heatmap may be color-coded, where colors are used to indicate the number of observation points; for example, red indicates areas with extensive coverage and multiple viewpoints, yellow highlights areas with less coverage, and so on. Embodiments herein determine a coverage score, which can be encoded into the vertices of a 3D mesh structure, allowing for dynamic rendering of the heatmap from any camera angle, providing an intuitive and interactive way to evaluate the quality of spatial reconstruction. In embodiments, the position of each vertex of the 3D mesh structure can be optimized. The coverage score is determined by analyzing the 3D representation of the space using a mesh system. The system records the coverage quality at each point. The coverage score is encoded into the vertices of the 3D mesh, enabling a heatmap that visually represents coverage density. Colors (such as red and yellow) indicate areas with different coverage levels. Heatmaps can be dynamically rendered from any angle, which helps to assess the quality of spatial reconstruction.

[0079] like Figure 16 The method described herein, according to embodiments of the present disclosure, provides modeling and recreating of lighting within a location based on the direction of a light source. The quality of media capture can be highly dependent on the direction of the light source. For example, the quality of media capture will be better in the direction facing the light source and worse in the direction away from the light source. To compensate for poor capture quality, the user needs to improve the lighting or capture more from the direction away from the light source. For example, applications have a lighting estimation application programming interface (API) that uses cues (such as, but not limited to, shadows, ambient light, textures, specular highlights, reflections, etc.) to provide detailed information about the lighting of a scene. Embodiments herein may determine a quality metric associated with the viewing direction based on a weighted representation of the principal direction of light and ambient spherical harmonics.

[0080] Therefore, users can capture accurate acoustic reconstructions of scenes guided by the proposed method without hiring professional assistance or equipment. Thus, compared to traditional 2D devices such as mobile devices or computers, the aforementioned factors (such as the 3D experience, several new sensors, and the requirement for complex guidance methods) make this method best suited for VST.

[0081] Figure 17This is a flowchart depicting a method (1700) for providing user guidance for reconstructing a multidimensional XR scene according to embodiments of the present disclosure. One or more steps in method 1700 may be performed by an XR device 230. In step 1702, while a user is performing initial scene capture, the XR device 230 (e.g., a head-mounted display (HMD) device) receives data from one or more sensors included in the XR device 230. In step 1704, information corresponding to the semantics and geometry of the scene can be derived from the sensor data during a first round of scene capture. In step 1706, a coarse semantic scene mesh can be created using the derived information while optionally marking capture areas. In step 1708, a pre-trained model can be used to determine the scene type and capture accuracy threshold, wherein the multidimensional scene data includes texture properties, spatial properties, acoustic properties, and lighting properties. In step 1710, user guidance for a second scene capture, including a path to be followed by the user, can be generated based on the scene type and capture threshold. In step 1712, the generated user guidance can be used to capture multidimensional scene data during the second round. The various actions in method 1700 can be performed in the order of presentation, in different orders, or simultaneously. Furthermore, in some embodiments, actions may be omitted. Figure 17 Some of the actions listed in the document.

[0082] Figure 18This is a flowchart (1800) depicting a method for reconstructing a multidimensional XR scene. XR device 230 can reconstruct a multidimensional extended reality (XR) scene. In step 1802, when user 232 optionally marks a capture area, XR device 230 creates a coarse semantic scene mesh by utilizing derived information related to the semantics and geometry of the scene. In step 1804, XR device 230 uses a pre-trained model to determine the scene type and establishes one or more capture accuracy thresholds to capture multidimensional scene data (including texture properties, spatial properties, acoustic properties, and lighting properties). In step 1806, XR device 230 confirms whether the capture area is on the ground plane, and in step 1808, XR device 230 highlights an estimated path for the user to follow for effective scene capture. In step 1810, as the user navigates this path, XR device 230 provides guidance to ensure accurate scene data collection. In step 1810, XR device 230 processes the captured data to reconstruct an XR scene that integrates spatial, acoustic, lighting, and material appearance characteristics into a single frame, thereby providing real-time visual feedback in step 1812 to achieve high-quality, rich XR scene capture with customizable and realistic lighting, acoustics, and virtual object manipulation. Furthermore, in step 1814, XR device 230 provides user assistance for creating multi-dimensional XR scenes by integrating texture, spatial, audio, visual, and lighting experiences within a unified frame. It also serves as a benchmark for evaluating the quality of the reconstructed input data, providing user guidance tailored to individual profiles and scenes, while supporting default and custom user roles to meet different user needs. The various actions in method 1800 can be performed in a presented order, in different orders, or simultaneously. Additionally, in some embodiments, these actions may be omitted. Figure 18 Some of the actions listed in the document.

[0083] The embodiments disclosed herein enhance collected data to improve scene XR reconstruction by refining the geometry, texture, acoustics, and lighting of the scene. The embodiments processed and enriched the raw data to create a more accurate and immersive virtual or augmented representation of the real-world environment, ensuring that the reconstructed scene is highly similar to the physical space in terms of visual detail, sound quality, and lighting effects.

[0084] The embodiments disclosed herein can be implemented by at least one software program that runs on at least one hardware device and performs network management functions to control network elements. Elements include blocks that may be at least one of hardware devices or a combination of hardware devices and software modules.

[0085] The embodiments disclosed herein describe a method for user guidance, which reconstructs a multidimensional extended reality (XR) scene using an XR device and provides real-time feedback and proactive guidance to the user as data is captured. Therefore, it should be understood that when the program runs on a server or mobile device or any suitable programmable device, the scope of protection is extended to such programs, and in addition to computer-readable means having messages, such computer-readable storage means (e.g., computer-readable storage media) contains (or stores) program code means for implementing one or more steps of the method. In at least one embodiment, the method is implemented by or in combination with a software program written in another programming language (e.g., Very High Speed ​​Integrated Circuit Hardware Description Language (VHDL)), or by one or more VHDL or several software modules executed on at least one hardware device. The hardware device can be any kind of portable device that can be programmable. The device may also include means that can be, for example, a hardware device (e.g., an ASIC) or a combination of hardware and software devices (e.g., an ASIC and an FPGA), or at least one microprocessor and at least one memory having software modules located therein. The method embodiments described herein may be implemented partly in hardware and partly in software. Alternatively, the present invention can be implemented on different hardware devices (e.g., using multiple CPUs).

[0086] The description of the specific embodiments above will fully reveal the general nature of the embodiments herein. Others can readily modify and / or adapt these specific embodiments for various applications by applying existing knowledge without departing from the general concept. Therefore, such modifications and adaptations should and are intended to be understood within the meaning and scope of equivalent forms of the disclosed embodiments. It should be understood that the wording or terminology used herein is for descriptive purposes and not for limitation. Therefore, although embodiments herein have been described with reference to examples and examples, those skilled in the art will recognize that modifications can be made within the scope of the embodiments described herein to practice the embodiments and examples disclosed herein.

Claims

1. A method for reconstructing a multidimensional extended reality (XR) scene, the method comprising: When a user of the XR device (230) moves within the scene during scene capture, the XR device (230) uses sensor data to obtain information corresponding to at least one of the following: the semantics of the scene, the geometry of the scene, or the acoustics of the scene; The XR device (230) uses a trained model based on the obtained information to determine the scene type and at least one capture threshold for capturing a multidimensional scene, wherein the multidimensional scene includes at least one of the following: texture characteristics of the scene, spatial characteristics of the scene, acoustic characteristics of the scene, or lighting characteristics of the scene. The XR device (230) generates user guidance for assisting scene capture using the scene type and the at least one capture threshold, wherein the user guidance includes at least one of the following: a path to be followed by the user or at least one action to be performed by the user along the path; and The XR device (230) captures the multidimensional scene during the auxiliary scene capture using generated user guidance, representing at least one of the following: spatial characteristics of the scene, lighting characteristics of the scene, or acoustic characteristics of the scene.

2. The method as described in claim 1, wherein, The operations for obtaining the information include: In response to the user starting to capture the scene in the XR device (230), the XR device (230) obtains sensor data from at least one sensor in the XR device (230).

3. The method as claimed in claim 1 or claim 2, wherein, The method further includes the XR device (230) performing at least one of the following: indicating a gaze point for the user to focus on, indicating a walking speed based on at least one detail of a portion of the scene, or enhancing captured data for at least one of optimized texture, optimized acoustics, optimized lighting, or optimized material reconstruction in the scene.

4. The method according to any one of claims 1 to 3, wherein, The operation of capturing the multidimensional scene by the XR device (230) includes: Process at least one of the following: image-associated labels, image depth, microphone response signals, material classification associated with the scene, sound generation time, lighting information, or acoustic information; and The XR device (230) captures the multidimensional scene based on the result of the processing.

5. The method according to any one of claims 1 to 4, wherein, The multidimensional scenario is determined through the following operations: Generate a map when you begin traversing the environment; Mark the areas and purposes in which the user plans to travel within the environment on the map; Estimate the ground plane, user height, and attitude region in the environment; as well as Occlusion blind spots are estimated by ray propagation from the attitude region of the environment.

6. The method according to any one of claims 1 to 5, wherein, The semantics of the scene are obtained by using a trained object detection and segmentation module, and the geometry of the scene is obtained by using a coarse grid depiction of the scene by employing Simultaneous Localization and Mapping (SLAM) technology.

7. The method according to any one of claims 1 to 6, wherein, The user guidance includes modifying the lighting in the scene to model scene lighting with different light sources by at least one of the following: turning on the lights, turning off the lights, adjusting the brightness of the lights, or changing the color of the lights.

8. The method according to any one of claims 1 to 7, wherein, The capture thresholds include predefined criteria using a pre-trained model, which are optimal conditions for capturing the texture, spatial, acoustic, and lighting characteristics of the scene, wherein each threshold included in the capture thresholds is determined based on the semantics and geometry of the scene using the received sensor data.

9. The method according to any one of claims 1 to 8, wherein, The geometry of the scene includes at least one of the three-dimensional structures of the scene, wherein the three-dimensional structure includes at least one of the following: the shape of the objects in the scene, the distance between the objects in the scene, or the relative position of the objects in the scene.

10. An XR device (230), comprising, At least one processor (230a), including processing circuitry; and The memory (230b) includes one or more storage media storing one or more instructions. in, When executed individually or jointly by the at least one processor, the one or more instructions cause the XR device (230) to perform the following operations: When a user moves within the scene during scene capture, sensor data is used to obtain information corresponding to at least one of the following: the semantics of the scene, the geometry of the scene, or the acoustics of the scene; Based on the obtained information, a trained model is used to determine the scene type and at least one capture threshold for capturing a multidimensional scene, wherein the multidimensional scene includes at least one of the following: the texture characteristics of the scene, the spatial characteristics of the scene, the acoustic characteristics of the scene, or the lighting characteristics of the scene. The scene type and the at least one capture threshold are used to generate user guidance for assisting scene capture, wherein the user guidance includes at least one of the following: a path to be followed by the user or at least one action to be performed by the user along the path; and The generated user guidance is used to capture the multidimensional scene during the auxiliary scene capture, representing at least one of the following: the spatial characteristics of the scene, the lighting characteristics of the scene, or the acoustic characteristics of the scene.

11. The XR device (230) of claim 10, wherein, when operated individually or jointly by the at least one processor, the one or more instructions further cause the XR device (230) to perform the following operations: When the user of the XR device (230) begins to capture the scene, sensor data is obtained from at least one sensor in the XR device (230).

12. The XR device (230) of claim 10 or claim 11, wherein, when operated individually or jointly by the at least one processor, the one or more instructions further cause the XR device (230) to perform at least one of the following: instructing the user to have a gaze point for focusing, instructing a walking speed based on at least one detail in a portion of the scene, or enhancing capture data for at least one optimized texture, optimized acoustics, optimized lighting, or optimized material reconstruction in the scene.

13. The XR device (230) according to any one of claims 10 to 12, wherein, When executed individually or jointly by the at least one processor, the one or more instructions also cause the XR device (230) to perform the following operations: Process at least one of the following: image-associated labels, image depth, microphone response signals, material classification associated with the scene, sound generation time, lighting information, or acoustic information; as well as The multidimensional scene is captured based on the results of the processing.

14. The XR device (230) according to any one of claims 10 to 13, wherein, In order to determine the multidimensional scene, when run individually or jointly by the at least one processor, the one or more instructions also cause the XR device (230) to perform the following operations: Generate a map when you begin traversing the environment; Mark the areas and purposes in which the user plans to travel within the environment on the map; Estimate the ground plane, user height, and attitude region in the environment; as well as Occlusion blind spots are estimated by ray propagation from the attitude region of the environment.

15. The XR device (230) according to any one of claims 10 to 14, wherein, when operated individually or jointly by the at least one processor, the one or more instructions further cause the XR device (230) to: obtain the semantics of the scene by using a trained object detection and segmentation module, and obtain the geometry of the scene by developing a coarse grid depiction of the scene by using a simultaneous localization and mapping (SLAM) technique.

Images