Adaptive dynamic loading strategy for rendering large three-dimensional models

Abstract

To provide an adaptive dynamic loading strategy for rendering large three-dimensional models.SOLUTION: Methods, systems and apparatus including medium-encoded computer program products for loading and rendering include: obtaining a three-dimensional spatial access tree data structure encoding location information about objects in a 3D model of an environment, where the 3D model is stored on a remote computer system; ranking a set of the objects in the 3D model to form an object hierarchy based at least on distances between each object of the set of objects and a specified viewpoint for a user within the environment, as determined using the three-dimensional spatial access tree data structure; selecting a proper subset of the set of objects to be rendered based on the object hierarchy and a current model load limit; downloading the proper subset to a local memory; and rendering the proper subset from the local memory to a display device based on the specified viewpoint within the environment for the user.SELECTED DRAWING: None

Description

[Technical field]

[0001] [Background technology] This specification relates to visualization of three-dimensional models on a client device, and further relates to model data and / or computer visualization systems and techniques used in computer graphics applications, such as computer-generated animation. Summary of the Invention [Problem to be solved by the invention]

[0002] This specification relates to an adaptive dynamic loading technique that determines objects from a 3D model for loading and rendering, taking into account model load constraints for rendering frames of the 3D model by a client device along with the user's position when viewing the 3D model. [Means for solving the problem]

[0003] In general, one or more aspects of the subject matter described in this specification may be embodied in one or more methods (and one or more non-transitory computer-readable media tangibly encoding a computer program operable to cause a data processing apparatus to perform operations) including: obtaining, by a computer having a display device and local memory, a three-dimensional spatial access tree data structure encoding location information for objects in a three-dimensional model of an environment, the three-dimensional model being stored on a remote computer system; ranking, by the computer, a set of objects in the three-dimensional model based at least on a distance between each object of the set of objects determined using the three-dimensional spatial access tree data structure and a defined viewpoint for a user in the environment to form an object hierarchy; selecting, by the computer, an appropriate subset of the set of objects to be rendered based on the object hierarchy and current model load constraints; downloading, by the computer and from the remote computer system, the appropriate subset of the set of objects to the local memory; and rendering, by the computer, the appropriate subset of the set of objects from the local memory to the display device based on the viewpoint in the environment defined for the user.

[0004] By utilizing the described technique, the following advantages are achieved: Based on the adaptive dynamic loading technique implemented for the client application as described in this disclosure, rendering of 3D data on the user device can be facilitated and performed more efficiently, with less computational resource requirements for the user device and without adversely affecting the user's experience. The loading technique does not require the entire 3D model to be loaded, so the solution can work on models of any scale. The loading technique is auto-adjusted based on how the rendering device performs. Based on device capabilities, in some cases more objects can be rendered compared to others. By determining which objects are loaded and rendered according to a dynamic loading strategy, flexible yet efficient switching between different display modes of data and different levels of data rendered during user movement in the 3D space can be supported. Instead of downloading the entire model at the beginning of rendering, which may significantly reduce performance and slow down the client application, it can be iteratively determined whether and which objects are downloaded from the 3D model. Furthermore, the preemptive loading and unloading of objects is based on determining a zone of confidence around the user's viewpoint within the model's 3D environment and taking into account the performance of the user device. Such adaptive loading techniques can improve the performance of client applications and can support smooth rendering during user navigation within the space. In that way, rendering is performed faster and the client application is more responsive to user interactions navigating within the space.Rendering different sets of objects while a user changes his / her viewpoint can be performed faster because the loading strategy identifies objects to be downloaded on the rendering device that are prioritized based on the object's location relative to the user's viewpoint and the current model load constraints of the user device. The loading strategy makes it possible to maintain a ranked list of objects that will be used to determine which objects to download as well as to observe frame-by-second generation and memory consumption on the rendering device.

[0005] The described techniques can support stable rendering performance on a user device that improves the user experience, taking into account the memory capacity and rendering load of the device. The frame generation rate can be variable, and the loading and rendering of objects can be adjusted to match a desired frame generation rate and improve the performance of the display device without adversely affecting the user experience. Furthermore, the loading strategy can support the loading of additional objects that cannot be instantly rendered for a particular scene, and the unloading of objects according to the monitored performance of the user device.

[0006] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the invention will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0007] [Figure 1] 1 illustrates an example of a system that can be used to support visualization of large-scale models, according to an embodiment of the present disclosure. [Diagram 2]1 illustrates an example process for navigating within a 3D model of an environment rendered according to a dynamic object loading strategy that efficiently utilizes resources on a client device for rendering the model, according to an embodiment of the present disclosure. [Diagram 3] 1A and 1B show examples of user interactions for navigating within a spatial view of a virtual reality (VR) environment rendered on a display device by a client device according to an embodiment of the present disclosure. [Figure 4] 1 illustrates an example system architecture of a client device used to render a 3D model based on an adaptive dynamic loading strategy according to an embodiment of the present disclosure. [Figure 5A] 1 illustrates an example of user interaction with a user interface provided by a client application on a user device for navigating within a 3D model of an environment in which objects are rendered according to an implemented adaptive dynamic loading strategy in accordance with an embodiment of the present disclosure. [Figure 5B] 1 illustrates an example of user interaction with a user interface provided by a client application on a user device for navigating within a 3D model of an environment in which objects are rendered according to an implemented adaptive dynamic loading strategy in accordance with an embodiment of the present disclosure. [Figure 6] 1 is a schematic diagram of a data processing system including a data processing device that can be programmed as a client or a server; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Like reference numbers and designations in the various drawings indicate like elements.

[0009] The present disclosure describes various tools and techniques for adaptive dynamic loading of three-dimensional (3D) models to be rendered on a client device. The described techniques support stable performance for rendering (e.g., without crashing the client device due to overload from processing large data portions). Implementations of the present disclosure allow objects to be rendered progressively while a user navigates within a 3D model of an environment and zooms in or out of the model. In some implementations, the rendering can be performed within a virtual reality environment or an augmented reality environment, as well as a combination thereof, where different users through different devices can render and interact with the model.

[0010] In some implementations, the 3D model can be rendered from different devices or platforms, including desktop devices, tablets, virtual reality (VR) devices, and / or augmented reality (AR) devices, among other examples. Computer graphics applications include different software products and / or services that support the generation of representations of 3D objects that can be used for visualization, animation, and video rendering of object models in the context of video games, or in other contexts. Computer graphics applications also include computer animation programs that show views or scenes during user review and navigation within or outside of a model.

[0011] In some cases, 3D computer visualizations can be produced in various scenarios and in the context of different technologies. For example, 3D models of objects, such as manufacturing sites, buildings, physical structures, etc., can be animated for display on the user interface of a native device application, a web application, a VR application, an AR application, among other examples. A user device can display a user interface in which the 3D model of the object is rendered.

[0012] In some implementations, a user may participate in a VR session, which requires that a 3D model be displayed in the context of navigating within the 3D model on a user device, such as a VR device or a laptop. The model to be rendered may be a large model that requires a lot of computational resources (memory capacity of the rendering device) to load and may be rendered completely (or more), which may pose some challenges for execution on small personal user devices that do not have the necessary resources for large model loading). For example, the model may be very large, even if a high-end personal computer with multiple graphical processing units cannot manage to load the model without delay in time. User devices that can be used to load the model may have constraints on their loading constraints. Thus, rendering objects of a large model may be associated with challenges and risks that result in poor user experience or performance (e.g., crashing of the client application used for rendering).

[0013] According to embodiments of the present disclosure, a client application can be configured with a loading strategy that enables objects of a model to be rendered while a user moves within an environment (e.g., a virtual reality environment), which can result in a smooth and flexible workflow supported by efficient tools to flexibly adjust the loading strategy and balance it against the memory consumption of loading objects in real-time and monitoring the current frame generation for rendering.

[0014] People can use different user interface applications to navigate in real time in the 3D space and change the viewpoint for viewing parts of the 3D model. For example, the viewpoint can initially be outside the model, which can be a model of a building, and based on the user's interaction and repositioning relative to the model, the viewpoint can be changed, and the building can be viewed from a closer distance or, for example, from within a particular room or floor of the building. At each step of changing the user's viewpoint, different objects of the 3D model can be relevant for rendering. With this embodiment, objects can be dynamically loaded as the user moves in the environment, and the number of objects rendered as part of the model can be dynamically adjusted based on monitored metrics of the rendering on the device (e.g., monitored frames per second (FPS) generated and memory consumption). The metrics can be taken into account when determining the amount of objects in the visualization, and this amount can be iteratively auto-adjusted while the user moves and changes their viewpoint in the model. Based on monitoring the measured frame rate and memory consumption for the currently rendered scene, model load constraints can be dynamically adjusted to scale the load on the processor to handle the iterative downloading and rendering of objects as the user moves.

[0015] In some implementations, the platform can be accessed through a VR device, such as a VR headset, head-mounted display, smartphone display, or other device, and the VR display can present a VR environment that can be rendered to one or more users wearing the VR headset providing them with an immersive experience of navigating within a 3D model rendered in a review mode within the VR environment. In some implementations, the VR environment can be accessed by users via a VR device, an AR device, and optionally non-VR and non-AR devices, to provide a display of the 3D model within the virtual world. Rendering capabilities for display of the model may differ from device to device, and they can be adjusted to the current rendering capabilities in the particular device.

[0016] FIG. 1 illustrates an example of a system 100 that can be used to support visualization of models without size constraints on the models in an adaptive yet computationally efficient manner by generating a dynamic loading strategy according to an embodiment of the present disclosure. The computer 110 includes a processor 112 and a memory 114, and the computer 110 can be connected to a network 140, which can be a private network, a public network, a virtual private network, or the like. The processor 112 can be one or more hardware processors, each of which can include multiple processor cores. The memory 114 can include both volatile and non-volatile memory, such as random access memory (RAM) and flash RAM. The computer 110 can include various types of computer storage media and programs, which can include the memory 114, for storing instructions of programs running on the processor 112, including VR programs, AR programs, video games, or design interaction applications that support view exploration and collaboration among other examples.

[0017] The computer 110 includes a client application 116, which includes implemented logic for rendering a 3D model of an environment and assisting a user involved in navigating within the 3D model. For example, the client application 116 can be groupware designed to assist a user 190 in a collaborative process with another user using another client application, where both users navigate within the 3D model. For example, the client application 116 can be used by the user 190 to purposefully navigate within the 3D model to show various aspects of the model (e.g., viewed from different viewpoints in the 3D model as defined for the other user) that can be rendered in another client application of the other user. For example, it can be part of a collaborative design process between a design user who uses the client application 116 to navigate within a construction project and a stakeholder of the construction project where views of the model are presented from the outside and the inside at different levels of detail. In some examples, the client application 116 can provide software functionality that facilitates a collaborative process between connected devices on a shared view of the 3D model of the environment. With this embodiment, the loading of objects from the model can be automatically scaled based on monitoring the state and current frame generation rate of the client application 116 while rendering the objects of the model.

[0018] The client application 116 may run locally on the computer 110, remotely on one or more remote computer systems 150 (e.g., one or more server systems of one or more third party providers accessible by the computer 110 via the network 140), or both locally and remotely. In some implementations, the client application 116 may be an access point for accessing services running on a platform 145 (e.g., a cloud platform) that supports the rendering of object model data stored in a database 160 on the platform 145. In some implementations, the client application 116 may implement rendering techniques to coordinate the loading and unloading of objects during changes in a user's viewpoint within a currently rendered model. In some implementations, the computer 110 may impose limitations on the amount of data that can be processed and rendered when the 3D model to be rendered is large and takes a long period of time to load (e.g., minutes or seconds). In some cases, rendering large amounts of data may be delayed or may even crash the memory of the computer 110 if the loading operations overwhelm the computer's ability to process and render a large number of objects. By implementing a rendering technique that coordinates the loading and unloading of objects according to an embodiment of the present disclosure, such problems can be avoided. By using such a rendering technique, objects from a particular scene in a user's field of view in a model can be prioritized and filtered to determine a subset of objects to be downloaded, loaded, and / or rendered. Also, such a technique can support stable rendering performance on a user device that improves the user experience and takes into account the memory capacity of the device.

[0019] In some implementations, filtering can also be based on user-defined rules associated with the type of object to be prioritized, visibility culling, sorting criteria, and / or post-filtering operations that can be associated with computationally expensive filtering such as occlusion culling. In some implementations, user selections that can be used for filtering can include user selection of specific object categories such as walls, windows, etc. The user selection can define whether the selected object is excluded or included. In some implementations, visibility culling can be used to filter objects to download, load, and / or render. Visibility culling can combine multiple volumes, including multiple user fields of view, respective zones of confidence, or other forms of spatial range definitions to determine objects that fit the filter. In some implementations, sorting of identified objects based on one or more criteria can include sorting based on criteria including distance, size, metadata, zones of confidence, material properties such as material transparency, and / or other properties of or about the object. In some implementations, filtering of objects can be performed iteratively using one technique over the other, and in order to minimize computational costs, more computationally expensive techniques can be applied at later stages when several subsets (possibly multiple) of objects are to be considered.

[0020] In some implementations, the client application 116 may request to render a model where the model is large enough that it is not possible to fit into memory and maintain rendering the object at a frame rate that meets the rendering criteria for the user (e.g., a frame rate that is comfortable for the end user and does not cause motion sickness). FPS is affected by the number of calculations performed on the computer 110 and is independent of the refresh rate. In some examples, the FPS rate may be maintained within a predefined range and may be constrained by the number of polygons that can be rendered. For example, it may be determined to maintain an FPS rate of 40-75. In some examples, the FPS rate may be associated with a low likelihood of causing motion sickness to the user and may be determined to be maintained within a determined range that is satisfactory for immersing the user in the model environment as rendered. The frame rate is a frame generation rate that measures how quickly several frames appear within a period of time (e.g., a few seconds). The frame rate can be calculated based on the formula: 1 / (the amount of time it takes to render a viewable model object and execute the logic of the client application 116 for the last frame provided to the display device 120 for display). The frame generation rate is distinct from the screen refresh rate for the display device 120 since the screen refresh rate is fixed for the device, the frame generation rate can be variable, and the loading and rendering of objects can be adjusted to match the frame generation rate and improve the performance of the rendering by efficiently utilizing device resources so that the frame generation rate matches the frame rate of the device. The applied techniques improve the performance of the client application 116 without adversely affecting the user experience.

[0021] In some implementations, when the client application 116 is used to render objects from a model from a 3D model on the database 160 of the platform 145, the client application 116 can obtain a 3D spatial access tree data structure 130 (such as an R-tree data structure), which can be used and processed to inform the loading strategy to select which objects of the model to render when navigating through the view and computer of the 3D model. The loading strategy can be defined to efficiently use the computing resources of the computer 110 to load and unload objects by an object hierarchy that ranks objects according to their priority for rendering and by monitored performance of the current rendering (e.g., measuring memory consumption and current frame rate). In some implementations, for example, priority calculation based on different factors for prioritization (e.g., factors can be optionally weighted) can be performed in the computer 110. In some implementations, the priority calculation for ranking the objects can be offloaded and performed in a remote system or an external service (e.g., a cloud service that implements the logic of the priority calculation).

[0022] In some implementations, client application 116 retrieves model descriptions (e.g., construction designs, floor plans, etc.) from database 160. User 190 can interact with client application 116 to launch virtual worlds (computer-generated representations of the physical world, such as interior spaces of a house representing collaborative spaces, etc.) or join already launched virtual worlds to navigate within the space (e.g., virtual environment) and view the model from different aspects. Rendering can scale any model size based on the implemented dynamic loading strategy as described in this application. For example, larger or smaller objects can be downloaded, loaded, and rendered depending on whether the user is scaling in or out.

[0023] In some implementations, database 160 can store multi-dimensional data including 3D models, image data, floor plans, etc. In some implementations, database 160 can include data about users associated with access rights for reading, writing, modifying, copying, or other operations to be performed when acting on data on database 160. For example, user 190 is defined as a user who has access rights to model 134 (and to the model description in the form of a 3D spatial access tree data structure) according to correspondingly defined privileges for user 190. Access rights and authorizations can be evaluated in an authorization service provided by platform 145.

[0024] In some implementations, the client application 116 can obtain a model description of a model to be rendered on the computer 110 when interacting with the visualization service 165 (which can be implemented as a separate, independent implementation or as part of the implementation of the client application 116). The visualization service 165 can implement logic to build a model description based on a model stored in the database 160, which can be streamed (e.g., in a communication stream) to the client application 116 and used to trigger a loading strategy. In some implementations, the visualization service 165 can provide the model description in the form of a 3D spatial access tree data structure 130, which can be used to generate a loading strategy using information about the currently generated FPS and memory usage for rendering on the user device used to display the 3D model, along with loading related factors such as the user's viewpoint and distance between objects in the field of view. The loading strategy can be applied to the 3D spatial access tree data structure 130 to identify objects within a particular volume in the model where the user is located. In some implementations, the loading strategy can include loading related factors in combination to determine a priority for loading objects. The combination of factors can be weighted in combination where different factors can be defined with different weights. For example, the user's model size and position can be two factors defined with different weights and used to calculate a priority (or rank), which can be used to sort objects according to their priority and prioritize the loading of objects and their rendering on the user device.In some embodiments, objects from the model can be downloaded and cached locally based on a determined priority for rendering, which is defined according to the current model load constraints for model rendering by computer 110, along with priority ranking rules based on tree data structure 130.

[0025] In some implementations, when a user is viewing a model and moving within the virtual environment, the user may have a viewpoint that is outside of the model, in which case the model may be seen as small. In those cases, prioritization of the objects of the model to be loaded and rendered is for objects that are relatively larger sized objects to form the shell of the model, and lower priority is given to smaller objects. The objects may be ordered based on their priority ranking, and those objects that fit the current model loading capabilities may be determined and downloaded for rendering. In some implementations, as the user navigates within the virtual environment, objects within the user's field of view (within a threshold distance around the user's field of view) may be prioritized for loading and rendering, and even smaller objects may be loaded while keeping at least a portion of the objects that form the shell of the visualized space (e.g., a floor or room of a building) also loaded. The ranking of the objects may be dynamically changed based on the user's repositioning and based on monitoring memory consumption and frame generation rate (e.g., within a given period, e.g., between two updates of the loading strategy).

[0026] In some implementations, the client application 116 can be operated using one or more input devices 118 (e.g., a keyboard and a mouse) of the computer 110. Although shown as separate devices in FIG. 1, the display device 120 and / or the input device 118 can also be integrated with each other and / or with the computer 110, such as in a tablet computer (e.g., a touch screen can be an input / output device 118, 120). Moreover, the computer 110 can include or be part of a VR or AR system. For example, the input / output devices 118 and 120 can include a VR / AR input controller, a glove, or other hand-operated tool 118a, and / or a VR / AR headset 120a. In some implementations, the input / output devices can include hand tracking devices that are based on sensors that track movements and reproduce interactions as performed by physical input devices. In some implementations, the VR and / or AR devices can be standalone devices that do not need to be connected to the computer 110. The VR and / or AR devices can be standalone devices having processing capabilities and / or an integrated computer, such as computer 110, having input / output hardware components, such as controllers, sensors, detectors, etc. The VR and / or AR devices can either be connected to computer 110 or can be standalone devices that integrate a computer (having a processor and memory) and can communicate with platform 145, and can immerse a user connected through those devices into a virtual environment, in which 3D models of objects can be presented within a simulated real physical environment (or a substantially similar environment), and the user navigates within the environment and is presented with different visual objects.

[0027] In some implementations, the system 100 can be used to display data from 3D documents / models that can be used to generate a virtual world (which can be a VR environment for one or more first users, an AR environment for one or more second users, and a non-VR or non-AR environment for one or more third users) that is presented within a corresponding interface of a client application 116, which corresponding interface enables the user to use the data to navigate and interact with the environment to move to different locations within the virtual world's space.

[0028] In some embodiments, platform 145 can be a cloud platform that can be accessed to support cloud services related to data visualization in different display modes (such as visualization service 165, which can be implemented as part of platform 145) and can support tools and techniques for interacting with the data to navigate in the display modes.

[0029] In some implementations, the user 190 can cause the computer 110 to render a view 133 of the model that is presented on the user interface 132 of the display device 120 and visually indicates how the user (or another user) interacts with the view, where the view is generated based on processing the 3D spatial access tree data structure 130 for the model. The display device 120 can render a particular view of the virtual world while the user navigates and changes his / her position within the model. In some implementations, the user 190 can be provided with the view 133 as a presentation of a view of the 3D model 134 via a VR device or other AR device, where the user's view of the model on the user interface 132 can change while the user navigates and interacts with the model. For example, the view 133 can change based on the user interaction with the interface 132, moving the user closer to the model.

[0030] According to embodiments of the present disclosure, the rendering of model 134 and at least some of the objects of model 134 may be performed based on a user's viewpoint (including the user's position and orientation), an object hierarchy as defined based on ranking objects according to prioritization criteria for rendering the objects (e.g., the ranking is performed based on a 3D spatial access tree structure), and a current model load constraint. In some embodiments, the current model load constraint may be kept up to date by monitoring the current frame rate for rendering and adjusting the model load constraint to scale the rendering to the frame rendering rate.

[0031] In some embodiments, the loading strategy monitors the system's health, and the quantity thresholds for the loaded objects can be adjusted based on the degree of data complexity that can be rendered (e.g., some objects are more complex and associated with higher processing costs, such as transparent objects or complex material objects). For example, if the FPS at a given time is lower than the defined model load constraints for some frames, it can be understood that the system has reached the rendering capacity constraints that can be supported. In some examples, the model load constraints for the number of FPS can be adjusted and reduced based on the distance between the current load constraints and the average value of the currently monitored FPS rendered during the last monitored period. In some examples, the distance can be determined as the normalized distance between the minimum satisfactory FPS targeted by the system, multiplied by the current amount of loaded geometry and the average FPS value during the currently monitored period. In some embodiments, the loading strategy applied to determine whether to render a degree of objects can be implemented to increase the positive trend in rendering and add the ability to slow down the negative trend. In such cases, when it is determined that the data will be unloaded according to the observed performance on the user device (e.g., a higher frame generation rate compared to the threshold level considered to start unloading), the loading can be accelerated by increasing the speed of loading for which there is a positive trend in the loading performance on the user device, and the loading can be slowed down to decrease the speed. Such automatic adjustment of the loading strategy can be performed based on the currently observed FPS rate and the target FPS rate on the user device.Additionally, the automatic adjustment can be configured to perform unloading more slowly when it is determined to begin unloading and to increase the speed of loading when it is determined that more objects can be loaded based on the current frame rate. As described, loading and unloading is performed in an adaptive manner that optimizes computing resources and is balanced against the frame rate per second experienced at the user device.

[0032] The systems and techniques described herein are applicable to any suitable application environment capable of graphically rendering any portion of a virtual world, including objects therein. Thus, in some implementations, for example as described in connection with Figures 3A, 3B, and 5, model data from a model stored in database 160 can be used to render objects in different modes and can be accessed from multiple types of devices, allowing a user to navigate within the virtual environment in which the model is rendered.

[0033] FIG. 2 illustrates an example process 200 for navigating around and within a 3D model of an environment rendered according to a dynamic object loading strategy that efficiently utilizes resources on a client device to render the model, according to an embodiment of the present disclosure.

[0034] In some implementations, navigation within the 3D model can be in the context of user interaction in a video game. In some implementations, when a game is built, the created objects and scenes are used to generate a three-dimensional model that can be defined for different layout levels and based on a scripted game plan. The game can be accessed from a computer for rendering. In some implementations, the rendering can be in a virtual 3D model environment in which a user is positioned. In some implementations, the model of the game can be built to include tiles that can be dynamically loaded while a user is playing the game on a computer. When a 3D model of a game is rendered and the model is a large model, fully rendering the model can be a time-consuming and resource-expensive task. Model loading may not be possible to be performed within the cache of the computer, for example, the computer may be a computer of a user device that does not support heavy computation and large data storage.

[0035] By increasing the complexity and size of the three-dimensional model, the scalability of rendering the model may be diminished, which may affect the performance and user experience during rendering, for example, during a VR game. In the case of increasing model complexity and rendering data, costly and limited hardware may require consideration of optimizing object loading and unloading. In some implementations, when a game is rendered, not all objects may need to be loaded initially, but loading and rendering can be optimized and dynamically adjusted to the current model loading constraints, which can be automatically adjusted to the user's position in the environment and the performance of the computer rendering.

[0036] In some implementations, 3D models can be generated and used for rendering in various contexts. In some implementations, a user can generate, create, or upload a model of a physical entity or environment, including various objects and spatial organization. The generated model can be provided by the platform and can be accessible through various access points, such as different user interfaces that can be accessed via desktop devices, mobile devices, other devices, services, or other entities. For example, the 3D model can be a 3D model of a building, including 3D geometry, metadata, and floor plans. Based on the spatial distribution of the model, there can be multiple hierarchies in the 3D geometry (e.g., starting from the outside, bounding volumes of each part of the model), but there can also be multiple hierarchies for the metadata (e.g., building→floor→room).

[0037] At 210, a 3D spatial access tree data structure is obtained (received or generated) by a computer having a display device and local memory. In some implementations, the 3D spatial access tree data structure is substantially similar to the 3D spatial access tree data structure 130 of FIG. 1. The 3D spatial access tree data structure is a data structure that includes a model description for a 3D model, including location information for objects within the 3D model of an environment. The 3D spatial access tree data structure is obtained at a computer that is a user device used to render the 3D model. The 3D model is stored on a remote computer system, for example, a platform such as platform 145 of FIG. 1.

[0038] In some implementations, generation of the 3D spatial access tree data structure can be performed on the computer or can be invoked from an external service (e.g., a cloud service), which can generate the tree data structure based on model data for the 3D model. In some implementations, the 3D spatial access tree data structure can be received from another computer, which can be different from the remote computer system on which the 3D model is stored. Obtaining can include streaming the data structure from the other computer and / or through a service, and the data structure can be generated from the received model description of the 3D model.

[0039] In some implementations, when a user of a computer initiates an opening of a model, a 3D spatial access tree data structure can be obtained for the 3D model, and model description data can be collected and cached in the computer. The 3D spatial access tree data structure can be used to initialize a loading strategy for rendering objects of the model in the computer. The 3D spatial access tree data structure can be constructed based on the obtained model description data for the 3D model. The 3D spatial access tree data structure can include a bounding volume hierarchy that is computed to include nodes corresponding to bounding volumes defined for each spatial portion of an environment (e.g., a floor or a room of a 3D model of a building). The bounding volume hierarchy can include leaf nodes corresponding to objects in the 3D model of the environment. The bounding volume hierarchy can be constructed as a structure that can be queried to support efficient spatial queries that are performed when determining which portions of objects of the 3D model to load for rendering based on a user's position within the 3D model in the environment. When a computer determines what to render to a user positioned in proximity to a 3D model in a virtual space, the computer can efficiently determine the set of objects to be loaded and rendered based on the user's position, the size of the model, and the processing capabilities of the user's device (e.g., model loading constraints for rendering a frame on the computer).

[0040] At 220, the set of objects in the 3D model is ranked to form an object hierarchy based at least on a distance between each object in the set of objects and a defined viewpoint for the user in the environment. In some implementations, the defined viewpoint for the user can include a location of the user and an orientation of the user's camera at that location.

[0041] In some implementations, the set of objects can be determined by querying the tree data structure to determine objects that are within a "zone of trust" around the user's location in the environment. Such a zone of trust can be thought of as an area within a threshold range around the user's defined viewpoint in the environment. In some implementations, the zone of trust can be defined as a volume of a given shape (a sphere, capsule, or frustum, among other example shapes). In some implementations, the zone of trust can be thought of as a room of a building (for example, a 3D model) that the user's camera view (e.g., the user's avatar in the 3D environment) is currently viewing from inside the room. In some implementations, once the set of objects is determined, the objects can be ranked based on a calculated distance between each object and the user's defined viewpoint in the environment, such that closer objects can be given a higher ranking (or priority). The distance between the object and the user's defined viewpoint can be determined using the 3D spatial access tree data structure and location information for the objects in the model.

[0042] For example, when a user's viewpoint is defined, objects that fall within the user's direct field of view can be determined, but there may be objects in direct proximity to the user that do not fall within the user's field of view (e.g., directly behind the user), or there may be objects that are obscured by other objects but are still in close proximity to the user. In some implementations, querying the tree data structure can provide a set of objects that fall within the user's zone of trustworthiness and can be either visible or obscured to the user from the user's defined viewpoint. In some implementations, the loading strategy can be configured to determine the zone of trustworthiness as an area around the user's avatar in the 3D environment (the user's position viewing the 3D model in the environment). In some cases, the zone of trustworthiness can be defined to include objects that are outside the viewing perimeter of the user's camera but in close proximity to the user's position (e.g., defined in relation to a threshold distance for filtering close objects). For example, if a user is currently located inside a room of a building and has a heading towards the interior of the room, the zone of confidence for the user may be determined to include objects that are directly behind the user and are not visible to the user, but if the user changes direction or switches the user's orientation, such objects may quickly become part of the user's viewpoint and become visible to the user. Further, for example, some objects may be occluded even if they fall within the area around the user and the user's viewpoint. Although such objects may not necessarily need to be rendered immediately because they are not visible from the user's current viewpoint, it is conceivable to include them within the zone of confidence due to proximity or other factors and take them into account for ranking and prioritizing which objects to load.In some embodiments, objects from the reliability zone are ranked according to a loading strategy, which can be configured to take into account the occlusion and rendering of objects that are not the closest within the user's viewing zone. Such preemptive loading, which can be done taking into account the user's position, model and object size, as well as the FPS and memory consumption of the rendering device, can support smooth rendering during the process of the user moving within the 3D space where sudden changes in direction may occur. A query for a set of objects is executed to determine all objects of 3D models (and each leaf node of the tree data structure) that are to be rendered within a defined volume (e.g., frustum, capsule, or others). In some embodiments, an occlusion mechanism can be implemented such that it is possible to determine that objects that are within the reliability zone but not visible need to be ranked lower. The occlusion mechanism may be executed before objects are prioritized and a selection of objects for rendering is made.

[0043] In some implementations, a set of objects are provided with a rank corresponding to a priority for their loading. The priority can be defined according to priority rules (and prioritization criteria), which can be used to weight the model size and the distance of the object to the user's position. Furthermore, in some cases, the priority for objects within the zone of confidence can be calculated taking into account the visibility of the object from the user's defined viewpoint (e.g., falling entirely within the "zone of confidence" that is occluded or partially occluded by other objects). Thus, objects that are in close proximity to the user but fall behind another object and are obscured (partially or completely hidden behind the object or based on the shape of the volume) and / or behind the user and do not fall within the viewing area but are still close can be considered by the loading strategy and ranked according to the loading strategy. In some implementations, the prioritization criteria for object loading can define factors associated with prioritizing the loading of a first object compared to the loading of another object. In some implementations, the prioritization can be based on user-defined criteria related to object characteristics. In some cases, metadata about the object may be stored including the object's category, e.g., wall, staircase, pipe, etc. In some further cases, the metadata about the object may also include metadata about the object's location within the environment, such as the current floor or room in which the object is located.The factors include a comparable ratio between the size of the object and the size of the model, the user's defined viewpoint, the distance between each object of the set of objects and the viewpoint for the user, the user's location within the three-dimensional model, a comparable position between each of the two of the objects in the environment from the user's viewpoint, and the physical properties of the objects (e.g., the object made of the material through which it will be seen, i.e., transparency, shader complexity, or other precise metrics for the physical properties of the object). For example, based on the user's location, a set of objects may fall within the user's direct viewing surroundings, where some objects located within that area may be occluded by other objects. In some cases, the occlusion may be partial such that some objects can overlay some other objects without completely covering them. Furthermore, in some cases, some objects may have physical properties such as being transparent or reflective (e.g., they may be semi-transparent or fully transparent materials (e.g., windows, glass doors, crystal bases, etc.), or reflective as a mirror). When generating a loading strategy, some or all of the factors can be considered, in some cases even in case of dynamic modifications, which factors to include in the loading strategy, iterations can be determined during runtime. The prioritization criteria can be defined as a weighted combination of two or more defined factors, each factor of the two or more factors being associated with a predefined weight for the combination.

[0044] In some implementations, the objects determined at 220 can be filtered to provide a subset of objects that are relevant to the current rendering, so that loading can be performed on that subset of objects as they are objects of higher importance.

[0045] At 230, an appropriate subset of the set of objects is selected by the computer to be rendered based on the object hierarchy and the current model load constraints. The selection is performed based on filtering those of the objects that have the highest priority (as determined by the ranking) and that meet the current model load constraints.

[0046] In some implementations, the determination of the appropriate subset may be performed in an external service or platform where the computer can exchange information about the objects to be visualized with a visualization service or platform (such as visualization service 165 of FIG. 1 or platform 145 of FIG. 1). In some cases, the filtering and determining which objects are rendered by the computer on the display device may be performed in such an external service or platform.

[0047] At 240, the appropriate subset of the set of objects is downloaded to a local memory of the computer. At 250, the appropriate subset of the set of objects is rendered from the local memory to a display device based on a defined viewpoint within the environment for the user.

[0048] At 260, the current model load constraints based on the current frame generation rate for rendering are updated. In some implementations, the current frame generation rate is monitored and updates to the load constraints are performed periodically or dynamically based on identifying an event, such as, for example, based on detecting a change in the rate that exceeds a threshold.

[0049] At 270, downloaded objects cached in the local memory can be removed from the cache based on the object hierarchy (e.g., respective ranks for the objects) and in response to reaching the current model load constraint within a threshold distance as a result of the download. In some implementations, the downloaded objects are a proper subset that includes some objects that reach above or close to the load constraint for rendering the frame. Thus, the loading strategy can include a dynamic unloading step that includes lowering the model load constraint that may result in a lowering of the number of objects to be held in the computer's cache. Thus, at least a portion of the already downloaded objects can be removed in response to reaching the current model load constraint within a threshold distance.

[0050] For example, the current model load constraint can be defined as an FPS rate and memory usage percentage of the CPU / GPU that should not be exceeded. In some examples, the FPS rate can be set as acceptable within a certain range, e.g., 40-75 FPS, and the memory usage can be set as acceptable up to 80% to allow the user device to smoothly render objects during user navigation in the space even if some of the objects that need to be loaded have not been preloaded. In some implementations, the threshold distance defined around the current model load constraint can be defined as the distance between the highest acceptable FPS and the memory usage and percentage for the currently observed metrics. For example, the distance between the highest acceptable FPS and the currently observed FPS can be set to 5 or 10 FPS, so that when the FPS reaches 65 or 70 FPS, the loading strategy can consider starting to remove objects, as discussed in connection with step 270. The threshold distance can be defined as a percentage below a maximum acceptable value (e.g., 80%), e.g., below a maximum acceptable load on the CPU / GPU memory, which can be 5% or 10%. It will be appreciated that these are merely example thresholds and example maximum acceptable metrics that can be configured to be different values ​​and used in the loading strategy to determine when and if to remove objects as described in 270.

[0051] At 280, it is determined whether the user has changed their viewpoint. For example, it may be determined whether the user has changed their orientation without changing their location, whether they have changed their location, whether they have changed their location and maintained the same viewing direction, or whether they have changed their location and viewing direction. If the user has changed their location and / or orientation, the steps of selecting, downloading, rendering, and removing are repeated iteratively as changes to the defined viewpoint in the environment for the user are received. As the user navigates in the environment and changes the user's viewpoint, a different set of objects is associated with the ranking as the distance between the objects and the user's new viewpoint is changed. The steps are repeated iteratively during the user's interaction and movement with respect to the 3D model. If it is determined that the user has not changed their viewpoint, at 280, the rendering of an appropriate subset of the set of objects is maintained. In some examples, the determination that the user has changed their viewpoint may be made based on receiving notifications for the user's interaction with the 3D space, moving and / or changing orientation within the 3D space.

[0052] 3A and 3B show example user interactions for navigating within a spatial view of a VR environment rendered on a display device by a client device, according to an embodiment of the present disclosure.

[0053] In FIG. 3A, a user interface of a client application, such as the client application 116 of FIG. 1, is presented in which a 3D model of an environment is rendered. In some implementations, the model rendering is performed in response to a user interaction requesting to access the 3D model from another platform or service. In some implementations, a user can access a platform that stores the 3D model and data for visualization of the model. Access to the 3D model can be through different access applications or through a web browser (or other access point). In some implementations, a user can enter a VR environment representing a model of a building, which is rendered in a user interface. The user interface can provide tools and options for rendering the model or parts of the model while the user moves within the environment. In some implementations, the model is rendered from the outside and from the user's viewpoint, as shown in FIG. 3A. In some implementations, an application used to render the objects can be configured to perform the method of FIG. 2 to render the objects of the model according to a dynamic loading strategy that takes into account the user's viewpoint and the model loading constraints of the client device used for rendering.

[0054] In some implementations, the rendering in Figures 3A and 3B is based on obtaining a 3D spatial access tree data structure and ordering and filtering objects based on an object hierarchy defined according to priority rank and current model load constraints. The obtaining of the tree data structure as well as the ordering / ranking and filtering / selecting can be performed in a manner substantially similar to that described in connection with steps 210, 220, and 230 of Figure 2.

[0055] In some implementations, a user can navigate within the space and define a location within the model of interest to view. For example, the definition of the location can be performed based on an interaction, such as a hand-based interaction with a VR user device or a hand-based interaction with a computer mouse and keyboard. In some implementations, using sensors on the user's headset to interact with the VR environment, the application can map the user's real-world desk to their virtual desk within the virtual experience when viewing the 3D model. In some implementations, a virtual user interface can be provided to allow the user to select a model for review within the virtual space, and the user can point to a portion of the model to switch the user's location.

[0056] In some implementations, different views of the 3D model can be provided based on the user's distance to the objects of the 3D model and the current load constraints for rendering frames by the user device. In some implementations, the decision to load and render objects while the user repositions themselves with respect to the model of the environment can be performed dynamically according to embodiments of the present disclosure, for example, as described in connection with FIG.

[0057] For example, Figure 3A includes a presentation 300 of a 3D model shown from a greater distance from the user's viewpoint as compared to the distance from the user's viewpoint in Figure 3B. In some cases, the presentation in Figure 3A can be referred to as a "dollhouse model" or scaled model" view. As the user moves further away from the model in Figure 3A, the objects determined to be rendered as described in Figure 2 can be those objects that match the proper subset as selected in Figure 2 230, which is the set of objects having the highest rank for priority for rendering and are selected based on the current model load constraints.

[0058] According to an embodiment of the present disclosure, a loading strategy can be generated for a 3D model such that at a point in time and at a position and orientation of the user in the environment, the user is presented with a representation 300 of the 3D model. As discussed, this is the representation 300 as seen from the outside of the model building. In the example of FIG. 3A, when the user's position is outside the building, the user may not see all or part of the objects inside the building, even when they are not occluded. For example, the building may have windows, which may be transparent, such that it is entirely visible through the windows and objects in the rooms that span across the floors can be seen (e.g., the user should be able to see the stove in the kitchen (or other room) through the building's windows). However, a loading strategy for rendering the building representation is generated to prioritize the building shell (as it is closer to the user's position and falls within the user's zone of confidence) in the view of the distance between the building and the user. Thus, when the building is rendered, objects such as those ranked from the user's zone of confidence can be filtered based on the limitations of the rendering device. As previously discussed, the filtering can be by consideration of whether enough memory is available to load the interior objects of the building, identify the parts of the objects that are ranked with a higher rank for relevance (priority) for rendering, and render them first as shown in Figure 3A. The objects presented in Figure 3A are the objects that are considered to be closer to the user's parts and define the shell of the model, and thus are ranked higher as they are larger objects.

[0059] In some cases, while viewing the representation 300 of a building model, a user may be interested in navigating closer to a portion of the model, for example, to the bottom right region 305 of the building. The portion of the building shown on the representation 300 is rendered to include larger objects that can be seen from the outside, while smaller objects inside the building are not shown, even if they can be seen through the windows. To view that portion of the representation in more detail, the user may navigate in the 3D space to reposition himself or herself and adjust the viewing direction. For example, the viewing point (i.e., the user's location and viewing direction) may be moved to the user's viewing point associated with the representation 310 of FIG. 3B. In that new representation, different objects associated with that portion of the model are rendered based on a loading strategy. Due to the change in the user's distance to that portion of the model, smaller objects that fall within the zone of confidence of the new viewpoint may be considered and ranked according to the loading strategy.

[0060] In some further cases, and as will be further discussed in relation to Figures 5A and 5B, when a user navigates in 3D space and is inside a room of a building (e.g., the building of Figures 3A and 3B), the user may not see objects outside the building through the window (as shown in Figure 3A) even if they are not occluded (e.g., trees outside the building that can be seen through a window in the wall of the room) without moving closer to the window (Figure 5B) or orienting the user's view towards the window. Such prioritization of objects that will or will not be rendered is the result of performing a loading strategy, in which the determination of a zone of confidence for a given viewpoint may cause all nearby objects (small or large) in the room to be downloaded, which may take up memory and does not allow space to load objects outside the room that should be visible based on the viewpoint. In some cases, if the user changes the user's location, for example, closer to a window, trees outside the room may move into the zone of confidence and may be considered for loading and rendering according to embodiments of the present disclosure.

[0061] The object presented in FIG. 3B is considered closer to the user's new position, which is closer to a particular part of the model, and thus from that position of the user, a set of different objects can be considered as relevant and can be ranked based on their distance from the user's position. In the case of FIG. 3B, smaller objects can be rendered because they are located closer to the user's position. As described in this disclosure, the decision of which object to render is made based on the object hierarchy determined for objects that fall within a zone of confidence around the user's field of view and the load constraints for the number of frames to be rendered. The rendering can be dynamically auto-adjusted by adjusting the selection of objects to be downloaded and rendered based on the current model load constraints. Furthermore, the rendering of frames can be monitored and the constraints can be adjusted according to the current frame rate while keeping the frame rate within a range that does not affect the user experience (e.g., causing motion sickness). Thus, 3D models can be loaded incrementally on a defined pedestal in the user interface at a sufficient level of detail without impacting application performance (e.g., ~60 FPS and low latency) and always showing a high-level representation of the model, such as a complete "shell" (or external view) of the model.

[0062] In some implementations, a new ranking of objects in the environment is performed based on determining a change in the user's location in the environment, as may be determined in FIG. 3B, a subsequent view of the environment after rendering the view of FIG. 3A, and the ranking may be repeated for new objects, which are determined based on querying the tree data structure and performing the ranking according to a prioritization criterion. In some implementations, the repeated ranking may be for objects that at least partially include previously loaded objects for the view of FIG. 3A, but overlapping objects may be ranked with different priorities based on changed locations and potential indications of (or updates to) model load constraints. If an object previously ranked and downloaded for the rendering of FIG. 3A is also selected for loading and rendering in FIG. 3B, the object may not need to be downloaded as it may still be in the device's cache. Thus, if the loading strategy can support the loading of additional objects that have not been immediately rendered for a particular scene, and the loaded objects are relevant for a subsequent scene, rendering operations may be optimized and performance may be improved.

[0063] In some implementations, loading objects of the 3D model and rendering them on the display device can be determined at least in part based on monitored measurements of frame rate / second when rendering a view from the model along with memory consumption required for rendering. In some implementations, the level of detail to be applied when rendering the 3D model can be dynamically determined based on dynamic adaptation or adjustment to the loading based on the current frame generation rate for rendering. Such dynamic considerations can support faster loading and slower unloading of objects downloaded to the cache to support efficient utilization of resources, which allows for rendering already cached objects first (and thus rendering faster) compared to objects not yet downloaded but associated with new positions and repositioning of the user during interaction with the environment.

[0064] In some implementations, when a 3D model is presented, a portion or all of the 3D data associated with the model can be downloaded and used for rendering. For example, different levels of detail associated with rendering a 3D model may affect the amount of data downloaded for data visualization. By reducing the amount of data to be downloaded, network bandwidth can be conserved. In some implementations, a user may configure the level of detail associated with the presentation of data in a particular user interface used by the user to show the 3D model, and based on such configuration, some, but not all, of the stored 3D data for the 3D model can be downloaded. Such filtering of the amount of data to be downloaded can be dynamically adjusted or iteratively trained based on an evaluation of user interactions.

[0065] 4 illustrates an example system architecture 400 of a client device used to render a 3D model based on an adaptive dynamic loading strategy, according to an embodiment of the present disclosure. In some implementations, the client application 460 can be substantially similar to the client application 116 of FIG. 1, and the client application 460 can be configured to perform at least a portion of the steps of the method 200 of FIG.

[0066] In some implementations, the client application 460 can be configured to execute a dynamic adaptive loading strategy to determine which objects of the 3D model to download and render locally. The loading strategy can be defined to scale the rendering of large models that cannot be downloaded completely into the cache. In some implementations, the client application 460 can request to present a model from the 3D model 415 of the platform 405. The platform 405 can be substantially similar to the platform 145 of FIG. 1 and can be considered as the remote system of FIG. 2 where the objects of the 3D model can also be downloaded.

[0067] In some implementations, the client application 460 can open the model from the 3D model 415, fetch the model description, and generate a 3D spatial access tree data structure that encodes location information for the objects in the 3D model, or request to obtain the 3D spatial access tree data structure directly. In some implementations, the client application can download it directly from the platform 405 if the platform generated it, or from the visualization service 410, which can connect with the platform 405 and generate the tree data structure from the model 415.

[0068] In some implementations, the visualization service 410 can be implemented as a separate set of services or can be integrated into the logic of the client application 460. The visualization service 410 can include a data source integration service 425 that can be configured to collect model description data for the model(s) from the model 415 and provide the model description data to the data production service 420. The data source integration service 425 can provide an abstraction layer on top of the data source (e.g., the source of the model) to obtain identifiers for faces and vertices in the model along with properties associated with objects and implementations in the model. The data source integration service 425 can be used in conjunction with the data production service 420 to query data associated with the model and determine the level of detail of objects from the model.

[0069] The data production service 420 can be used for data processing optimization and prioritization decisions for loading objects, or can be configured to implement logic to pre-calculate some properties from metadata about the 3D models that cannot be used when performing object selection (as described in connection with steps 220, 230, and 240 of FIG. 2). The data source integration service 425 can collect properties about the models and objects, including data about textures, data about material details, data about pixel colors, or other data. In some implementations, such data about textures, such data about material details, and other metadata about the 3D models can be stored in an asset store 450 accessible by the visualization service 410. In some implementations, the spatial indexing service 430 can generate a 3D spatial access tree data structure (e.g., a bounding volume hierarchy). In some implementations, the data creation service 420 can communicate with an asset store where files for different three-dimensional scenes and models are stored (e.g., in various file formats such as glTF, DirectDrawSurface (DDS) files, or another format).

[0070] In some implementations, the data source integration service 425 can be communicatively coupled to a spatial indexing service 430. In some implementations, the spatial indexing service 430 can generate a tree data structure that includes the objects of the model as leaves along with their location within the model. The spatial indexing service 430 can initialize the hierarchical information and store it locally or in a remote storage device, and the generated tree data structure can be queried to obtain prioritized objects within a given volume. By querying the generated tree data structure, an unordered list of results of a set of object identifiers can be obtained. In some implementations, the list can be used to obtain objects based on the data in the data source integration service.

[0071] In some implementations, when a client application 460 connects with the visualization service 410, a loading web socket 470 of the client application 460 can be used to connect with the visualization service 410 to open a model and render objects on a display device (not shown). Upon receiving a request at the visualization service 410, the spatial indexing service 430 can generate a list of unordered identifiers of objects associated with a given viewpoint of the user and the requested model, and can provide the list to the prioritization service 440, which can perform ranking (e.g., substantially similar to the ranking described for step 220 of FIG. 2). The prioritization service 440 can implement logic to sort a set of objects based on a set of prioritization rules, which can be defined for the client application 460 (as an external input for a particular rendering in the client application or as a configuration for a default setting in the client application). In some implementations, the prioritization rules can be defined in the client application at different granularities for different target users (e.g., user roles, user groups, etc.).

[0072] In some implementations, the rendering of the 3D model's objects can be performed in the context of a user interacting with another user in space (e.g., playing a game, collaborating on a task), the other user interacting with another device on which the 3D model's objects are displayed. In some implementations, each of the two users navigates in the 3D space and the rendered objects are displayed as determined on each of their devices. The two users can rely on loading strategies used separately to determine which objects to load. In some implementations, a zone of trust (as previously discussed) can be determined for each of the two users. In some cases, the zone of trust of one of the users can be determined by consideration of what is determined as the "zone of trust" for the other user. For example, if one of the users has object A within their zone of trust, the zone of trust of the other user can be expanded to include object A (if it does not already include that object A). The decision of whether to adjust one zone of reliability for another user can be defined based on configured importance rules, which may include the roles used in the application (e.g., two players of the same game, users of connected roles, etc.), or based on the proximity and / or orientation between their locations to each other. For example, if two users are located within a threshold distance in the same room of a building and their orientation is towards each other, the determination of the zone of reliability can be performed taking into account the location and orientation of the other user and / or also taking into account constraints from rendering at the client device (e.g., memory consumption and FPS).

[0073] The prioritization rules can define a weighting factor to generate a rank for each of the objects based on considering multiple factors including the user's location and the user's distance from the object, the size of the model and its relative size to the object, the user's viewpoint (including the user's location and orientation), a "zone of confidence" definition, and a model load constraint initial configuration for scaling rendering by the client application 460. The prioritization service 440 can process an input list of unordered identifiers of objects and generate as output an array of ordered identifiers of objects that can be retrieved and loaded into the cache 480 from the data source integration service 425. In some implementations, the data source integration service 425 can be implemented as part of the services provided by the platform 405. In some implementations, the data source integration service 425 is supported from the platform and the data creation and tree data structure generation as implemented in the data creation and spatial indexing services 430 can be implemented as an external service or as part of the services implemented by the client application 460.

[0074] In some implementations, the prioritization service 440 can provide an ordered list of outputs to a dynamic loading service 445, which can implement auto-tuning rules 447 to dynamically determine the amount of objects to be loaded and rendered based on the frame rate per second as monitored at the client application 460 and / or based on the memory consumption of the client application 460 while processing and loading the objects for rendering. The auto-tuning rules 447 can be processed iteratively, for example, to update the current model load constraints based on the observed current frame generation rate for rendering as a set of objects are rendered.

[0075] In some embodiments, the dynamic loading service 445 can maintain pending operations on the client side from the most recent update of the model loading constraints, along with the state that defines which objects are loaded and unloaded. The dynamic loading service 445 can implement filtering rules to determine which subset of objects from the ordered list of outputs will be downloaded to the cache and rendered on the display device. The dynamic loading service 445 calculates the current state of loading and unloading during the monitored period and in relation to a given session for a user interacting with the model, and utilizes information from the spatial indexing service 430 and the filtering service 435 for the loaded model to select those objects that comply with the auto-adjustment constraints that define the current model loading constraints (automatically adjusted based on the current frame generation rate for rendering). Based on the determined identifiers of the objects by the dynamic loading service, the objects can be invoked through the data source integration service 425 and downloaded in the cache 480.

[0076] In some embodiments, when objects and object characteristics are downloaded, the client application engine 475 can render the determined subset of objects as being downloaded to the cache and can render a scene including the objects on the display device.

[0077] 5A and 5B show examples of user interactions with a user interface including presentations 500 and 550 of 3D data for a 3D model provided by a client application on a user device. The presentation 500 of FIG. 5A can provide tools for navigating within a 3D model of an environment in which objects are rendered according to an implemented adaptive dynamic loading strategy, according to an embodiment of the present disclosure. The user interface 500 can include a view of rendered objects from a model that is the same model as the one associated with the rendering displayed on FIGS. 3A and 3B. In some embodiments, a user can navigate based on interactions with menu options provided in the user interface 500 for moving within the model and for rendering objects associated with the user's new position to be provided, and the selection of objects to be loaded and rendered and subsequently unloaded based on updates to the current model loading constraints can be performed substantially similarly to the described method 200 of FIG. 2, including iteratively repeating ranking, selecting, downloading, rendering, updating, and removing objects from the 3D model as the user's viewpoint in the environment changes.

[0078] When the 3D model is a 3D model of a building, such as the presented view of one of the buildings in Figures 3A and 3B, the shell of the model is a "dollhouse" representation of the building as viewed from the outside. Generally, even if the dollhouse representation of the 3D model has fine detail, or a lower level of detail (or anything in between), by moving within the environment, desired portions of the 3D data (not fully rendered in the dollhouse representation) can be easily identified based on the identified user position and can be substantially preserved to a certain extent for rendering on each device used for rendering.

[0079] A user interacting with a 3D model and navigating within a 3D environment (as shown on Figures 3A and 3B) can switch from a viewpoint that is from outside the building to a viewpoint that is within a room of the building (as shown on Figures 5A and 5B). The presentation 500 of the 3D model in Figure 5A is from inside a room that includes a first wall 505 with a window on the right side and a second wall 510 with a window. Even when the window is a glass window and is viewed through it, the presentation 500 does not include all objects that are outside the building and can be seen. The loading strategy used to render the presentation 500 takes into account the user's position and the distance between the user and other objects in the room to determine which of the objects within the user's zone of trust will be presented according to observing the current load model constraints. The loading strategy does not select all objects that are outside the window, for example, including some objects on wall 500 but not objects on wall 510.

[0080] In contrast, presentation 550 in Figure 5B is from a user's viewpoint that is closer to the window on wall 505 in Figure 5A. When that presentation 550 was being rendered, a loading strategy considered the distance of the external objects from the user's viewpoint and was used to select those that would be loaded and rendered on the device. As shown, the window 555 is seen through the window and based on the loading strategy, objects that can be seen from the window are rendered, and thus more objects are visible from within the building model compared to what was seen on Figure 5A as some were not selected for loading and rendering when the user had the viewpoint as in Figure 5A.

[0081] As shown in Fig. 5A, when the user's viewpoint is inside a room in a building, the user cannot be presented with objects outside the building even when they are not occluded (e.g., it should be possible to see other buildings outside through a window in the wall of the room), because the zone of confidence as determined and filtered forces the download of all nearby objects in the room even when they should be visible from that viewpoint, thus leaving no memory for objects outside the room to be downloaded and rendered. In Fig. 5B, the objects outside the window are rendered because the loading strategy has evaluated a new set of objects to be included in the new zone of confidence, and the objects have been ranked based on their distance to the user viewpoint and filtered to fit the memory capacity of the user device. In this example, the objects shown as visible through the window 555 are those in the 3D model that have been filtered to be included for download and rendering as ranked according to the loading strategy.

[0082] By determining which objects are loaded and rendered according to a dynamically adjustable loading strategy according to the present embodiment, flexible and efficient switching between different display modes and different levels of data to be rendered can be supported. As shown, the loading strategy can determine some objects to be rendered when the user is viewing from outside the model (as shown in FIGS. 3A and 3B), and the loading strategy can determine which objects are rendered and which objects are removed from the device when switching viewpoints and moving to the interior of the mode (as shown in FIGS. 5A and 5B). Such an implementation can improve the timeliness of the rendering of the 3D data and can also improve the performance of the client application and its responsiveness to user interactions received through the device used by the user to interact with the data and navigate within the view of the 3D model.

[0083] The user interface 500 may be associated with a particular frame generation rate (e.g., 71, 77, or 70, among other examples), and based on the current frame generation rate, updates to the model load constraints may be determined as an auto-adjustment factor (e.g., an auto-adjustment factor of 29% to reduce the model load constraint). By dynamically adjusting the model load constraints, objects that need to be loaded and unloaded may be efficiently done to support scalability of rendering on the device and viewpoint changes that allow for consistent performance during user movement within the environment. The prioritized objects that are considered for rendering are those that can fit the model load constraints (constraint thresholds for polygons to be loaded) so that the system can allow them to be loaded and rendered.

[0084] FIG. 6 is a schematic diagram of a data processing system including a data processing device 600, which can be programmed as a client or as a server. The data processing device 600 is connected to one or more computers 690 through a network 680. While only one computer is shown in FIG. 6 as the data processing device 600, multiple computers can be used. The data processing device 600 includes various software modules that can be distributed among application layers and an operating system. They can include executable and / or interpretable software programs or libraries, including tools and services of the client application 604, including a user interface that allows a user of the client application 604 to display a 3D model of an environment in one or more review modes. Furthermore, the client application 604 can implement an adaptive dynamic loading strategy to determine and load parts of objects of the 3D model that facilitate efficient rendering that is sustainable for the client device without downtime due to loading overload.

[0085] The number of software modules used can vary from implementation to implementation. Moreover, the software modules can be distributed over one or more data processing devices connected by one or more computer networks or other suitable communications networks.

[0086] The data processing apparatus 600 also includes hardware or firmware devices including one or more processors 612, one or more additional devices 614, a computer-readable medium 616, a communication interface 618, and one or more user interface devices 620. Each processor 612 is capable of processing instructions for execution within the data processing apparatus 600. In some implementations, the processors 612 are single-threaded or multi-threaded processors. Each processor 612 is capable of processing instructions stored in a computer-readable medium 616 or a storage device, such as one of the additional devices 614. The data processing apparatus 600 uses the communication interface 618 to communicate with one or more computers 690, for example, over a network 680. Examples of user interface devices 620 include a display, a camera, a speaker, a microphone, a haptic feedback device, a keyboard, a mouse, and VR and / or AR devices. The data processing apparatus 600 may, for example, store instructions for implementing operations associated with the program(s) described above on a computer-readable medium 616 or one or more additional devices 614, such as one or more of a hard disk device, an optical disk device, a tape device, and a solid-state memory device.

[0087] The subject embodiments and functional operations described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including structures described herein and their structural equivalents, or one or more combinations thereof. The subject embodiments described herein can be implemented using one or more modules of computer program instructions encoded on a non-transitory computer-readable medium for execution by or for controlling the operation of a data processing apparatus. The computer-readable medium can be a manufactured product, such as a hard drive in a computer system, or an optical disk sold through a retail channel, or an embedded system. The computer-readable medium can be obtained separately and later encoded with one or more modules of computer program instructions, for example, after distribution of one or more modules of computer program instructions over a wired or wireless network. The computer-readable medium can be a machine-readable storage device, a machine-readable storage device substrate, a memory device, or one or more combinations thereof.

[0088] The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may include code that creates an execution environment for a subject computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, a runtime environment, or one or more combinations thereof. Additionally, an apparatus may employ a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0089] A computer program (also known as a program, software, software application, script, or code) can be written in any suitable form of programming language, including compiled or interpreted, declarative or procedural, and it can be deployed in any suitable form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a file dedicated to the program in question, or in multiple coordinated files (e.g., a file that stores one or more modules, subprograms, or portions of code), a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document). A computer program can be deployed to run on one computer or multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0090] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0091] Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or can be operatively coupled to receive data from or transfer data to, or both. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Suitable devices for storing computer program instructions and data include, by way of example, all types of non-volatile memory, media, and memory devices, including semiconductor memory devices, such as EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM disks and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0092] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a liquid crystal display (LCD) device, an organic light emitting diode (OLED) display device, or another monitor, for displaying information to a user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can be used to provide interaction with a user as well, e.g., feedback provided to the user can be any suitable form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any suitable form, including acoustic, speech, or tactile input.

[0093] A computing system may include clients and servers. Clients and servers are generally remote from one another and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to one another. An embodiment of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical or browser user interface through which a user may interact with an embodiment of the subject matter described herein, or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any suitable form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0094] While this specification includes many implementation details, they should not be construed as limitations on the scope of what is or may be claimed, but as descriptions of features specific to particular embodiments of the disclosed subject matter. Certain features described in the specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0095] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0096] Thus, specific embodiments of the invention have been described. Other embodiments are within the scope of the following claims. In addition, the actions recited in the claims can be performed in a different order and still achieve desirable results. EXAMPLES

[0097] While this application is defined in the appended claims, the invention may also (additionally or alternatively) be defined by the following examples.

[0098] Example 1. 1. A computer-implemented method comprising: acquiring, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding location information for objects within a three-dimensional model of an environment, the three-dimensional model being stored on a remote computer system; ranking, by the computer, the set of objects in the three-dimensional model based at least on a distance between each object in the set of objects and a defined viewpoint for a user within the environment as determined using the three-dimensional spatial access tree data structure to form an object hierarchy; selecting, by the computer, an appropriate subset of the set of objects to be rendered based on the object hierarchy and current model load constraints; downloading, by the computer and from the remote computer system, the appropriate subset of the set of objects into the local memory; rendering, by the computer, the appropriate subset of the set of objects from the local memory to the display device based on a viewpoint within the environment defined for the user; The method comprising:

[0099] Example 2. The three-dimensional spatial access tree data structure includes a bounding volume hierarchy, and the obtaining step includes: calculating the bounding volume hierarchy including nodes corresponding to bounding volumes defined for each spatial portion of the environment, leaf nodes of the bounding volume hierarchy corresponding to the objects in the three-dimensional model of the environment; The method described in Example 1.

[0100] Example 3. updating, by the computer, the current model load constraint based on a current frame generation rate for the rendering; removing, by the computer, downloaded objects from the local memory based on the object hierarchy and in response to reaching the current model load constraint within a threshold distance as a result of the downloading; The method of any one of the preceding embodiments, further comprising:

[0101] Example 4. The method of any one of the preceding embodiments, further comprising repeating the ranking, the selecting, the downloading, the rendering, the updating, and the removing when changes to a viewpoint within the defined environment for the user are received by the computer.

[0102] Example 5. Obtaining the three-dimensional spatial access tree data structure includes: receiving the three-dimensional spatial access tree data structure for the environment in a communication stream from the remote computer system; The method of any one of the preceding embodiments.

[0103] Example 6. Ranking the set of objects in the three-dimensional model comprises: identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to be located within a zone of confidence around the defined viewpoint of the user based on meeting prioritization criteria for object loading; the set of objects is selected based on identifying the set of objects as the highest ranked objects from the objects ranked to be loaded according to the current model load constraints. The method of any one of the preceding embodiments.

[0104] Example 7. 7. The method of claim 6, wherein the prioritization criteria for object loading defines factors associated with prioritizing the loading of a first object compared to the loading of another object, the factors including a comparable ratio between the size of the object and the size of a model, the defined viewpoint of the user, the distance between each object of the set of objects and the defined viewpoint for the user, the location of the user within the three-dimensional model, a comparable position between each two of the objects in the environment from the viewpoint of the user, and physical characteristics of the objects.

[0105] Example 8. 8. The method of example 7, wherein the prioritization criteria is defined as a weighted combination of two or more of the defined factors, each factor of the two or more factors being associated with a predefined weight for the combination.

[0106] Example 9. determining a change in the viewpoint of the user within the environment; repeating the ranking for a second set of objects in the environment, the ranking being repeated for objects that at least partially overlap the set of ranked objects that is the first set of ranked objects; the ranking is performed to generate a new object hierarchy based on a distance between objects in the second set of objects and a new viewpoint of the user determined based on the change in the location. The method of any one of the preceding embodiments.

[0107] Example 10. 10. The method of example 9, wherein at least one object of the second set of objects is downloaded to the local memory as part of the appropriate subset of the set of objects.

[0108] Example 11. The three-dimensional spatial access tree data structure encodes metadata about the objects including object categories and environmental characteristics that define locations of the objects within the environment, and the ranking of the set of objects is determined by: identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to meet a prioritization criterion based on metadata of the objects. The method of any one of the preceding embodiments.

[0109] Various similar operations and processes described in Examples 1-11 can be performed in a system including at least one processor and a memory communicatively coupled to the at least one processor, the memory storing instructions that, when executed, cause the at least one processor to perform the operations. Further, a non-transitory computer-readable medium storing instructions that, when instructed, cause the at least one processor to perform the operations as described in any one of Examples 1-11 can also be implemented.

Claims

1. 1. A computer-implemented method comprising: acquiring, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding location information for objects within a three-dimensional model of an environment, the three-dimensional model being stored on a remote computer system; ranking, by the computer, the set of objects in the three-dimensional model based at least on a distance between each object in the set of objects determined using the three-dimensional spatial access tree data structure and a defined viewpoint for a user within the environment to form an object hierarchy; selecting, by the computer, an appropriate subset of the set of objects to be rendered based on the object hierarchy and current model load constraints; downloading, by the computer, the appropriate subset of the set of objects from the remote computer system to the local memory; rendering, by the computer, the appropriate subset of the set of objects from the local memory to the display device based on a viewpoint within the environment defined for the user; The method comprising:

2. The three-dimensional spatial access tree data structure includes a bounding volume hierarchy, and the obtaining step includes: calculating the bounding volume hierarchy including nodes corresponding to bounding volumes defined for each spatial portion of the environment, leaf nodes of the bounding volume hierarchy corresponding to the objects in the three-dimensional model of the environment; The method of claim 1.

3. updating, by the computer, the current model load constraint based on a current frame generation rate for the rendering; removing, by the computer, downloaded objects from the local memory based on the object hierarchy and in response to reaching the current model load constraint within a threshold distance as a result of the downloading; The method of claim 1 further comprising:

4. 4. The method of claim 3, further comprising repeating the ranking, the selecting, the downloading, the rendering, the updating, and the removing when changes to a viewpoint within the environment defined for the user are received by the computer.

5. Obtaining the three-dimensional spatial access tree data structure includes: receiving the three-dimensional spatial access tree data structure for the environment in a communication stream from the remote computer system; The method of claim 1.

6. Ranking the set of objects in the three-dimensional model comprises: identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to be located within a zone of confidence around the defined viewpoint of the user based on meeting prioritization criteria for object loading; the set of objects is selected based on identifying the set of objects as highest ranked objects from the objects ranked to be loaded according to the current model load constraints. The method of claim 1.

7. 7. The method of claim 6, wherein the prioritization criteria for object loading defines factors associated with prioritizing the loading of a first object compared to the loading of another object, the factors including a comparable ratio between a size of the object and a size of a model, the defined viewpoint of the user, the distance between each object of the set of objects and the defined viewpoint for the user, the location of the user within the three-dimensional model, a comparable position between each two of the objects in the environment from the viewpoint of the user, and physical characteristics of the object.

8. 8. The method of claim 7, wherein the prioritization criteria is defined as a weighted combination of two or more of the defined factors, each factor of the two or more factors being associated with a predefined weight for the combination.

9. determining a change in the viewpoint of the user within the environment; repeating the ranking for a second set of objects in the environment, the ranking being repeated for objects that at least partially overlap the set of ranked objects that is the first set of ranked objects; the ranking is performed to generate a new object hierarchy based on a distance between objects in the second set of objects and a new viewpoint of the user determined based on the change in the location. The method of claim 1.

10. 10. The method of claim 9, wherein at least one object of the second set of objects is downloaded to the local memory as part of the appropriate subset of the set of objects.

11. The three-dimensional spatial access tree data structure encodes metadata about the objects including object categories and environmental characteristics that define locations of the objects within the environment, and the ranking of the set of objects is determined by: identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to meet a prioritization criterion based on metadata of the objects. The method of claim 1.

12. 1. A system comprising: a non-transitory storage medium having computer-aided design program instructions stored thereon; and one or more data processing devices configured to run the instructions of the computer-aided design program to perform operations, the operations including: acquiring, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding location information for objects within a three-dimensional model of an environment, the three-dimensional model being stored on a remote computer system; ranking, by the computer, the set of objects in the three-dimensional model based at least on a distance between each object in the set of objects and a defined viewpoint for a user within the environment as determined using the three-dimensional spatial access tree data structure to form an object hierarchy; selecting, by the computer, an appropriate subset of the set of objects to be rendered based on the object hierarchy and current model load constraints; downloading, by the computer and from the remote computer system, the appropriate subset of the set of objects into the local memory; rendering, by the computer, the appropriate subset of the set of objects from the local memory to the display device based on a viewpoint within the environment defined for the user; The system comprising:

13. The non-transitory storage medium further comprises instructions that, when executed by the one or more data processing devices, perform operations, the operations including: updating, by the computer, the current model load constraint based on a current frame generation rate for the rendering; removing, by the computer, downloaded objects from the local memory based on the object hierarchy and in response to reaching the current model load constraint within a threshold distance as a result of the downloading; The system of claim 12 further comprising:

14. The non-transitory storage medium further comprises instructions that, when executed by the one or more data processing devices, perform operations, the operations including: repeating the ranking, the selecting, the downloading, the rendering, the updating, and the removing when changes to a viewpoint within the environment defined for the user are received by the computer. The system of claim 13.

15. The non-transitory storage medium further comprises instructions that, when executed by the one or more data processing devices, perform operations, the operations including: determining a change in the viewpoint of the user within the environment; repeating the ranking for a second set of objects in the environment, the ranking being repeated for objects that at least partially overlap the set of ranked objects that is the first set of ranked objects; the ranking is performed to generate a new object hierarchy based on a distance between objects in the second set of objects and a new viewpoint of the user determined based on the change in the location. The system of claim 12.

16. A non-transitory computer readable medium encoding instructions operable to cause a data processing apparatus to perform an operation, the operation comprising: acquiring, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding location information for objects within a three-dimensional model of an environment, the three-dimensional model being stored on a remote computer system; ranking, by the computer, the set of objects in the three-dimensional model based at least on a distance between each object in the set of objects and a defined viewpoint for a user within the environment as determined using the three-dimensional spatial access tree data structure to form an object hierarchy; selecting, by the computer, an appropriate subset of the set of objects to be rendered based on the object hierarchy and current model load constraints; downloading, by the computer and from the remote computer system, the appropriate subset of the set of objects into the local memory; rendering, by the computer, the appropriate subset of the set of objects from the local memory to the display device based on a viewpoint within the environment defined for the user; The non-transitory computer-readable medium.

17. Ranking the set of objects in the three-dimensional model comprises: identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to be located within a zone of confidence around the defined viewpoint of the user based on meeting prioritization criteria for object loading; the set of objects is selected based on identifying the set of objects as highest ranked objects from the objects ranked to be loaded according to the current model load constraints.

20. The non-transitory computer-readable medium of claim 16.

18. 20. The non-transitory computer-readable medium of claim 17, wherein the prioritization criteria for object loading defines factors associated with prioritizing the loading of a first object compared to the loading of another object, the factors including a comparable ratio between a size of an object and a size of a model, the defined viewpoint of the user, the distance between each object of the set of objects and the defined viewpoint for the user, a location of the user within the three-dimensional model, a comparable position between each two of the objects in the environment from the viewpoint of the user, and physical characteristics of the object.

19. 20. The non-transitory computer-readable medium of claim 18, wherein the prioritization criteria is defined as a weighted combination of two or more of the defined factors, each factor of the two or more factors being associated with a predefined weight for the combination.

20. The three-dimensional spatial access tree data structure encodes metadata about the objects including object categories and environmental characteristics that define locations of the objects within the environment, and the ranking of the set of objects is determined by: identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to meet a prioritization criterion based on metadata of the objects.

20. The non-transitory computer-readable medium of claim 18.

Images