Adaptive dynamic loading strategy for rendering large three-dimensional models

Abstract

Methods, systems, and apparatus, including medium-encoded computer program products, for loading and rendering comprising: obtaining a 3D spatial access tree data structure encoding location information about objects in a 3D model of an environment, where the 3D model is stored on a remote computer system; ranking a set of objects in the 3D model to form an object hierarchy based at least on a distance between each object in the set of objects as determined using the three-dimensional space access tree data structure and a specified perspective of a user within the environment; selecting an appropriate subset of the set of objects to render based on the object hierarchy and a current model load limit; downloading the appropriate subset to a local memory; and rendering the appropriate subset from the local memory to the display device based on the specified perspective of the user within the environment.

Description

Technical Field

[0001] The present specification relates to visualization of three-dimensional models on a client device, and more particularly to an adaptive dynamic loading strategy for rendering large three-dimensional models. Background Art

[0002] The present specification relates to visualization of three-dimensional models on a client device. In addition, the present specification relates to model data used in computer graphics applications such as computer-generated animation and / or computer visualization systems and techniques. Summary of the invention

[0003] The present specification relates to adaptive dynamic loading techniques for determining objects from a 3D model for loading and rendering that take into account the position of a user when viewing the 3D model and the model load limit of a frame of the 3D model rendered by a client device.

[0004] Generally speaking, one or more aspects of the subject matter described in this specification may be embodied in one or more methods (and may also be embodied in one or more non-transitory computer-readable media, the one or more non-transitory computer-readable media tangibly encoding a computer program, the computer program being operable to cause a data processing device to perform operations), the method comprising: obtaining, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding position information about objects in a three-dimensional model of an environment, wherein the three-dimensional model is stored on a remote computer system; ranking, by the computer, a set of objects in the three-dimensional model to form an object hierarchy based at least on a distance between each object in the set of objects and a specified viewing angle of a user within the environment as determined using the three-dimensional spatial access tree data structure; selecting, by the computer, an appropriate subset of the set of objects to be rendered based on the object hierarchy and a current model load limit; downloading, by the computer and from the remote computer system, the appropriate subset of the set of objects to a local memory; and rendering, by the computer, the appropriate subset of the set of objects from the local memory to the display device based on the specified viewing angle of the user within the environment.

[0005] By utilizing the described techniques, the following advantages are achieved. Based on the dynamic adaptive loading techniques implemented for client applications as described in the present disclosure, the rendering of 3D data on a user device can be facilitated and executed more efficiently, with lower requirements on the computing resources of the user device and without negatively affecting the user experience. The loading technique does not require the 3D model to be loaded as a whole, and therefore the present solution can work at any scale of the model. The loading technique is automatically tuned based on the execution of the rendering device. Based on the performance of the device, more objects can be rendered in some cases than in other cases. By determining which objects to load and render according to a dynamic loading strategy, it is possible to support flexible and efficient switching between different display modes and different levels of data to be rendered during the user's movement in the 3D space. It is possible to iteratively determine whether to download objects from the 3D model and which objects to download from the 3D model, rather than downloading the entire model at the beginning of rendering, which downloads the entire model at the beginning of rendering, which significantly reduces performance and slows down the client application. In addition, the preemptive loading and unloading of objects is based on determining a confidence zone around the user's perspective in the 3D environment of the model and taking into account the performance of the user's device. Such adaptive loading techniques can improve the performance of client applications and support smooth rendering during user navigation within the space. In this way, rendering is performed faster and the client application is more responsive to user interactions for navigating within the space. Since the loading policy identifies objects to be downloaded on the rendering device first based on the location of the objects relative to the user's perspective and the current model load limit of the user's device, rendering a different set of objects can be performed faster as the user changes their perspective. The loading policy allows for maintaining a ranked list of objects for determining which objects to download while observing the number of frames generated per second and memory consumption on the rendering device.

[0006] The described techniques can support stable rendering performance at a user device, which improves the user experience and takes into account the memory capacity and rendering load of the device. The frame generation rate can be varied, and the loading and rendering of objects can be adjusted to match the expected frame generation rate and improve the performance of the display device without negatively affecting the user experience. In addition, the loading strategy can support the loading of additional objects that are not immediately rendered in a specific scenario, and the unloading of objects based on the monitored performance of the user device.

[0007] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the invention will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example of a system that may be used to support visualization of large models according to implementations of the present disclosure is shown.

[0009] Figure 2 An example of a process for navigating within a 3D model of an environment that is rendered according to a dynamic object loading strategy that efficiently utilizes resources at a client device to render the model is shown in accordance with implementations of the present disclosure.

[0010] Figure 3A and Figure 3B An example of user interaction for navigating within a spatial view of a virtual reality (VR) environment rendered by a client device at a display device is shown in accordance with implementations of the present disclosure.

[0011] Figure 4 An example of a system architecture of a client device for rendering a 3D model based on an adaptive dynamic loading strategy according to an implementation of the present disclosure is shown.

[0012] Figure 5A and Figure 5B An example of user interaction with a user interface provided by a client application on a user device to navigate within a 3D model of an environment in which objects are rendered according to an implemented adaptive dynamic loading strategy is shown in accordance with implementations of the present disclosure.

[0013] Figure 6 is a schematic diagram of a data processing system including a data processing device that can be programmed as a client or a server.

[0014] Like reference numbers and designations throughout the various drawings indicate like elements. DETAILED DESCRIPTION

[0015] The present disclosure describes various tools and techniques for adaptively dynamically loading objects of a three-dimensional (3D) model to be rendered on a client device. The described techniques support stable performance for rendering (e.g., client devices will not crash due to overload caused by processing large data portions). According to implementations of the present disclosure, objects can be progressively rendered as a user navigates within a 3D model of an environment and zooms in or out of the model. In some implementations, rendering can be performed in a virtual reality environment or an augmented reality environment, as well as a combination of the two, where different users can render and interact with the model through different devices.

[0016] In some implementations, the 3D model may be rendered from different devices or platforms, including desktop devices, tablets, virtual reality (VR) and / or augmented reality (AR) devices, etc. Computer graphics applications include different software products and / or services that support generating representations of 3D objects, which can be used to visualize object models in the context of video games or other contexts for animation and video rendering. Computer graphics applications also include computer animation programs that show views or scenes during user review and navigation within or outside the model.

[0017] In some cases, 3D computer visualizations may be created in a variety of scenarios and in the context of different technologies. For example, a 3D model of an object (such as a manufacturing plant, a building, a physical structure) may be animated for display on a user interface of a native device application, a web application, a VR application, an AR application, and other examples. A user device may display a user interface in which an object of the 3D model is rendered.

[0018] In some implementations, a user may participate in a VR session in which a request is made to display a 3D model in the context of navigating within the 3D model on a user device (such as a VR device or a laptop). The model to be rendered may be a large model that requires a large amount of computing resources (such as the memory capacity of the rendering device) to load and render in its entirety (or to a greater extent), which may present some challenges for execution on a small personal user device that does not have the resources required for loading a large model. For example, the model may be very large, in which case even a high-end personal computer with multiple graphics processing units may not be able to load the model without time delays. The user device available for loading the model may have limitations in terms of its load limit. Therefore, rendering objects of large models may be associated with challenges and risks of providing a poor user experience or performance (e.g., crashing of the client application used for rendering).

[0019] According to an implementation of the present disclosure, a client application may be configured to enable loading strategies for objects of a model to be rendered as a user moves within an environment (e.g., a virtual reality environment), thereby resulting in a smooth and flexible workflow supported by efficient tools that can flexibly adjust the loading strategy and tune it to be suitable for real-time loading of objects' memory consumption and monitoring of the current frame generation rate of the rendering.

[0020] People can use different user interface applications to navigate in real time in 3D space and change the perspective of viewing parts of the 3D model. For example, the perspective may initially be outside the model, which may be a model of a building, where the perspective may be changed based on the user's interaction and repositioning relative to the model, and the building may be viewed from a closer distance or from the inside (e.g., within a specific room or floor of the building). At each step of changing the user's perspective, different objects of the 3D model may be associated with rendering. According to the present implementation, objects may be dynamically loaded as the user moves within the environment, and the number of objects rendered as part of the model may be dynamically adjusted based on a metric of monitoring of the rendering at the device (e.g., monitoring of the number of frames per second (FPS) generated and memory consumption). This metric may be taken into account when determining the number of objects to be visualized, wherein the number may be iteratively automatically tuned as the user moves within the model and changes its perspective. Based on monitoring the measured frame rate and memory consumption of the currently rendered scene, the model load limit may be dynamically adjusted to scale the load on the processor to iteratively handle the downloading and rendering of objects as the user moves.

[0021] In some implementations, the platform may be accessed through a VR device (such as a VR headset, a head mounted display, a smartphone display, or other device), and the VR display may present a VR environment that may be rendered to one or more users wearing the VR headset to provide them with an immersive experience so that they can navigate in review mode within a 3D model rendered in the VR environment. In some implementations, the VR environment may be accessed by a user via VR, AR, and optionally non-VR and non-AR devices to provide display of the 3D model in a virtual world. The rendering capabilities used for display of the model may vary from device to device, and they may be adjusted to suit the current rendering performance at a particular device.

[0022] Figure 1An example of a system 100 according to an implementation of the present disclosure is shown, which can be used to support visualization of a model in an adaptive and computationally efficient manner by generating a dynamic loading strategy, without any size restrictions on the model. The computer 110 includes a processor 112 and a memory 114, and the computer 110 can be connected to a network 140, which can be a private network, a public network, a virtual private network, etc. The processor 112 can be one or more hardware processors, and the one or more hardware processors can each include multiple processor cores. The memory 114 can include both volatile memory and non-volatile memory, such as random access memory (RAM) and flash RAM. The computer 110 may include various types of computer storage media and devices, which may include memory 114 to store instructions for programs running on the processor 112, the programs including VR programs, AR programs, video games, or design interactive applications that support view exploration and collaboration, etc.

[0023] The computer 110 includes a client application 116, which includes logic for implementing a method of helping a user participate in rendering a 3D model of an environment and navigating within the 3D model. For example, the client application 116 may be groupware designed to support a user 190 in a collaborative process with another user using another client application, wherein the two users navigate within the 3D model. For example, the client application 116 may be used by the user 190 to intentionally navigate into the 3D model to show various aspects of the model that may be rendered at another client application of another user (e.g., from another perspective in the 3D model specified for the other user). For example, this may be part of a collaborative design process between a design user navigating within a construction project using the client application 116 and stakeholders of the construction project, with views of the model presented to the stakeholders from the outside and inside at different levels of detail. In some cases, the client application 116 may provide software functionality to facilitate a collaborative process between connected devices to obtain a shared view of the 3D model of the environment. According to the present implementation, the loading of objects from the model may be automatically scaled based on monitoring the state of the client application 116 and the current frame generation rate when rendering the objects of the model.

[0024] The client application 116 may be run locally on the computer 110, remotely on a computer of one or more remote computer systems 150 (e.g., one or more server systems of one or more third-party providers accessible by the computer 110 via the network 140), or both locally and remotely. In some implementations, the client application 116 may be an access point for accessing a service running on a platform 145 (e.g., a cloud platform) that supports rendering object model data stored in a database 160 on the platform 145. In some implementations, the client application 116 may implement rendering techniques to adjust the loading and unloading of objects during changes in the user's perspective within the currently rendered model. In some implementations, when the 3D model to be rendered is large and takes a long time (e.g., one or two minutes) to load, the computer 110 may impose a limit on the amount of data that can be processed and rendered. In some cases, if the loading operation exceeds the computer's ability to process and render a large number of objects, the rendering of the large amount of data may be delayed or even the memory of the computer 110 may crash. Such problems may be avoided by implementing rendering techniques that adjust the loading and unloading of objects according to implementations of the present disclosure. By using such rendering techniques, objects from a particular scene within the user's field of view within the model may be prioritized and filtered to determine a subset of objects to download, load, and / or render. Thus, such techniques may support stable rendering performance at the user's device, which improves the user experience and takes into account the device's memory capacity.

[0025] In some implementations, filtering may also be based on user-defined rules associated with the type of objects to be prioritized, visibility culling, sorting criteria, and / or post-filtering operations that may be associated with high computational cost filtering (such as occlusion culling). In some implementations, user selections available for filtering may include user selections of specific object categories (such as walls, windows, etc.). The user selection may define whether to exclude or include selected objects. In some implementations, visibility culling may be used to filter objects to be downloaded, loaded, and / or rendered. Visibility culling may combine multiple volumes defined by spatial ranges including multiple users' fields of view, corresponding confidence zones, or other forms to determine objects that meet the filtering. In some implementations, sorting the identified objects based on one or more criteria may include sorting based on criteria including distance, size, metadata, confidence zones, material properties (e.g., material transparency), and / or other characteristics of or for the objects. In some implementations, filtering of objects may be performed iteratively using one technique after another, wherein a more computationally expensive technique may be applied at a later stage, when a smaller subset of objects will be considered, thereby minimizing computational cost.

[0026] In some implementations, the client application 116 may request to render a model where the model is large enough to not fit into memory and the object cannot be rendered at a frame rate that matches the user's rendering criteria (e.g., a frame rate that is comfortable for the end user and does not cause motion sickness). The FPS is affected by the number of calculations performed on the computer 110 and is independent of the refresh rate. In some cases, the FPS rate may be maintained within a predefined range and is limited by the number of polygons that can be rendered. For example, it may be determined to maintain the FPS rate between 40 and 75. In some cases, it may be determined to maintain the FPS rate within a determined range that may be associated with a low likelihood of causing motion sickness in the user and comfortable for the user to be immersed in the rendered model environment. The frame rate is a frame generation rate that measures the speed at which a certain number of frames occur within a period of time (e.g., one second). The frame rate may be calculated based on the following formula: 1 / (the amount of time it takes to render visible model objects and execute the logic of the client application 116 relative to the most recent frame provided to the display device 120 for display). The frame generation rate is different from the screen refresh rate of the display device 120 because the screen refresh rate is fixed for the device and the frame generation rate can vary, and the loading and rendering of objects can be adjusted to match the frame generation rate and improve rendering performance by efficiently utilizing device resources so that the frame generation rate matches the frame rate of the device. The applied techniques improve the performance of the client application 116 without negatively affecting the user experience.

[0027] In some implementations, when the client application 116 is used to render objects from the model from the 3D model on the database 160 of the platform 145, the client application 116 may obtain a 3D spatial access tree data structure 130 (such as an R-tree data structure), which may be used and processed to inform a loading strategy to select which objects of the model to render when navigating views and scenes of the 3D model. The loading strategy may be defined to efficiently use the computing resources of the computer 110 to load and unload objects according to an object hierarchy that ranks objects according to the priority of rendering of the object and according to the monitored performance of the current rendering (e.g., memory consumption, and a measure of the current frame rate). In some implementations, the calculation of the priority may be performed at the computer 110, for example, based on different factors that are considered for prioritization (e.g., the factors may be optionally weighted). In some implementations, the calculation of the priority for ranking the objects may be offloaded and performed at a remote system or external service (e.g., a cloud service that implements the logic for the priority calculation).

[0028] In some implementations, the client application 116 obtains a model description (e.g., a model description of a construction design, a floor plan, etc.) from the database 160. A user 190 may interact with the client application 116 to launch or join an already launched virtual world (a computer-generated representation of the physical world, such as the interior of a house, a representation of a collaboration space, etc.) to navigate within the space (e.g., a virtual environment) and view the model from different aspects. The rendering may scale to any model size based on a dynamic loading strategy implemented as described in the present application. For example, larger or smaller objects may be downloaded, loaded, and rendered depending on whether the user is zooming in or out.

[0029] In some implementations, database 160 may store multi-dimensional data, including 3D models, image data, floor plans, etc. In some implementations, database 160 may include data of users associated with access rights for reading, writing, modifying, copying, or other operations to be performed when handling data on database 160. For example, user 190 is defined as a user with access rights to model 134 (and to a model description in the form of a 3D spatial access tree data structure) according to the correspondingly defined permissions of user 190. Access rights and authorizations may be evaluated at an authorization service provided by platform 145.

[0030] In some implementations, when interacting with a visualization service 165 (which may be implemented as part of the platform 145, a separate standalone implementation, or part of an implementation of the client application 116), the client application 116 may obtain a model description of a model to be rendered on the computer 110. The visualization service 165 may implement logic for building a model description based on the model stored at the database 160, wherein the model description may be streamed to the client application 116 (e.g., in a communication stream) and used to initialize a loading policy. In some implementations, the visualization service 165 may provide a model description in the form of a 3D space access tree data structure 130, which may be used to generate a loading policy using loading-related factors (such as the user's viewing angle and the distance between objects within the field of view), and information about the currently generated FPS and memory usage of the rendering at the user device used to display the 3D model. The loading policy may be applied to the 3D space access tree data structure 130 to identify objects within a specific volume within the model where the user is located. In some implementations, the loading policy may include the loading-related factors as a combination to determine the priority of loading objects. The combination of factors may be a weighted combination, where different factors may be defined with different weights. For example, the model size and location of the user may be two factors defined with different weights and used to calculate a priority (or ranking) that may be used to sort the objects according to their priority and prioritize the loading of the objects and their rendering on the user's device. In some implementations, objects from the model may be downloaded and cached locally based on a determined priority for rendering, the determined priority being defined based on a prioritization ranking rule based on the tree data structure 130 and a current model load limit for model rendering by the computer 110.

[0031] In some implementations, when the user is viewing the model and moving within the virtual environment, the user may have a perspective outside the model, and then, it can be seen that the model looks small. In those cases, the priority of the objects of the model to be loaded and rendered will be for objects with relatively larger sizes to form the shell of the model, and smaller objects will be given lower priority. The objects can be ordered based on their priority ranking, and those objects that adapt to the current model load capacity can be determined and downloaded for rendering. In some implementations, when the user navigates inside the virtual environment, objects within the user's field of view (or within a threshold distance around the user's field of view) can be loaded and rendered preferentially, and even smaller objects can be loaded preferentially, while at least some of the objects that form the shell of the visualization space (e.g., a floor or room of a building) are also loaded. The ranking of objects can be dynamically changed based on the repositioning of the user and based on monitoring memory consumption and frame generation rate (e.g., within a given time period, such as between two updates of the loading strategy).

[0032] In some implementations, the client application 116 may be operated using one or more input devices 118 (e.g., a keyboard and a mouse) of the computer 110. Figure 1 110, but the display device 120 and / or the input device 118 may also be integrated with each other and / or with the computer 110, such as in a tablet computer (e.g., a touch screen may be an input device 118 / output device 120). In addition, the computer 110 may include a VR or AR system or may be part of a VR or AR system. For example, the input device 118 / output device 120 may include a VR / AR input controller, a glove or other manual manipulation tool 118a, and / or a VR / AR headset 120a. In some implementations, the input / output device may include a hand tracking device that is sensor-based and tracks movement and recreates interactions as if performed with a physical input device. In some implementations, the VR and / or AR device may be a stand-alone device that may not need to be connected to the computer 110. The VR and / or AR device may be a stand-alone device with processing capabilities and / or with an integrated computer (such as the computer 110) with, for example, input / output hardware components (such as controllers, sensors, detectors, etc.). VR and / or AR devices are either connected to the computer 110 or are stand-alone devices that integrate a computer (with a processor and memory), and can communicate with the platform 145 and immerse users connected through these devices in a virtual environment, where 3D models of objects can be presented in a simulated real physical environment (or a substantially similar environment), and users navigate within the environment and are presented with different visual objects.

[0033] In some implementations, the system 100 may be used to display data from a 3D document / model that may be used to generate a virtual world (which may be a VR environment for one or more first users, an AR environment for one or more second users, and a non-VR or AR environment for one or more third users) to be presented in a corresponding interface of the client application 116, allowing the user to use the data to navigate and interact with the environment to move to different locations in the space of the virtual world.

[0034] In some implementations, platform 145 may be a cloud platform that may be accessed to support cloud services related to visualization of data in different display modes (such as visualization service 165 that may be implemented as part of platform 145) and may support tools and techniques for interacting with data to navigate in display modes.

[0035] In some implementations, the user 190 may cause the computer 110 to render a view 133 of the model presented on the user interface 132 of the display device 120 and visually show how he / she (or another user) interacts with the view, wherein the view is generated based on processing the 3D spatial access tree data structure 130 of the model. As the user is navigating within the model and changing his position, the display device 120 may render a particular view of the virtual world. In some implementations, the view 133 may be provided to the user 190 via a VR device or other AR device as a presentation of a view of the 3D model 134, wherein as the user is navigating within the model and interacting with the model, the view of the model seen by the user on the user interface 132 may change. For example, the view 133 may change based on a user interaction with the interface 132 to bring the user closer to the model.

[0036] According to implementations of the present disclosure, rendering of the model 134 and at least some of the objects of the model 134 may be performed based on a dynamic loading strategy for loading objects of the model 134 according to a user's perspective (including the user's position and orientation), an object hierarchy defined by ranking objects according to a priority criterion for rendering objects (e.g., where ranking is performed based on a 3D space access tree structure), and a current model load limit. In some implementations, the model load limit may be maintained up to date by monitoring the current frame rate of the rendering and adjusting the current model load limit to scale the rendering to the frame rendering speed.

[0037] In some implementations, a loading strategy monitors the health of the system, where a threshold for the number of objects loaded can be adjusted based on the complexity of the data that can be rendered (e.g., some objects are more complex and associated with higher processing costs, such as transparent objects or complex material objects). For example, if the FPS at a given time is lower than a specified model load limit for multiple frames, it can be understood that the system has reached the limit of the rendering capabilities that can be supported. In some cases, the model load limit of the FPS number can be adjusted and reduced based on the distance between the current load limit and the mean of the currently monitored FPS rendered during the most recent monitored period. In some cases, the distance can be determined as the normalized distance between the FPS average during the current monitored period and the lowest comfortable FPS targeted by the system multiplied by the current amount of loaded geometry. In some implementations, a loading strategy for determining which objects to render can be implemented to increase the ability to promote positive trends and mitigate negative trends in rendering. In this case, when there is a positive progress in loading performance at the user device, loading can be accelerated by increasing the speed used for loading, and when it is determined that data will be unloaded based on the observed performance at the user device (e.g., a higher frame generation rate compared to a threshold level for considering starting unloading), loading can be slowed down to reduce the speed. This automatic tuning of the loading strategy can be performed based on the observed current FPS rate at the user device and the target FPS rate. In addition, the automatic tuning can be configured to perform unloading more slowly when it is determined that unloading is initiated, and to increase the speed of loading when it is determined that more objects can be loaded based on the current frame rate. As described, loading and unloading are performed in an adaptive manner that optimizes computing resources and balances with the frame rate per second experienced at the user device.

[0038] The systems and techniques described herein are applicable to any suitable application environment that can graphically render any portion of a virtual world (including objects therein). Thus, in some implementations, model data from models stored in database 160 can be used to render objects in different modes and accessed from a variety of types of devices, where a user can navigate in a virtual environment of the rendered model, such as with respect to Figure 3A , Figure 3B and as described in FIG5 .

[0039] Figure 2 An example of a process 200 for navigating around or within a 3D model of an environment that is rendered according to a dynamic object loading strategy that efficiently utilizes resources at a client device to render the model is shown in accordance with implementations of the present disclosure.

[0040] In some implementations, navigation within the 3D model may be in the context of user interaction in a video game. In some examples, when building a game, the objects and scenes created are used to generate a three-dimensional model that can be defined for different layout levels and based on a scripted game plan. The game can be accessed from a computer for rendering. In some implementations, the rendering can be in a virtual 3D model environment where the user is. In some implementations, the model of the game can be built to include tiles that can be dynamically loaded when the user plays the game on the computer. The 3D model of the game is rendered, and if the model is a large model, the complete rendering of the model may be a time-consuming and resource-consuming task. Model loading may not be performed within the computer's cache, for example, the computer may be a user device that does not support heavy computing and large data storage.

[0041] As the complexity and size of three-dimensional models increase, the scalability of rendered models may decrease and affect performance and user experience during rendering (e.g., during VR gaming). As the complexity of models grows and the rendering of data is expensive and hardware is limited, it may be necessary to consider optimizing the loading and unloading of objects. In some implementations, when rendering a game, it may not be necessary to initially load all objects, but rather loading and rendering may be optimized and dynamically adjusted to suit the user's position in the environment and the current model load limit, which may be automatically tuned by the computer to suit the performance of the rendering.

[0042] In some implementations, 3D models may be generated and used for rendering in various contexts. In some implementations, a user may generate, create, or upload a model of a physical entity or environment that includes various objects and spatial organization. The generated model may be provided by the platform and may be accessed through various access points (such as different user interfaces accessible via desktop devices, mobile devices, other devices, services, or other entities). For example, a 3D model may be a building, including 3D geometry, metadata, and floor plans. Multiple hierarchies may exist within the 3D geometry based on the spatial distribution of the model (e.g., the bounding volume of each part of the model, starting from the outside), and based on the metadata (e.g., building->floor->room).

[0043] At 210, a 3D space access tree data structure is obtained (received or generated) by a computer having a display device and a local memory. In some implementations, the 3D space access tree data structure is associated with Figure 1The 3D space access tree data structure 130 is substantially similar to the 3D space access tree data structure 130 of FIG. 1 . The 3D space access tree data structure is a data structure including a model description of a 3D model, wherein the model description includes position information of objects in the 3D model of the environment. The 3D space access tree data structure is obtained at a computer that is a user device for rendering the 3D model. The 3D model is stored on a remote computer system, for example, stored on a platform such as Figure 1 at platform 145).

[0044] In some implementations, generation of a 3D spatial access tree data structure may be performed at a computer or may be called from an external service (e.g., a cloud service), which may generate a tree data structure based on model data of a 3D model. In some implementations, the 3D spatial access tree data structure may be received from another computer, which may be different from a remote computer system storing the 3D model. Obtaining may include streaming a data structure from another computer and / or through a service, where the data structure may be generated from a received model description of the 3D model.

[0045] In some implementations, when a user of a computer initiates the opening of a model, a 3D spatial access tree data structure of the 3D model may be obtained, and model description data may be collected and cached at the computer. The 3D spatial access tree data structure may be used to initialize a loading strategy for objects of the model rendered at the computer. The 3D spatial access tree data structure may be constructed based on the obtained model description data of the 3D model. The 3D spatial access tree data structure may include a bounding volume hierarchy structure, which is calculated to include nodes corresponding to bounding volumes defined for corresponding spatial portions of the environment (e.g., floors or rooms of a 3D model of a building). The bounding volume hierarchy structure may include leaf nodes corresponding to objects in the 3D model of the environment. The bounding volume hierarchy structure may be constructed as a structure that can be queried to support efficient spatial queries that are performed when determining which part of an object of the 3D model is to be loaded for rendering based on the user's position in the 3D model of the environment. When a computer determines what to render for a user positioned near a 3D model in virtual space, the computer can efficiently determine a set of objects to load and render based on the user's location, the size of the model, and the processing capabilities of the user's device (e.g., the model load limit for rendering a frame at the computer).

[0046] At 220, a set of objects in the 3D model is ranked to form an object hierarchy based at least on a distance between each object in the set of objects and a specified viewing angle of a user within the environment. In some implementations, the specified viewing angle of the user may include a position of the user and an orientation of the user's camera at the position.

[0047] In some implementations, a set of objects may be determined by querying a tree data structure to determine objects within a "confidence zone" around a user's position within the environment. Such a confidence zone may be considered to be an area within a threshold range around a user's specified viewing angle within the environment. In some implementations, a confidence zone may be defined as a volume of a given shape (such as a sphere, capsule, or frustum, among other example shapes). In some examples, a confidence zone may be considered to be a room of a building (as an example 3D model) that a user's camera view (e.g., the user's avatar in a 3D environment) is currently viewing from inside the room. In some implementations, once a set of objects is determined, those objects may be ranked based on calculating the distance between each object and the user's specified viewing angle within the environment, such that closer objects may be given a higher ranking (or priority). The distance between an object and a user's specified viewing angle may be determined using a 3D spatial access tree data structure and positional information about the objects of its model.

[0048] For example, when specifying a user's perspective, objects that fall within the user's direct field of view may be determined, however, there may be objects that are in close proximity to the user but not in their field of view (e.g., directly behind the user), or there may be objects that are hidden by other objects but are still in close proximity to the user. In some implementations, the query tree data structure may provide a set of objects that fall within the user's confidence zone and may be objects that are visible or hidden to the user from the user's specified perspective. In some implementations, the loading strategy may be configured to determine the confidence zone as an area around the user's avatar within the 3D environment (the location where the user views the 3D model in the environment). In some cases, the confidence zone may be defined to include objects that may be outside the field of view of the user's camera but within a distance (e.g., defined with respect to a threshold distance for filtering close objects) of the location. For example, if the user is currently positioned in a room of a building and has an orientation toward the interior of the room, it may be determined that the user's confidence zone includes objects that are directly behind the user and not visible to the user, but if the user turns around or changes its orientation, such objects may quickly become part of its field of view and become visible to the user. In addition and for example, some objects may be obscured even if they fall within the area around the user and the user's perspective. Such objects may not need to be rendered immediately because they are not visible from the user's current perspective, but due to close proximity or other factors, those objects may be considered for inclusion in the confidence zone and considered to rank and prioritize which objects to load. In some implementations, objects from the confidence zone are ranked according to a loading strategy that can be configured to consider rendering objects that are occluded or not immediately in the user's viewing area. This preemptive loading, which can take into account the user's position, the size of the model and the object, and the FPS and memory consumption of the rendering device, can support smooth rendering during the process of the user moving in the 3D space, where the direction may change suddenly. A query is performed on a set of objects to determine the objects (and correspondingly the leaf nodes of the tree data structure) to be rendered in a specified volume (e.g., a frustum, capsule, or other) among all objects of the 3D model. In some implementations, an occlusion mechanism may be implemented to be able to determine that objects that are within the confidence zone but not visible must be ranked lower. Once the objects are prioritized and before the selection of the objects to be rendered is made, an occlusion mechanism may be implemented.

[0049] In some implementations, a group of objects are provided with a ranking corresponding to their loading priority. The priority may be defined according to a priority rule (and a prioritization criterion) that may be used to weigh the model size and the distance of the object from the user's location. In addition, in some cases, the priority of the objects in the confidence zone may be calculated by considering the visibility of the object from the user's specified perspective (e.g., completely within the "confidence zone", occluded by other objects, or partially occluded). Therefore, objects that are closely adjacent to the user but fall behind another object and are obscured (partially or completely hidden behind the object or based on the shape of the volume) and / or behind the user but do not fall in the viewing area but are still closely adjacent may be considered by the loading strategy and ranked according to the loading strategy. In some implementations, the prioritization criterion for object loading may define a factor associated with loading a first object in preference to loading another object. In some implementations, the prioritization may be based on user-defined criteria related to the properties of the object. In some cases, metadata of the object may be stored to include the category of the object, such as a wall, staircase, pipe, etc. In more cases, the metadata of the object may also include metadata of the location of the object within the environment, such as the current floor or room where the object is located. The factors include the ratio between the size of the object and the size of the model, the user's specified viewing angle, the distance between each object in a set of objects and the user's viewing angle, the user's position within the three-dimensional model, the relative position between each of the objects in the environment from the user's perspective, and the physical properties of the objects (e.g., objects made of see-through (i.e., transparent) materials, shader complexity, or other precise indicators of the physical characteristics of the objects). For example, based on the user's position, a set of objects may fall within the user's direct field of view, where some objects located in this area may be occluded by other objects. In some cases, the occlusion may be partial, as some objects may cover some other objects but not completely cover them. In addition, in some cases, some objects may have physical properties, such as being transparent or reflective (e.g., they may be translucent or fully transparent materials (e.g., windows, glass doors, crystal vases, etc.) or reflective like mirrors). Some or all of these factors may be considered when generating a loading strategy, where in some cases it may even be determined during runtime which factors to include in the loading strategy and which dynamic modifications to make at which iteration. The prioritization criteria may be defined as a weighted combination of two or more of the defined factors, wherein each of the two or more factors is associated with a predefined weight of the combination.

[0050] In some implementations, the objects determined at 220 can be filtered to provide a subset of objects that are relevant to the current rendering, such that loading can be performed for the subset of objects because those objects have a higher importance.

[0051] At 230, an appropriate subset of a set of objects to render is selected by the computer based on the object hierarchy and the current model load limit. The selection is performed based on those objects in the filtered objects having the highest priority (as determined by the ranking) and fitting within the current model load limit.

[0052] In some implementations, determining the appropriate subset can be performed at an external service or platform, where the computer can communicate with a visualization service or platform (such as Figure 1 Visualization service 165 or Figure 1 In some cases, filtering and determining which objects will be rendered by the computer at the display device may be performed on such an external service or platform.

[0053] At 240, an appropriate subset of the set of objects is downloaded to a local memory of the computer. At 250, an appropriate subset of the set of objects is rendered from the local memory to a display device based on a user's specified viewing angle within the environment.

[0054] At 260, the current model load limit is updated based on the rendered current frame generation rate. In some implementations, the current frame generation rate is monitored and updates to the load limit are performed periodically or dynamically, for example based on identifying events (such as based on detection of a rate change exceeding a threshold).

[0055] At 270, downloaded objects cached at the local memory may be removed from the cache based on the object hierarchy (e.g., and the corresponding ranking of the objects) and in response to reaching the current model load limit within a threshold distance due to the download. In some implementations, the downloaded objects are a suitable subset that includes a plurality of objects that are above or close to the load limit for rendering the frame. Thus, the loading strategy may include a dynamic unloading step that includes reducing the model load limit, which may reduce the number of objects to be stored in the cache of the computer. Thus, at least some of the objects that have been downloaded may be removed in response to reaching the current model load limit within a threshold distance.

[0056] For example, the current model load limit may be defined as an FPS rate and a percentage of memory usage of the CPU / GPU that should not be exceeded. In some cases, the FTS rate may be set to be acceptable within a certain range (e.g., between 40 and 75 FPS), and the memory usage may be set to be acceptable up to 80% to allow the user device to render objects smoothly during navigation of the user in the space, even if some of the objects that need to be loaded have not been preloaded. In some implementations, the threshold distance defined around the current model load limit may be defined as the distance between the highest acceptable FPS and the percentage of memory usage and the currently observed metrics. For example, the distance between the highest acceptable FPS and the currently observed FPS may be set to 5 or 10 FPS, so when the FPS reaches 65 or 70 FPS, the loading strategy may consider starting to remove objects, as discussed with respect to step 270. The threshold distance may be defined as a percentage below the highest acceptable load of the memory of the CPU / GPU, which may be, for example, 5% or 10% below the highest acceptable value (e.g., 80%). It should be understood that these are merely example thresholds and example maximum acceptable metrics that may be configured to different values ​​and used in a load strategy to determine when and whether to remove an object, as described at 270 .

[0057] At 280, it is determined whether the user has changed his or her perspective. For example, it may be determined whether the user has changed his or her orientation without changing position, has changed position and maintained the same viewing direction, or has changed position and viewing direction. If the user has changed his or her position and / or orientation, the steps of selecting, downloading, rendering, updating, and removing are iteratively repeated upon receiving a change in the user's specified perspective within the environment. The user navigates within the environment and changes his or her perspective, so different groups of objects are associated with rankings because the distances between the objects and the user's new perspective have changed. These steps are iteratively repeated during the user's interaction and movement relative to the 3D model. If at 280, it is determined that the user has not changed their perspective, the rendering of an appropriate subset of the group of objects is maintained. In some cases, it may be determined that the user has changed their perspective based on receiving a notification that the user interacts with the 3D space to move and / or change orientation in the 3D space.

[0058] Figure 3A and Figure 3B Example user interactions for navigating within a spatial view of a VR environment rendered by a client device at a display device are shown in accordance with implementations of the present disclosure.

[0059] exist Figure 3A On the top, client applications such as Figure 1The user interface of the client application 116 of the embodiment of the present invention is a user interface in which a 3D model of the environment is rendered. In some implementations, the model rendering is performed in response to a user interaction for requesting access to the 3D model from another platform or service. In some implementations, the user can access a platform storing 3D models and data for visualization of the model. The 3D model can be accessed through different access applications or through a web browser (or other access point). In some implementations, the user can enter a VR environment representing a model of a building, which is rendered at the user interface. The user interface may provide tools and options for rendering the model or parts of the model as the user moves within the environment. In some implementations, such as Figure 3A As shown, the model is rendered from the outside and from the user's perspective. In some implementations, the application for rendering the object may be configured to execute Figure 2 The method renders objects of a model according to a dynamic loading strategy that takes into account a user's perspective and a model load limit of a client device used for rendering.

[0060] In some implementations, Figure 3A and Figure 3B The rendering at can be based on obtaining a 3D space access tree data structure and ranking and filtering objects based on an object hierarchy defined according to a priority ranking and current model load limit. The tree data structure is obtained and the ordering / ranking and filtering / selection can be done with Figure 2 Steps 210, 220 and 230 are performed in a substantially similar manner as described above.

[0061] In some implementations, the user can navigate within the space and define viewing locations of interest within the model. For example, the definition of the locations can be performed based on interactions, such as hand-based interactions with a VR user device or with a mouse and keyboard of a computer. In some implementations, using sensors on a user's headset that interacts with the VR environment, the application can map the user's real-world desk to a virtual desk within the virtual experience when displaying the 3D model. In some implementations, a virtual user interface can be provided to allow the user to select a model to review in the virtual space, where the user can point to various parts of the model to switch his / her location.

[0062] In some implementations, different views of the 3D model may be provided based on the distance of the user to the object of the 3D model and the current load limit of the user device rendering the frame. Figure 2 The decision to load and render objects is performed dynamically as described as the user repositions themselves relative to a model of the environment.

[0063] For example, Figure 3Aincluding a 3D model rendering 300, with Figure 3B In some cases, the presentation is shown from a distance farther from the user's viewing angle than from a distance from the user's viewing angle. Figure 3A The presentation in the example can be called a "dollhouse model" or "zoomed model" view. Figure 3A The model on is farther away, such as in Figure 2 Determining the objects to render can be done by Figure 2 An appropriate subset of matching objects is selected at 230 and those objects are the highest ranked set of objects for rendering priority and are selected based on the current model load limit.

[0064] According to implementations of the present disclosure, a loading strategy may be generated for a 3D model in which a representation 300 of the 3D model is presented to the user at a specific point in time and at a specific location and orientation of the user within the environment. As discussed, this is a representation 300 as seen from outside the model building. Figure 3A In the example of , when the user's location is outside a building, the user may not be able to see objects inside the building even if all or some of the objects are not obscured. For example, the building has windows, and those windows may be transparent so that objects in rooms distributed on each floor are generally visible through the windows and can be seen (e.g., a user should be able to see a stove in a kitchen (or other room) through a window of the building). However, a loading strategy for rendering a representation of the building may be generated to prioritize the outer shell of the building (because it is closer to the user's location and falls within the user's confidence zone) based on the distance between the building and the user. Therefore, when rendering the building, objects from the user's confidence zone, as ranked, may be filtered based on the constraints of the rendering device. As previously discussed, filtering may take into account whether sufficient memory is available to load objects inside the building and identify portions of higher ranked objects that are ranked as relevance (priority) for rendering and render them first, such as in Figure 3A As shown above. Figure 3A The objects presented at are objects that are considered to be closer to the user's location and define the outer shell of the model, so these objects are ranked higher because they are larger objects.

[0065] In some cases, while viewing the rendering 300 of the building model, the user may be interested in navigating closer to a portion of the model, e.g., the lower right area 305 of the building. That portion of the building as shown on the rendering 300 is rendered to include larger objects that can be seen from the outside, while smaller objects inside the building are not shown even though they can be seen through the windows. To view the portion 305 of the rendering in further detail, the user may navigate in the 3D space to reposition him / herself and adjust the viewing direction. For example, the perspective (i.e., the user position and viewing direction) may be moved to a position that is closer to the portion 305 of the building. Figure 3B The user's perspective associated with the new presentation 310 of the model is changed. In the new presentation, different objects associated with the portion of the model are rendered based on the loading policy. Due to the change in the distance of the user from the portion of the model, smaller objects that fall within the confidence zone of the new perspective may be considered and ranked according to the loading policy.

[0066] In some cases, such as regarding Figure 5A and Figure 5B Further discussion, if the user navigates in 3D space and moves around in a building (e.g. Figure 3A and Figure 3B ), without moving closer to the window or directing his gaze toward the window ( Figure 5B ), users may not be able to see objects outside the building through windows (such as Figure 3A Such prioritization of objects to be rendered is a result of the implementation of a loading strategy, where determining a confidence zone for a given viewpoint may force the download of all nearby objects in the room (regardless of whether they are large or small), which would take up memory and not allow room to load objects outside the room that should be visible based on the viewpoint. In some cases, if the user changes his position, for example, closer to a window, a tree outside the room may move to the confidence zone and be considered for loading and rendering according to implementations of the present disclosure.

[0067] exist Figure 3B The objects presented at are objects that are considered closer to the user's new location, which is closer to a particular part of the model, and therefore from that location of the user, a different set of objects may be considered relevant and ranked based on their distance from the user's location. Figure 3B In some cases, smaller objects may be rendered because they are positioned closer to the user's position. As described in the present disclosure, determining which objects to render is based on an object hierarchy defined for objects that fall within a confidence zone around the user's field of view and a load limit for multiple frames to be rendered. By adjusting the selection of objects to be downloaded and rendered based on the current model load limit, the rendering can be dynamically and automatically tuned. In addition, the rendering of frames can be monitored and the limit can be adjusted based on the current frame rate while keeping the frame rate within a range that will not affect the user experience (e.g., causing motion sickness). Thus, the 3D model can be progressively loaded on a base defined at the user interface at a sufficient level of detail that will not affect the performance of the application (e.g., approximately 60 FPS and low latency) and will always show a high level representation of the model, such as a complete "shell" (or external view) of the model.

[0068] In some implementations, based on determining a change in the user's position within the environment, such as in rendering Figure 3A The environment of subsequent views after the view Figure 3B In some implementations, the repeated ranking may be for objects that at least partially include previously loaded objects, for example, for Figure 3A Overlapping objects can be ranked with different priorities based on the changed position and possible indication of model load limits (or updates thereof). Figure 3A The rendered ranking and downloaded objects are also selected for Figure 3B If the object is not immediately rendered by a particular scene, then the object may not need to be downloaded because it may still be in the device's cache. Therefore, if the loading strategy can support loading additional objects that are not immediately rendered by a particular scene, the rendering operation can be optimized and performance improved if the loaded object is relevant to subsequent scenes.

[0069] In some implementations, loading objects of the 3D model and rendering them at the display device may be determined based at least in part on memory consumption required for rendering and measurements from monitoring of the frame rate per second when rendering views of the model. In some implementations, the level of detail applied when rendering the 3D model may be dynamically determined based on dynamic adaptation or adjustment of loading based on the current frame generation rate of the rendering. Such dynamic considerations may support faster loading and slower unloading of objects downloaded to the cache to support efficient use of resources, allowing first objects that have been cached to be rendered (and therefore rendered faster) compared to objects that have not yet been downloaded but may be relevant to the user's new location during interaction and repositioning with the environment.

[0070] In some implementations, when presenting a 3D model, some or all of the 3D data associated with the model may be downloaded and used for rendering. For example, different levels of detail associated with rendering a 3D model may affect the amount of data downloaded for data visualization. By reducing the amount of data to be downloaded, network bandwidth may be saved. In some implementations, a user may configure the level of detail associated with the presentation of data at a particular user interface that the user uses to show the 3D model, and based on such configuration, some but not all of the stored 3D data for the 3D model may be downloaded. This filtering of the amount of data to be downloaded may be adjusted dynamically, or may be iteratively trained based on an evaluation of user interactions.

[0071] Figure 4An example system architecture 400 of a client device for rendering a 3D model based on an adaptive dynamic loading strategy according to an implementation of the present disclosure is shown. In some implementations, the client application 460 may be substantially similar to Figure 1 The client application 116 and the client application 460 may be configured to execute Figure 2 At least some of the steps of method 200.

[0072] In some implementations, the client application 460 can be configured to execute a dynamically adaptive loading strategy to determine which objects of the 3D model to download locally and render. Loading strategies can be defined to scale the rendering of large models that cannot be fully downloaded in cache. In some implementations, the client application 460 can request a model to be rendered from the 3D model 415 of the platform 405. The platform 405 can communicate with Figure 1 The platform 145 is substantially similar and may be considered as Figure 2 A remote system, where 3D models of objects can also be downloaded.

[0073] In some implementations, the client application 460 may request to open the model from the 3D model 415 and grab the model description to generate a 3D spatial access tree data structure encoding position information about objects of the 3D model or directly obtain the 3D spatial access tree data structure. In some implementations, the client application may download the 3D spatial access tree data structure directly from the platform 405 if the platform has already generated the 3D spatial access tree data structure, or download the 3D spatial access tree data structure from the visualization service 410 that can connect to the platform 405 and generate the tree data structure from the model 415.

[0074] In some implementations, visualization service 410 may be implemented as a set of separate services, or may be integrated into the logic of client application 460. Visualization service 410 may include data source integration service 425, which may be configured to collect model description data of the model from model 415 and provide the model description data to data preparation service 420. Data source integration service 425 may provide an abstraction layer on top of a data source (e.g., a model source) to obtain properties associated with objects and implementations within the model, as well as identifiers of faces and vertices in the model. Data source integration service 425 may be used in conjunction with data preparation service 420 to query data associated with the model and determine the level of detail of objects from the model.

[0075] Data preparation service 420 may be configured to implement logic for pre-computing certain properties from the metadata of the 3D model that may be used to optimize data processing and determine whether to prioritize objects when performing a selection of the object (e.g., such as the order of the objects in the image). Figure 240 ). The data source integration service 425 can collect data about the properties of models and objects, including textures, material details, pixel colors, or other data. In some implementations, such data about textures, material details, and other metadata of the 3D models can be stored in an asset store 450 accessible to the visualization service 410. In some implementations, the spatial indexing service 430 can generate a 3D spatial access tree data structure (e.g., a bounding volume hierarchy). In some implementations, the data preparation service 420 can communicate with an asset store that stores files of different three-dimensional scenes and models (e.g., in various file formats, such as gITF, DirectDrawSurface (DDS) files, or other formats).

[0076] In some implementations, the data source integration service 425 can be communicatively coupled with the spatial index service 430. In some implementations, the spatial index service 430 can generate a tree data structure that includes objects of the model as leaves and their locations within the model. The spatial index service 430 can initialize the hierarchy information and store it at a local or remote storage device, wherein the generated tree data structure can be queried to obtain the priority objects within a given volume. By querying the generated tree data structure, a resulting unordered list of a set of identifiers of the objects can be obtained. In some implementations, the list can be used to obtain the objects based on the data at the data source integration service.

[0077] In some implementations, when the client application 460 connects with the visualization service 410, the loaded web socket 470 of the client application 460 can be used to connect with the visualization service 410 to open the model and render the object on a display device (not shown). Upon receiving the request at the visualization service 410, the spatial index service 430 can generate an unordered list of identifiers of objects associated with the user's given perspective and the requested model and provide the list to the prioritization service 440, which can perform a ranking (e.g., substantially similar to the ranking of objects with respect to the visualization service 410). Figure 2 The prioritization service 440 may implement logic for sorting a set of objects based on a set of prioritization rules that may be defined for the client application 460 (e.g., as external input to a particular rendering, or as a configuration of default settings in the client application). In some implementations, the prioritization rules may be defined at different granularities for different target users (e.g., user roles, user groups, etc.) at the client application.

[0078] In some implementations, rendering of the object of the 3D model may be performed in a context where a user interacts with another user in the space (e.g., playing a game, collaborating on a task), wherein the other user interacts with another device that displays the object of the 3D model. In some implementations, each of the two users navigates in the 3D space and is displayed with the rendered object as determined at each of their devices. The two users may rely on loading strategies that are used to determine which objects to load, respectively. In some implementations, for each of the two users, a confidence zone may be determined (as previously discussed). In some cases, the confidence zone of one of the users may be determined by considering the area determined to be the "confidence zone" of the other user. For example, if one of the users has object A in its confidence zone, the confidence zone of the other user (if it does not already include object A) may be extended to include object A. The determination of whether to adjust the confidence zone of one user relative to another user may be defined based on configured importance rules, which may include using roles in an application (e.g., two players of the same game, users of connected roles, etc.) or based on proximity between their locations and / or orientations to each other. For example, if two users are positioned within a threshold distance in the same room of a building and are oriented facing each other, the confidence region may be determined taking into account the position and orientation of the other user and / or also taking into account constraints from rendering at the client device (e.g., memory consumption and FPS).

[0079] The prioritization rules may define weighting factors for generating a ranking for each of the objects based on consideration of a number of factors, including the user's location and distance from the user to the object, the size of the model and relative to the size of the object, the user's perspective (including the user's location and orientation), the definition of a "confidence zone," and an initial configuration of the model load limit for scaling the rendering performed by the client application 460. The prioritization service 440 may process an unordered input list of identifiers of the objects and generate as output an ordered array of identifiers of the objects, which may be obtained from the data source integration service 425 and loaded into the cache 480. In some implementations, the data source integration service 425 may be implemented as part of a service provided by the platform 405. In some implementations, while the data source integration service 425 is supported from the platform, the data preparation and tree data structure generation implemented at the data preparation service and spatial index service 430 may be implemented as an external service or part of a service implemented by the client application 460.

[0080] In some implementations, the prioritization service 440 can provide the output ordered list to the dynamic loading service 445, which can implement automatic tuning rules 447 for dynamically determining the number of objects to load and render based on the frames per second rate monitored at the client application 460 and / or based on the memory consumption of the client application 460 when processing and loading the objects to be rendered. The automatic tuning rules 447 can be iteratively processed (e.g., when rendering a group of objects) to update the current model load limit based on the observed current frame generation rate of the rendering.

[0081] In some implementations, the dynamic loading service 445 may maintain state defining which objects are loaded and unloaded and pending operations on the client side since the last update of the model load limit. The dynamic loading service 445 may implement filtering rules that are used to determine which subset of those objects from the output ordered list will be downloaded into the cache and rendered at the display device. The dynamic loading service 445 may be configured with tools for calculating the current state of loading and unloading over a monitored time period and related to a given session for user interaction with the model, and utilize information from the spatial index service 430 and filtering service 435 for the model being loaded to select those objects that meet the auto-tuning limits that define the current model load limit (automatically adjusted based on the current frame generation rate of the rendering). Based on the object identifier determined by the dynamic loading service, the object can be called by the data source integration service 425 and downloaded at the cache 480.

[0082] In some implementations, when downloading the properties of the objects and the objects, the engine 475 of the client application may render the determined subset of objects downloaded in the cache and render the scene including the objects at the display device.

[0083] Figure 5A and Figure 5B An example of user interaction with a user interface including a presentation 500 and 550 of 3D data of a 3D model provided by a client application on a user device is shown. Figure 5A The user interface 500 may provide a tool for navigating within a 3D model of an environment, wherein objects are rendered according to an adaptive dynamic loading strategy implemented in accordance with an implementation of the present disclosure. The user interface 500 may include a view of an object rendered from the model, the model being associated with the 3D model. Figure 3A and Figure 3BIn some implementations, the user can navigate based on interaction with menu options provided at user interface 500 to move within the model and be provided with rendered objects associated with his new location, wherein the selection of objects to be loaded and rendered and subsequently unloaded based on an update to the current model load limit can be associated with Figure 2 The described method 200 proceeds in a substantially similar manner, including iteratively repeating the ranking, selecting, downloading, rendering, updating, and removing of objects from the 3D model as the user's perspective within the environment changes.

[0084] When the 3D model is a 3D model of a building (such as Figure 3A and Figure 3B In one embodiment, when a user is viewing a rendered view of a building at a certain location, the shell of the model is a "dollhouse" representation of the building from the outside. Typically, even if the dollhouse representation of the 3D model has a fine detail granularity or a lower level of detail (or somewhere in between), by moving within the environment, the desired portion of the 3D data (which is not fully rendered in the dollhouse representation) can be easily identified based on the identified user position and maintained in an amount that can be rendered sustainably on the corresponding device used for rendering.

[0085] Interact with 3D models and navigate within the 3D environment (such as Figure 3A and Figure 3B A user (as shown above) can switch from a perspective outside a building to a perspective inside a room in the building (as shown in Figure 5A and Figure 5B shown above). Figure 5A The rendering 500 of the 3D model is viewed from inside a room, the room including a first wall 505 with a window and a second wall 510 with a window on the right. Although the windows are glass windows and are see-through, the rendering 500 does not include all objects that are outside the building and visible. The loading strategy used to render the rendering 500 takes into account the user's position and the distance between the user and other objects in the room to determine which objects in the user's confidence zone to render based on observing the current load model limits. The loading strategy does not select all objects outside the windows, and for example, includes some objects on wall 500, but does not include objects on wall 510.

[0086] In contrast, Figure 5B The presentation of 550 is from closer Figure 5A500. When rendering the rendering 550, the loading strategy has taken into account the distance from the user's perspective to the external objects, and the loading strategy is used to select the objects to be loaded and rendered on the device. As shown, the window 555 is a see-through window, and based on the loading strategy, the objects that can be seen from the window are rendered, so the rendering is different from the rendering in FIG. Figure 5A More objects from within the building model can be seen than what can be seen on the Figure 5A When you open the viewport in , some objects are not selected for loading and rendering.

[0087] like Figure 5A As shown, when the user's perspective is inside a room of a building, objects outside the building may not be presented to the user, even if these objects are not obstructed (for example, another building outside a window in the wall of the room should be visible), because the confidence region has forced the download of all nearby objects in the room when determining and filtering, thus leaving no memory for objects outside the room to download and render, even if these objects should be visible from this perspective. Figure 5B As shown, objects outside the window are rendered because the loading policy has evaluated a set of new objects to be included in the new confidence zone and those objects have been ranked based on their distance to the user's viewing angle and filtered to fit the memory capacity of the user's device. In this example, the objects shown as visible through the window 555 are those objects in the 3D model that were filtered to be included for download and rendering as ranked according to the loading policy.

[0088] By determining which objects to load and render according to a dynamically adjustable loading strategy according to the present implementation, flexible and efficient switching between different display modes and different levels of data to be rendered can be supported. As shown in the figure, the loading strategy can determine when the user is viewing from outside the model (such as Figure 3A and Figure 3B ), and the loading strategy determines which objects to render and when switching perspectives and moving inside a mode (as shown Figure 5A and Figure 5B Such an implementation can improve the timeliness of rendering of 3D data, and can also improve the performance of client applications and their responsiveness to user interactions received through the device used by the user to interact with the data and navigate within the view of the 3D model.

[0089] The user interface 500 may be associated with a particular frame generation rate (e.g., 71, 77, or 70, among other examples), wherein based on the current frame generation rate, an update to the model load limit may be determined as an automatic tuning factor (e.g., an automatic tuning factor update to reduce the model load limit by 29%). By dynamically adjusting the model load limit, the selection of objects to be loaded and objects to be unloaded may be made efficiently to support scalability of rendering at the device, which allows for consistent performance as the user moves within the environment and changes perspective. Priority objects considered for rendering are those that can accommodate the model load limit (the limit threshold of polygons to be loaded) so that the system can afford to load and render these objects.

[0090] Figure 6 6 is a schematic diagram of a data processing system including a data processing device 600, which can be programmed as a client or a server. The data processing device 600 is connected to one or more computers 690 via a network 680. Figure 6 Only one computer is shown as data processing device 600, but multiple computers may be used. Data processing device 600 includes various software modules that may be distributed between the application layer and the operating system. These may include executable and / or interpretable software programs or libraries, including tools and services of client application 604, which includes a user interface that allows a 3D model of the environment to be displayed to a user of client application 604 in one or more audit modes. In addition, client application 604 may implement an adaptive dynamic loading strategy to determine and load a portion of an object of the 3D model, which will facilitate the client device to continue to perform efficient rendering without downtime due to load overload.

[0091] The number of software modules used may vary from implementation to implementation. In addition, the software modules may be distributed on one or more data processing devices connected via one or more computer networks or other suitable communication networks.

[0092] The data processing device 600 also includes a hardware or firmware device, including one or more processors 612, one or more additional devices 614, a computer-readable medium 616, a communication interface 618, and one or more user interface devices 620. Each processor 612 is capable of processing instructions for execution within the data processing device 600. In some implementations, the processor 612 is a single-threaded or multi-threaded processor. Each processor 612 is capable of processing instructions stored on a computer-readable medium 616 or a storage device (such as one of the additional devices 614). The data processing device 600 uses a communication interface 618 to communicate with one or more computers 690, for example, via a network 680. Examples of user interface devices 620 include displays, cameras, speakers, microphones, tactile feedback devices, keyboards, mice, and VR and / or AR devices. The data processing device 600 may store instructions for implementing operations associated with the above-mentioned programs on, for example, a computer-readable medium 616 or one or more additional devices 614, such as one or more of a hard disk device, an optical disk device, a tape device, and a solid-state memory device.

[0093] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented using one or more computer program instruction modules encoded on a non-transitory computer-readable medium for execution by a data processing device or for controlling the operation of a data processing device. The computer-readable medium may be a manufactured product, such as a hard drive in a computer system or an optical disk sold through a retail channel, or an embedded system. The computer-readable medium may be obtained separately and later encoded with one or more computer program instruction modules, for example, after transmitting one or more computer program instruction modules through a wired or wireless network. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more of them.

[0094] The term "data processing apparatus" encompasses all devices, means and machines for processing data, including, for example, a programmable processor, a computer or a plurality of processors or computers. In addition to hardware, the apparatus may also include code that generates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, a runtime environment, or a combination of one or more of them. In addition, the apparatus may employ a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0095] Computer programs (also referred to as programs, software, software applications, scripts, or codes) may be written in any suitable form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any suitable form, including as independent programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files that store portions of one or more modules, subroutines, or codes). A computer program may be deployed to execute on one computer, or on multiple computers located at one location or distributed across multiple locations and interconnected by a communication network.

[0096] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and the device may also be implemented as, a special purpose logic circuit, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0097] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. In general, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or operatively coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer does not need to have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and storage devices, including, for example, semiconductor memory devices, such as EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0098] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having: a display device, such as a liquid crystal display (LCD) device, an organic light emitting diode (OLED) display device, or another monitor for displaying information to a user; and a keyboard and pointing device, such as a mouse or trackball, by which a user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any suitable form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user in any suitable form may be received, including acoustic, voice, or tactile input.

[0099] A computing system may include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The relationship between the client and the server is generated by a computer program that runs on a corresponding computer and has a client-server relationship with each other. The embodiments of the subject matter described in this specification may be implemented in a computing system, which includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a browser user interface that a user can use to interact with the implementation of the subject matter described in this specification), or any combination of such back-end, middleware or front-end components. The components of the system may be interconnected by digital data communication (e.g., a communication network) of any suitable form or medium. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0100] Although this specification contains many implementation details, these should not be interpreted as limitations on the scope of what is claimed or may be claimed, but rather as descriptions of features that are peculiar to a particular embodiment of the disclosed subject matter. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as acting in certain combinations and even initially claimed for this purpose, one or more features from a claimed combination may be separated from the combination in some cases, and a claimed combination may involve a sub-combination or a variation of a sub-combination.

[0101] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can be generally integrated together in a single software product or packaged into multiple software products.

[0102] Thus, certain embodiments of the invention have been described. Other embodiments are within the scope of the following claims. Additionally, the actions recited in the claims can be performed in a different order and still achieve desirable results.

[0103] Example

[0104] Although the present application is defined in the appended claims, it will be appreciated that the invention may also (additionally or alternatively) be defined according to the following examples:

[0105] Example 1. A computer-implemented method comprising:

[0106] obtaining, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding position information about objects in a three-dimensional model of an environment, wherein the three-dimensional model is stored on a remote computer system;

[0107] ranking, by the computer, the set of objects to form an object hierarchy based at least on a distance between each object in the set of objects in the three-dimensional model and a specified viewing angle of a user within the environment as determined using the three-dimensional spatial access tree data structure;

[0108] selecting, by the computer, an appropriate subset of the set of objects to render based on the object hierarchy and a current model load limit;

[0109] downloading, by said computer and from said remote computer system, said appropriate subset of said set of objects to said local memory; and

[0110] The appropriate subset of the set of objects is rendered, by the computer, from the local memory to the display device based on the specified perspective of the user within the environment.

[0111] Example 2. The method of Example 1, wherein the three-dimensional spatial access tree data structure comprises a bounding volume hierarchy, and the obtaining comprises:

[0112] The bounding volume hierarchy is calculated, the bounding volume hierarchy comprising nodes corresponding to bounding volumes defined for respective spatial portions of the environment, wherein leaf nodes of the bounding volume hierarchy correspond to the objects in the three-dimensional model of the environment.

[0113] Example 3. A method as described in any of the preceding examples, further comprising:

[0114] updating, by the computer, the current model load limit based on a current frame generation rate of the rendering; and

[0115] The downloaded objects are removed from the local memory by the computer based on the object hierarchy and in response to the current model load limit being reached within a threshold distance due to the downloading.

[0116] Example 4. The method as described in Example 3 further includes:

[0117] The ranking, the selecting, the downloading, the rendering, the updating, and the removing are repeated by the computer upon receiving a change in the specified perspective of the user within the environment.

[0118] Example 5. A method as described in any of the preceding examples, wherein obtaining the three-dimensional space access tree data structure comprises:

[0119] The three-dimensional spatial access tree data structure of the environment is received in a communications stream from the remote computer system.

[0120] Example 6. The method of any of the preceding examples, wherein ranking the set of objects in the three-dimensional model comprises:

[0121] identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to be within a confidence zone around the specified viewing angle of the user based on matching a prioritization criterion for object loading,

[0122] Wherein the group of objects is selected based on identifying the group of objects as highest ranked objects from the objects as ranked to be loaded according to the current model load limit.

[0123] Example 7. A method as described in Example 6, wherein the prioritization criteria for object loading define factors associated with prioritizing the loading of a first object over the loading of another object, wherein the factors include a comparison ratio between the size of the object and the size of the model, the specified viewing angle of the user, the distance between each object in the set of objects and the specified viewing angle of the user, the position of the user within the three-dimensional model, the comparative position between each two of the objects within the environment from the user's viewing angle, and the physical properties of the objects.

[0124] Example 8. The method of Example 7, wherein the prioritization criteria is defined as a weighted combination of two or more of the defined factors, wherein each of the two or more factors is associated with a predefined weight of the combination.

[0125] Example 9. A method as described in any of the preceding examples, comprising:

[0126] determining a change in the perspective of the user within the environment; and

[0127] The ranking is repeated for a second set of objects in the environment, wherein the ranking is repeated for objects that at least partially overlap with the set of ranked objects as the first set of ranked objects, wherein the ranking is performed based on distances between objects in the second set of objects and a new perspective of the user determined based on the change in position to generate a new object hierarchy.

[0128] Example 10. The method of Example 9, wherein at least one object of the second set of objects is downloaded to the local memory as part of the appropriate subset of the set of objects.

[0129] Example 11. A method as described in any of the preceding examples, wherein the three-dimensional spatial access tree data structure encodes metadata for the object, the metadata comprising an object category and an environmental property, the environmental property defining a location of the object within the environment, and wherein the ranking of the set of objects comprises:

[0130] The set of objects is identified as including one or more objects from the objects in the three-dimensional model that are determined to match a prioritization criterion based on the metadata of the objects.

[0131] Similar operations and processes as described in Examples 1 to 11 may be performed in a system including at least one process and a memory, the memory being communicatively coupled to at least one processor, wherein the memory stores instructions that, when executed, cause at least one processor to perform operations. In addition, a non-transitory computer-readable medium may also be implemented that stores instructions that, when executed, cause at least one processor to perform operations as described in any one of Examples 1 to 11.

Claims

1. A computer-implemented method comprising: obtaining, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding position information about objects in a three-dimensional model of an environment, wherein the three-dimensional model is stored on a remote computer system; ranking, by the computer, the set of objects to form an object hierarchy based at least on a distance between each object in the set of objects in the three-dimensional model and a specified viewing angle of a user within the environment as determined using the three-dimensional spatial access tree data structure; selecting, by the computer, an appropriate subset of the set of objects to render based on the object hierarchy and a current model load limit; downloading, by said computer and from said remote computer system, said appropriate subset of said set of objects to said local memory; as well as The appropriate subset of the set of objects is rendered, by the computer, from the local memory to the display device based on the specified perspective of the user within the environment.

2. The method of claim 1 , wherein the three-dimensional spatial access tree data structure comprises a bounding volume hierarchy, and the obtaining comprises: The bounding volume hierarchy is calculated, the bounding volume hierarchy comprising nodes corresponding to bounding volumes defined for respective spatial portions of the environment, wherein leaf nodes of the bounding volume hierarchy correspond to the objects in the three-dimensional model of the environment.

3. The method of claim 1, further comprising: updating, by the computer, the current model load limit based on a current frame generation rate of the rendering; as well as The downloaded objects are removed from the local memory by the computer based on the object hierarchy and in response to the current model load limit being reached within a threshold distance due to the downloading.

4. The method of claim 3, further comprising: The ranking, the selecting, the downloading, the rendering, the updating, and the removing are repeated by the computer upon receiving a change in the specified perspective of the user within the environment.

5. The method of claim 1, wherein obtaining the three-dimensional space access tree data structure comprises: The three-dimensional spatial access tree data structure of the environment is received in a communications stream from the remote computer system.

6. The method of claim 1, wherein ranking the set of objects in the three-dimensional model comprises: identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to be within a confidence zone around the specified viewing angle of the user based on matching a prioritization criterion for object loading, Wherein the group of objects is selected based on identifying the group of objects as highest ranked objects from the objects as ranked to be loaded according to the current model load limit.

7. A method as claimed in claim 6, wherein the prioritization criteria for object loading define factors associated with prioritizing the loading of a first object over the loading of another object, wherein the factors include a comparison ratio between the size of the object and the size of the model, the specified viewing angle of the user, the distance between each object in the set of objects and the specified viewing angle of the user, the position of the user within the three-dimensional model, the comparative position between each two of the objects within the environment from the user's viewing angle, and the physical properties of the objects.

8. The method of claim 7, wherein the prioritization criteria is defined as a weighted combination of two or more of the defined factors, wherein each of the two or more factors is associated with a predefined weight of the combination.

9. The method of claim 1, comprising: determining a change in the perspective of the user within the environment; as well as The ranking is repeated for a second set of objects in the environment, wherein the ranking is repeated for objects that at least partially overlap with the set of ranked objects as the first set of ranked objects, wherein the ranking is performed based on distances between objects in the second set of objects and a new perspective of the user determined based on the change in position to generate a new object hierarchy.

10. The method of claim 9, wherein at least one object of the second set of objects is downloaded to the local memory as part of the appropriate subset of the set of objects.

11. The method of claim 1 , wherein the three-dimensional spatial access tree data structure encodes metadata for the objects, the metadata comprising object categories and environmental properties, the environmental properties defining the locations of the objects within the environment, and wherein the ranking of the set of objects comprises: The set of objects is identified as including one or more objects from the objects in the three-dimensional model that are determined to match a prioritization criterion based on the metadata of the objects.

12. A system comprising: a non-transitory storage medium having instructions of a computer-aided design program stored thereon; as well as one or more data processing devices configured to execute the instructions of the computer-aided design program to perform operations comprising: obtaining, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding position information about objects in a three-dimensional model of an environment, wherein the three-dimensional model is stored on a remote computer system; ranking, by the computer, the set of objects to form an object hierarchy based at least on a distance between each object in the set of objects in the three-dimensional model and a specified viewing angle of a user within the environment as determined using the three-dimensional spatial access tree data structure; selecting, by the computer, an appropriate subset of the set of objects to render based on the object hierarchy and a current model load limit; downloading, by said computer and from said remote computer system, said appropriate subset of said set of objects to said local memory; and The appropriate subset of the set of objects is rendered, by the computer, from the local memory to the display device based on the specified perspective of the user within the environment.

13. The system of claim 12, wherein the non-transitory storage medium further has instructions that, when executed by the one or more data processing devices, perform operations, the operations further comprising: updating, by the computer, the current model load limit based on a current frame generation rate of the rendering; as well as The downloaded objects are removed from the local memory by the computer based on the object hierarchy and in response to the current model load limit being reached within a threshold distance due to the downloading.

14. The system of claim 13, wherein the non-transitory storage medium further has instructions that, when executed by the one or more data processing devices, perform operations, the operations further comprising: The ranking, the selecting, the downloading, the rendering, the updating, and the removing are repeated by the computer upon receiving a change in the specified perspective of the user within the environment.

15. The system of claim 12, wherein the non-transitory storage medium further has instructions that, when executed by the one or more data processing devices, perform operations, the operations further comprising: determining a change in the perspective of the user within the environment; as well as The ranking is repeated for a second set of objects in the environment, wherein the ranking is repeated for objects that at least partially overlap with the set of ranked objects as the first set of ranked objects, wherein the ranking is performed based on distances between objects in the second set of objects and a new perspective of the user determined based on the change in position to generate a new object hierarchy.

16. A non-transitory computer-readable medium encoding instructions operable to cause a data processing apparatus to perform operations comprising: obtaining, by a computer having a display device and a local memory, a three-dimensional spatial access tree data structure encoding position information about objects in a three-dimensional model of an environment, wherein the three-dimensional model is stored on a remote computer system; ranking, by the computer, the set of objects to form an object hierarchy based at least on a distance between each object in the set of objects in the three-dimensional model and a specified viewing angle of a user within the environment as determined using the three-dimensional spatial access tree data structure; selecting, by the computer, an appropriate subset of the set of objects to render based on the object hierarchy and a current model load limit; downloading, by said computer and from said remote computer system, said appropriate subset of said set of objects to said local memory; as well as The appropriate subset of the set of objects is rendered, by the computer, from the local memory to the display device based on the specified perspective of the user within the environment.

17. The non-transitory computer readable medium of claim 16, wherein ranking the set of objects in the three-dimensional model comprises: identifying the set of objects as including one or more objects from the objects in the three-dimensional model that are determined to be within a confidence zone around the specified viewing angle of the user based on matching a prioritization criterion for object loading, Wherein the group of objects is selected based on identifying the group of objects as highest ranked objects from the objects as ranked to be loaded according to the current model load limit.

18. A non-transitory computer-readable medium as described in claim 17, wherein the prioritization criteria for object loading define factors associated with prioritizing the loading of a first object over the loading of another object, wherein the factors include a comparison ratio between the size of the object and the size of the model, the specified viewing angle of the user, the distance between each object in the set of objects and the specified viewing angle of the user, the position of the user within the three-dimensional model, the comparative position between each two of the objects within the environment from the user's viewing angle, and physical properties of the objects.

19. The non-transitory computer-readable medium of claim 18, wherein the prioritization criteria is defined as a weighted combination of two or more of the defined factors, wherein each of the two or more factors is associated with a predefined weight for the combination.

20. The non-transitory computer-readable medium of claim 18, wherein the three-dimensional spatial access tree data structure encodes metadata for the object, the metadata comprising an object category and an environmental property, the environmental property defining a location of the object within the environment, and wherein the ranking of the set of objects comprises: The set of objects is identified as including one or more objects from the objects in the three-dimensional model that are determined to match a prioritization criterion based on the metadata of the objects.

Images