Virtual or real-world multidimensional 3D engine computing and virtualization-based dynamic load balancing

Abstract

To provide a system, method and non-transitory computer-readable medium that enable a distributed 3D engine for performing dynamic load balancing through virtual worlds.SOLUTION: A system 100 comprises one or more servers comprising memory and at least one processor. The memory stores: a data structure representing at least one portion of a virtual or real world as a plurality of cells storing virtual objects; and a distributed 3D engine comprising a resource manager implemented in a distributed deployment and a plurality of individual software engines. Resources are dynamically allocated via the distributed deployment to one or more cells based on a current load and a corresponding computed and ranked demand. In some embodiments, the demand further considers one or more of the amount of virtual objects and the level of interactions within the portion of the persistent virtual world system 112 visible to a user avatar.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 955,247, filed December 30, 2019, the contents of which are incorporated herein by reference. [Background technology]

[0002] Conventional systems designed to run virtual worlds, particularly massively multiplayer online games (MMOGs), are limited in how they can scale to support a large number of players or entities within the virtual world. Because these virtual worlds typically run on a single physical computer and rely on a single instance of the game or application, the virtual world is supported by the limited bandwidth of the system and the processor capacity of the server hosting the virtual world. The server hosting the game or simulation engine must provide simulation functionality for all objects and players within the virtual world. However, as the number of players and entities increases, resource requirements similarly increase, degrading the quality of service (QOS) and user experience within the virtual world.

[0003] Other approaches involve splitting the map and distributing it evenly, running parts of the map on multiple servers, each responsible for a corresponding part of the map. While more resources are available to serve the entire map, this method still has some drawbacks. For example, congestion in one zone can make this type of geographic division inefficient for use in virtual worlds, and transitioning from one zone to another can provide a discrete view, as opposed to a continuous view, to users crossing zones due to the required server switching.

[0004] Additionally, current 3D engines used in virtual world development and processing have several drawbacks that limit the capabilities of virtual world provisioning. For example, typical current 3D engines tend to be monolithic, centralized, can be difficult to scale up, and can be platform dependent. This, in addition to the drawbacks mentioned above, prevents the ability to maximize the capabilities of each particular module of the 3D engine during virtual world development and execution. Summary of the Invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are described in more detail below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] The present disclosure relates generally to computer systems and methods, and more particularly to systems and methods that enable multi-dimensional 3D engine computing and virtualization-based dynamic load balancing for virtual or real worlds.

[0007] The disclosed system includes one or more server computers with a memory and at least one processor, wherein the memory stores a data structure that virtualizes a virtual world or at least a portion of a real world into a plurality of cells that store virtual objects that form a persistent virtual world system. In some embodiments, at least some of the virtual objects are virtual replicas of corresponding real-world elements. The memory further stores a distributed 3D engine implemented in a distributed deployment, the distributed 3D engine including a resource manager and a plurality of individual distributed software engines. Resources are dynamically allocated by the distributed deployment to one or more cells based on current load and corresponding calculated and ranked demand of the one or more cells. In some embodiments, the demand is based at least in part on the amount of virtual objects in one or more cells, the level of interaction within the portion of the persistent virtual world system visible to user avatars in the one or more cells, or a combination thereof. In some embodiments, the resources include computing power, memory, network resources (e.g., bandwidth), or a combination thereof.

[0008] In further embodiments, the demand is further based on the distance between a user device accessing one or more cells and network equipment (e.g., a server and a network antenna), the type of application being used by the user, or the type of entitlement a user accessing one or more cells may have (e.g., based on the user's individual user profile or contractual terms), or a combination thereof.

[0009] In some embodiments, virtual objects further include purely virtual objects that are not available in the real world, virtual avatars of real people, and applications.

[0010] In some embodiments, a persistent virtual world system includes multiple simulations of the same virtual or real world, varying from high fidelity to low fidelity, with the low fidelity simulations used for demand assessment and load balancing and the high fidelity simulations used to improve the user experience. This allows for different simulations depending on the occasion or situation. LOD management also makes it possible to provide an optimized user experience depending on the specific requirements and context. Thus, the same virtual or real world can be simulated at different LODs to facilitate the load balancing methods of the present disclosure while providing an appropriate experience to the user.

[0011] In some embodiments, virtual objects of a persistent virtual world system include data and models that refer to any graphical, mathematical, or logical representation of aspects of virtual objects or corresponding real objects. In some embodiments, the data and models provide entities of the persistent virtual world system with self-computing capabilities and autonomous behavior. In some embodiments, suitable models include one or more of 3D models, geometric models, dynamic models, and machine learning models. Virtual objects can be fed with data obtained from a simulation, data input by a user via a user device, or data captured by sensor devices such as Internet of Things devices (e.g., in the case of a persistent virtual world system based on the real world). A continuous flow of data can maintain a persistent virtual world system that is periodically updated.

[0012] In some embodiments, a persistent virtual world system may include virtual world layers that may be separated into an augmented reality (AR) layer and a virtual reality (VR) layer. The separate layers may allow access to the persistent virtual world system in either augmented reality or virtual reality and combine elements stored in each layer, and may be activated via a client device connected to at least one server computer via a network whenever accessing one or the other type of reality. Each of the layers may include augmentations of real and virtual objects that may be unique to each layer.

[0013] In some embodiments, each distributed software engine may be a distributed 3D engine decomposed into multiple dynamic and independent software modules that may communicate with each other when configured locally or with remotely located individual software modules. In some embodiments, multiple distributed software engines may be used simultaneously or sequentially within a network to create synergistic effects with each other. For example, multiple 3D graphics engines hosted on the same or different server computers may allocate engine services sequentially or simultaneously via a distributed message exchange platform to complement each other's engine services for the realization of one or more specific tasks.

[0014] In some embodiments, the one or more processors are dedicated processing units, general-purpose processing units, or a combination thereof configured to process tasks from corresponding individual distributed software engines in hardware. In contrast to typical systems that may employ one or more standard central processing units (CPUs), graphics processing units (GPUs), and the like, in one embodiment, the dedicated hardware engines are each tailored to process an individual distributed software engine used in at least one of 3D graphics, artificial intelligence operations, physics operations, 3D structure processing operations, simulation operations, tracking and positioning, and communications. In some embodiments, each hardware engine may be combined as a chip or system on an electronic chip with one or more specific hardware processing cores designated to process each specific type of operation. The processing cores of the one or more electronic chips may be considered functional units or modules of the electronic circuitry of the electronic chip designed to physically execute the required algorithms of each 3D engine. For example, at least one of the one or more processing cores may be realized as an FPGA or any other form of integrated circuit implementing dedicated functions. Thus, the electronic chip includes at least one integrated circuit implementing at least some of the one or more processing cores, at least one of which implements a different dedicated hardware engine. These electronic chips may be located, for example, in one or more external servers or in the main server, or both. In some embodiments, the electronic chips may further comprise general-purpose processing cores, such as a CPU and / or GPU, or may be connected to an external CPU and / or GPU to provide additional support for each engine task. Providing a dedicated hardware engine for each of the sources and configuring each of the 3D engine modules in a distributed and independent manner can provide additional resources for the overall system, in addition to allowing each 3D engine module to have one or more dedicated hardware cores from one or more hardware engines.

[0015] In some embodiments, the data structure representing the virtual or real world into cells is an octree data structure, with each cell represented as a voxel within the octree data structure. In some embodiments, the voxels utilized within the octree data structure are sparse voxels for use in arranging larger portions of the real world. In other embodiments, the voxels utilized within the octree data structure are dense voxels for use in arranging smaller portions of the real world. Octrees and selected sparse or dense voxels are suitable data structures for representing three-dimensional space in virtual worlds, such as persistent virtual world systems that represent the real world. Octrees enable fast and easy combination of multiple objects, implement relatively simple rendering algorithms, enable rapid spatial searches, and enable model generation through virtualization of real objects (e.g., through LIDAR or other image scanning methods). For virtual worlds, including 3D virtual worlds that represent at least a portion of the real world, such 3D partitioning of the world enables three-dimensional-based engine services and cloud resource allocation over areas where users may be viewing and / or interacting.

[0016] In some embodiments, cells representing more resource-intensive areas of interest from at least a portion of the world are further divided into a larger number of cells. In further embodiments, each resource-intensive cell is allocated resources from one or more sources accordingly, with smaller cells being allocated more resources. In yet another embodiment, after completing an event associated with one or more requests, resources are returned from the multiple sources to the original one or more sources. At least a portion of the world is then consolidated into the original number of cells. In this manner, the system achieves efficient and dynamic allocation of resources by selecting areas of interest with higher demand.

[0017] In some embodiments, the resource manager performs the allocation through a distributed message exchange platform. In further embodiments, the distributed message exchange platform utilizes a publish / subscribe model, with one or more virtual objects subscribed to one or more cells in which resources are published. In further embodiments, the distributed message exchange platform shares the dynamically updated state of at least a portion of the world stored in memory with one or more of the client devices and the server. Thus, the resource manager of the distributed 3D engine of the present disclosure publishes each resource once to the distributed message exchange platform, which distributes the resource to each virtual object needing it by subscribing to cells having the published resource. This contrasts with typical 3D engine systems tailored to provision a virtual world, which output a complete data update packet for each client device in the map, but tailors each packet to the client device so that the client device receives only data that may be relevant to the client device and for objects in its field of view, thereby enabling the game or simulation engine to send the same data multiple times, once to each client device that can see the object.

[0018] According to one embodiment, the state of at least a portion of the world is further updated via a plurality of connected devices including sensors that provide sensor data to the persistent virtual world system. The connected devices may be, for example, Internet of Things (IoT) devices that may communicate with each other and with a server over a network and continuously capture multi-source sensor data from the real world to update the virtual replica of the persistent virtual world system. Thus, state updates may result from sensor data. However, state updates may also result from user input or from computations (e.g., artificial intelligence inference) by one or more 3D engines. The world state updates may then be published on a distributed message exchange platform for all client devices and sources to subscribe to and view on demand.

[0019] In some embodiments, each cell of the plurality of cells includes one or more streams, each containing one or more stream-specific virtual objects and configured to be enabled or disabled for viewing and interaction on a client device. In further embodiments, each stream is associated with one or more user applications. Each stream may be configured to obtain data from corresponding virtual objects and perform thematic and / or goal-specific simulations. Each stream may connect to a root node or root simulation object that generates multiple stream-related virtual objects. A stream may further connect to an import function that defines the virtual replicas on which the stream must perform the simulation. In some embodiments, multiple streams, including all or most of the virtual replicas in a given environment, such as a cell or multiple cells, their contexts, and the relationships and interactions between each virtual replica, form a universe or subuniverse. A network of subuniverses may form the universe of a persistent virtual world system. For example, a subuniverse of a city may be called a cityverse, which may represent all of the city's buildings, trees, people, streets, traffic lights, vehicles, etc., and their relationships, interactions, and contexts. Subuniverses may exist within other subuniverses, such as residential universes within a cityverse. A residential universe may contain a 3D design of the exterior and interior of the house, all the appliances in the house, people, walls, energy consumption data, etc. The network of all sub-universes may form a universe of a persistent virtual world system that can be viewed in virtual reality or augmented reality. Each of these sub-universes may be dynamically divided into multiple cells where resources are dynamically allocated.

[0020] In embodiments of the present disclosure, the system may be implemented within a cloud-to-edge infrastructure that may represent distributed computing capabilities, including the adoption of cloudlets / edge systems such as public or private clouds, fog servers, peer-to-peer cloud server systems, distributed ledger-based infrastructures, enterprise systems, mobile platforms, and user devices. The distributed ledger-based infrastructure may be a decentralized, immutable blockchain or distributed ledger network that facilitates the transfer and storage of data necessary to widely distribute a persistent virtual world system, including virtual replicas, purely virtual objects, applications, and any events or interactions within the persistent virtual world system. Resources, including physical servers and network equipment, enable shared storage and computing that can be dynamically allocated through the cloud-to-edge infrastructure depending on factors such as users' proximity to the resources and network, and their computational demands on each other and their location, directing them to compute more or less depending on the situation. In some embodiments, the dynamic allocation of resources is codified into smart contracts using the distributed ledger infrastructure, automating the transaction logic (e.g., the computer logic behind each allocation of storage and computing) and the conclusion or outcome of such allocation.

[0021] According to one embodiment, a method for enabling a distributed 3D engine to perform dynamic load balancing across a virtual world includes virtualizing, in the memory of at least one server computer, a data structure representing at least a portion of the virtual or real world into a plurality of cells that store virtual objects forming a persistent virtual world system, at least some of the virtual objects being virtual replicas of corresponding real-world elements. The method then provides, in the memory of at least one server computer, a distributed 3D engine implemented in a distributed deployment, the distributed 3D engine comprising a resource manager and a plurality of individual distributed software engines. The method then continues by calculating, by the resource manager, demands for individual cells of the plurality of cells based on current loads, the demands being based, at least in part, on the amount of virtual objects in the individual cells or the level of interaction within the portion of the persistent virtual world system visible to user avatars in the individual cells. The method then continues by ranking the individual cells according to their demands. Finally, the method concludes by dynamically allocating resources to individual cells (e.g., one or more more resource-intensive cells) based on the ranked demands.

[0022] According to one embodiment, the method further includes dividing an area of ​​interest of at least a portion of the virtual world or real world into additional cells based on the calculated demand, and allocating corresponding resources to the additional cells (e.g., each cell representing a resource-intensive area of ​​interest). In one embodiment, the method further includes returning the resources to the original one or more sources after completing an event associated with one or more demands in the more resource-intensive area of ​​interest, and consolidating at least a portion of the world in the persistent virtual world system back into the original number of cells.

[0023] In some embodiments, the allocation is performed by publishing resources to corresponding cells via a distributed message exchange platform of the resource manager using a publish / subscribe model, and by one or more virtual objects subscribing to cells of interest to obtain the required resources.

[0024] In some embodiments, the method further includes sharing, via a distributed message exchange platform, a dynamically updated state of at least a portion of the world to which one or more client devices and / or server computers subscribe to obtain an updated version of the world. In still further embodiments, the state of at least a portion of the world is obtained by a plurality of connected devices including sensors that provide sensor data to the persistent virtual world system, by user input, by server computation, or a combination thereof.

[0025] The above summary does not contain an exhaustive list of all aspects of the present disclosure. The present disclosure is intended to include all systems and methods that may be implemented from any suitable combination of the various aspects summarized above, as well as those systems and methods disclosed in the following Detailed Description and particularly pointed out in the claims filed herewith. Such combinations have certain advantages not specifically recited in the above summary. Other features and advantages of the present disclosure will be apparent from the accompanying drawings and the following detailed description.

[0026] Certain features, aspects, and advantages of the present disclosure will become better understood with reference to the following description and accompanying drawings. [Brief explanation of the drawings]

[0027] [Figure 1]1 is a schematic diagram of a system that enables multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of virtual or real worlds, according to one embodiment. [Figure 2] 1 is a schematic diagram of a server of a system enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of a virtual or real world according to one embodiment. [Figure 3] FIG. 1 is a schematic diagram of resource allocation via a distributed message exchange platform according to one embodiment. [Figure 4A-4B] FIG. 1 is a schematic diagram of engine service allocation via a distributed message exchange platform, according to one embodiment. [Figures 5A-5C] FIG. 2 is a schematic diagram of the relationships between elements of a persistent virtual world system, according to one embodiment. [Figures 6A-6B] FIG. 2 is a schematic diagram of division into multiple cells according to one embodiment. [Figure 7] FIG. 10 is a schematic diagram of another embodiment of division into multiple cells, according to one embodiment. [Figure 8] 1 is a schematic diagram of a client device according to one embodiment. [Figure 9] FIG. 1 is a block diagram of a computer-implemented method for enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of a virtual or real world, according to one embodiment. [Figure 10] FIG. 10 is a block diagram including further steps of a computer-implemented method for enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of virtual or real worlds, according to one embodiment. [Figure 11] FIG. 1 is a block diagram illustrating resource allocation steps via a distributed message exchange platform according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0028] In the following description, reference is made to drawings that illustrate various embodiments. The various embodiments are also described below with reference to several examples. It should be understood that the embodiments may include changes in design and construction without departing from the scope of the claimed subject matter.

[0029] FIG. 1 is a schematic diagram of a system 100 that enables multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of virtual or real worlds, according to one embodiment.

[0030] The system 100 includes multiple server computers 102 (e.g., external servers A-C) that implement a distributed 3D engine including a resource manager in a distributed deployment 106, connected via a network 104. The distributed deployment 106 refers to a software implementation of the distributed 3D engine on the multiple server computers 102 using a distributed architecture such as the architecture of the system 100. The server computers 102 dynamically allocate resources to one or more of multiple cells 108 (e.g., cells 1-3) via the resource manager using a publish / subscribe model. Each cell 108 virtually represents one or more different areas of the real or virtual world. The allocation is based on the current load and the corresponding calculated and ranked demand. In some embodiments, the demand further considers one or more of the quantity of virtual objects 110 and the level of interaction within the portion of the persistent virtual world system visible to user avatars, as described in more detail below.

[0031] One or more of the plurality of cells 108 include virtual objects 110, and a plurality of those virtual objects 110 within a cell 108 form a persistent virtual world system 112. In some embodiments, at least some of the virtual objects 110 are virtual replicas of corresponding real-world elements. In some embodiments, the demand for an individual cell takes into account the amount of virtual objects 110 within the individual cell, the level of interaction within the portion of the persistent virtual world system visible to the user via the respective client device 114, or a combination thereof, as described in further detail below. In further embodiments, the demand takes into account other parameters, such as the distance between the client device 114 and network equipment such as a server and / or antenna, the type of application being used by the user, or the type of entitlement the user may have depending on their individual user profile or contract, as described in further detail below.

[0032] In this disclosure, the term "persistent" is used to characterize a system state that can continue to exist without a continuously running process or network connection. For example, the term "persistent" may be used to characterize a virtual world system in which the virtual world system and all of the objects contained therein continue to exist independent of a user connected to the virtual world system after the process used to create the virtual objects has terminated. Thus, the virtual world system is stored in a non-volatile storage location (e.g., a server). In this manner, virtual replicas, pure virtual objects, and applications can interact and collaborate with each other when configured to achieve specific goals, even when the user is not connected to the server.

[0033] In some embodiments, to reduce hardware and network demands, contribute to reduced network latency, and improve the general mixed reality experience, the system can connect via a network 104 including millimeter wave (mmW) or a combination of mmW and sub-6GHz communication systems, e.g., via fifth generation wireless system communications (5G). In other embodiments, the system can connect via a wireless local area network (Wi-Fi). The provided communication system can typically enable low (e.g., about 1 to about 5 milliseconds) end-to-end (E2E) latency and high (e.g., 1 to 10 Gbps) downlink speeds to endpoints in the field, depending on the parameters required to run highly interactive digital reality applications or other high-demand applications. The result is high-quality, low-latency, real-time digital application content streaming. In other embodiments, the system can communicatively connect via fourth generation wireless system communications (4G), which may be supported by a 4G communication system, or may include other wired or wireless communication systems.

[0034] In other embodiments, a global navigation satellite system (GNSS), which refers to satellite navigation systems such as GPS, BDS, Glonass, QZSS, Galileo, and IRNSS, may be used to enable device positioning. Using signals from a sufficient number of satellites and techniques such as triangulation and trilateration, the GNSS can calculate the device's position, velocity, altitude, and time. In one embodiment, the external positioning system is extended by an assisted GNSS (AGNSS) through the architecture of an existing cellular communication network, where the existing architecture includes 5G. In other embodiments, the AGNSS tracking system is further supported by a 4G cellular communication network. In indoor embodiments, the GNSS is further extended through a wireless local area network, such as Wi-Fi, preferably providing data at 60 GHz, but not limited to Wi-Fi. In alternative embodiments, the GNSS is extended through other technologies known in the art, such as differential GPS (DGPS), a geostationary satellite-based augmentation system (SBAS), or a real-time kinematic (RTK) system. In some embodiments, tracking of the device is implemented by a combination of AGNSS and inertial sensors within the device.

[0035] In some embodiments of the present disclosure, system 100 may be implemented within a cloud-to-edge infrastructure that may represent a distributed computing capacity, employing computing devices and systems such as public or private clouds, fog servers, cloudlets, peer-to-peer cloud server systems, distributed ledger-based infrastructures, and edge devices / systems such as enterprise systems, mobile platforms, and user devices (all of which may connect via network 104). Using a cloud-to-edge computing network, access to computing power, computer infrastructure (e.g., via so-called infrastructure as a service, or IaaS), applications, and business processes may be delivered on demand as a service to users via client devices 114. In this manner, resources, including physical servers and network equipment, enable shared storage and computing that may be dynamically allocated. In some embodiments, client devices 114 include user devices and other devices. User devices may comprise, for example, mobile devices, personal computers, game consoles, media centers, head-mounted displays, and see-through devices (e.g., digital reality contact lenses). The other devices may be configured to, among other things, capture multi-source sensor data from real-world objects via sensor mechanisms to update and enrich the virtual replicas of the corresponding real-world objects. According to one embodiment, the other devices are one or more surveillance cameras, vehicles, traffic lights, buildings, roads, railroad tracks, household appliances, etc.

[0036] FIG. 2 is a schematic diagram of a server 102 of a system 100 that enables multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of virtual or real worlds, according to one embodiment.

[0037] The one or more servers 102 comprise one or more processors 202 and a memory 204 that stores data structures that virtualize at least a portion of a virtual or real world as a plurality of cells (e.g., cells 108 in FIG. 1 ), each containing a virtual object 110, forming a persistent virtual world system 112. In an embodiment in which the persistent virtual world system 112 is based on the real world, some of the virtual objects 110 are virtual replicas 206 of corresponding real-world elements. In other embodiments, the virtual objects 110 further include applications 208 and purely virtual objects 210 that do not exist in the real world. The memory 204 also stores a distributed 3D engine 212 that includes a resource manager 214 that manages resources. The resource manager 214 utilizes a distributed message exchange platform 216 that uses a publish / subscribe model to allocate resources, including computing power, memory, and network resources, by publishing the resources on one or more cells to which virtual objects can subscribe. The resource manager 214 may function as a virtual layer that interfaces the hardware and software components that manage the resources. Management actions may include assessing, provisioning, scaling, and replicating resources via the distributed message exchange platform 216 .

[0038] In this disclosure, the term "virtual replica" refers to an accurate, persistent virtual representation of a real-world element. In one embodiment, the virtual replica 206 of this disclosure refers to a virtual version or virtual twin of a real-world element or real-world twin, which may be mapped or modeled by computer-aided drawing (CAD) or computer-aided engineering (CAE) methods, input from a computer model of the real object (e.g., a Building Information Model (BIM)), or other suitable prior art methods, and may reflect not only the appearance but also the behavior of the real-world twin. Some of the real-world objects may further include sensors that can provide multi-source input data to the virtual replica 206 to enrich and synchronize the virtual replica 206 with their respective real-world counterparts.

[0039] In some embodiments, the virtual replica 206 includes one or more of 3D world / building data such as SLAM or differential mapping-based data, 3D geometry data, 3D point cloud data, or geographic information system data representing real-world structural characteristics that may be useful for modeling 3D structures for applications.

[0040] In this disclosure, the term "virtualization" refers to the act of creating a virtual world or an abstracted virtual version of at least a portion of the real world, or a persistent virtual world system 112. In the case of the real world, virtualization refers to creating virtual versions of elements found in the real world, as well as virtual versions of any resources (e.g., computing power, memory, or networks) and related equipment of the real world, and allowing a user (e.g., a human user, a computer program, or a machine) to interact with the real-world elements through the persistent virtual world system 112. In this way, the user can feel as if the interaction took place directly with the real-world elements. For example, a human user can interact with elements of a real city through a virtual city in augmented reality or virtual reality through the user interface of a user device (e.g., a mobile phone or a head-mounted display device), where multiple elements of the real city are virtualized within the persistent virtual world system 112. For physical manipulation of the real world through the virtual replica, electromechanical devices can be installed on the real-world objects to translate remote commands sent to the real-world objects through the virtual replica into physical actions from the real-world elements. Regardless of whether the persistent virtual world system 112 is based in a virtual world or a real world, virtual objects may further include models, which refer to any graphical, mathematical, or logical representation of aspects of the virtual object or a corresponding real object. In some embodiments, such models provide entities of the persistent virtual world system 112 with self-computing capabilities and autonomous behavior. In some embodiments, suitable models include one or more of a 3D model, a geometric model, a dynamic model, and a machine learning model.

[0041] The distributed 3D engine 212 of this disclosure refers to a distributed version of a 3D engine implemented in a distributed deployment (e.g., multiple server computers or client computers). The distributed 3D engine 212 is configured to virtually pool, allocate, provision, dynamically load balance, and allocate resources (e.g., computing power, memory, and network) via a resource manager 214. The distributed 3D engine's 212 connection and access to the persistent virtual world system 112 allows load balancing to be performed accurately by using spatial computing principles to assess demand in each of the cells representing at least a portion of the real or virtual world. As part of resource provisioning, the distributed 3D engine 212 provides multiple engine services (e.g., 3D rendering, graphics, physics, audio, artificial intelligence, positioning, and communications) on demand via multiple individual distributed software engines. The hardware operations performed to provide each of the above-mentioned engine services may be provided by multiple corresponding hardware engines. This resource pooling is implemented through virtualization, as described above. Thus, the concept of a distributed 3D engine 212 in this disclosure should not be interpreted simply as a collection of abstracted functions in a library that can be reused across different applications (e.g., video games), but rather as a distributed hardware and software implementation encompassing engine operations that are pooled, allocated, provisioned, and load-balanced via a resource manager 214 using spatial computing principles that employ a persistent virtual world system as the basis for said operations.

[0042] In this disclosure, the term "spatial computing" refers to the use of physical space to send input and receive output from a computer. Thus, spatial computing uses the geometric properties of virtual objects and their positions and orientations to perform required computing and dynamic load balancing.

[0043] The virtual objects 110 may be supplied with data obtained from a simulation, data input by a user through a user device, or data captured by sensor devices such as Internet of Things devices (e.g., in the case of a persistent virtual world system based on the real world). A continuous flow of data can keep the persistent virtual world system 112 periodically updated. Thus, a persistent virtual world system 112 from at least a portion of the real world is not only a visual 3D representation of the real world, as is the case with many current virtual world systems, but also a multidimensional representation of the real world, as the virtual objects 110 include models and data from other dimensions or aspects of the real world. For example, multiple virtual objects 110 may include data related to temperature, fuel consumption, speed, power, pressure, mass, sound, etc., also enabling connections and communication between each virtual object.

[0044] In some embodiments, virtualization of a virtual or real world involves running multiple simulations of the same world at various levels of detail (LOD). LOD management can associate different physics models with virtual replicas, from low- to high-fidelity models, thereby increasing the efficiency of computing processes such as rendering processes by reducing the workload of graphics pipeline usage, typically vertex transformations, or by enhancing physics simulations. This allows for different simulations to be performed depending on the case or situation. LOD management also enables providing an optimized user experience depending on specific requirements and context. Thus, the same virtual or real world can be simulated at different LODs to facilitate the load balancing methods of the present disclosure while providing an appropriate user experience. For example, the real-world simulation in the persistent virtual world system 112 may include a high-fidelity simulation of at least a portion of the real world intended for visualization by a human user and a low-fidelity simulation to assess the demand for each cell and facilitate load balancing of the high-fidelity simulation. Thus, the low-fidelity simulation may utilize fewer resources while making it easier to view which areas are overloaded and which are not. In another example, virtualization of a manufacturing plant may enable accurate management of the manufacturing plant, including demand assessment and corresponding resource allocation via the distributed 3D engine 212, with accurate spatial representations and inter-object relationships, as well as data and models related to the current energy consumption, production rate, efficiency, etc., of each machine in the manufacturing plant. The manufacturing plant may be simulated at high fidelity for management of the manufacturing plant (e.g., control of the manufacturing process and all elements within the plant) and at low fidelity for managing load balancing of the manufacturing plant. The same LOD principles described above can be applied to a persistent virtual world system 112 based on a virtual world, where a low-fidelity version of the virtual world may be simulated within the persistent virtual world system 112 for low load balancing of the virtual world, and a high-fidelity version of the virtual world may be simulated to provide a better user experience.

[0045] "Self-computing capability," also referred to as "self-management capability," herein refers to the ability to apply artificial intelligence algorithms to autonomously manage computing resources. In one embodiment, a virtual replica with self-computing capability in a persistent virtual world system can autonomously manage computing resources to adapt to changes in the environment of the corresponding real-world element or to changes in the real-world element itself. Self-management rules and conditions may be further managed through the use of smart contracts implemented on blockchain or distributed ledger technology to further codify the rules and conditions in a decentralized and transparent manner. Thus, in one embodiment, each virtual object, such as a virtual replica of a real object, may exhibit autonomous behavior, operating autonomously in response to real-world conditions reflected in the persistent virtual world system (e.g., by allocating required resources, autonomously sending and executing commands, and / or generating events as required by the respective situation). Achieving this type of behavior may require training the virtual object with artificial intelligence algorithms during modeling of the virtual object.

[0046] A virtual replica may obtain data from one or more sources (e.g., one or more real-world objects, environmental sensors, computing devices, etc.). As used herein, the term "multi-source data" refers to data that may be obtained from multiple sources. A virtual replica may be updated based on the multi-source data, for example, by updating known parameters or features, enriching the virtual replica with additional parameters or features, etc.

[0047] In this disclosure, the term "enriching" is used to describe the act of providing a virtual replica with additional characteristics based on multi-source data. For example, enriching a virtual replica may refer to providing real-world data captured from sensor mechanisms on a client device, where the additional real-world data includes video data, temperature data, real-time energy consumption data, real-time water consumption data, speed or acceleration data, etc.

[0048] In some embodiments, some of the virtual replicas 206 may be virtual avatars of real users who access the persistent virtual world system 112 through their client devices. In these embodiments, resources are published on the corresponding cells in which the user's virtual avatar is active, corresponding to the position and orientation the user can view through their client device, and the resources are then received by the user through the user's client device and the corresponding virtual avatar subscribing to the corresponding cell. Similarly, when applications 208 and / or pure virtual objects 210 are active in a cell, these virtual objects 110 subscribe to that cell and trigger the resource manager 214 to evaluate demand and, accordingly, publish more or less resources that the virtual objects 110 can obtain by subscribing to the corresponding cell.

[0049] For example, a particular traffic light in a city may be linked to a road or street and also associated with other objects, such as sensors or other traffic lights. These relationships can trigger events that can generate new interactions within the persistent virtual world system 112; for example, an emergency vehicle in a street may automatically trigger a priority event that changes traffic light patterns to avoid stops at intersections based on a simulation of the emergency vehicle's trajectory and the actual traffic conditions in the area. Management of resources during such events, including provisioning of devices (e.g., automobiles) involved in such events, is performed by the resource manager 212. In some embodiments, such conditions, events, and outcomes are programmed as smart contracts bound to a traffic-specific distributed ledger, e.g., a traffic sidechain with many smart contracts defining various codified traffic rules and expected behaviors, thus automating and recording the occurrence of such events.

[0050] In some embodiments, a purely virtual object 210 refers to an object that exists only within the persistent virtual world system 112 and therefore does not have a corresponding object in the real world.

[0051] In some embodiments, application 208 may be one or more traditional applications, distributed applications, or decentralized applications. Traditional applications are typically based on a traditional client / server model and run on dedicated servers in a static infrastructure. Decentralized applications are primarily applications stored on a cloud computing platform, such as the cloud servers of the present disclosure, and may run simultaneously on multiple systems and devices on the same network, or may run on a blockchain- or distributed ledger-based distributed database. Decentralized applications primarily run on a decentralized infrastructure, such as a blockchain- or distributed ledger-based distributed database.

[0052] In some embodiments, the persistent virtual world system 112 may comprise virtual world layers that may be separated into an augmented reality (AR) layer and a virtual reality (VR) layer. The separate layers may enable accessing the persistent virtual world system in either augmented reality or virtual reality and combining elements stored in each layer, which may be activated via a client device connected to at least one server computer 102 via a network whenever accessing one or the other type of reality. Each of the layers may include augmentations of real and virtual objects 110 that may be unique to each layer. For example, the AR layer may include only virtual replicas 206, applications 208, and purely virtual objects 210 that may only be visible when accessing the persistent virtual world system 112 in augmented reality. Similarly, the VR layer may include only virtual replicas 206, applications 208, and purely virtual objects 210 that may only be visible when accessing the persistent virtual world system 112 in virtual reality. However, the user may decide to activate both layers or adjust the level of each layer so that at least some objects of one layer are visible in the other layer. For example, when accessing a virtual reality scene, it may be possible to activate an augmented reality layer that allows viewing an extension of the real objects virtualized in the VR layer, such as seeing additional descriptions of buildings or tourist attractions.

[0053] In some embodiments, the distributed 3D engine 212 further comprises one or more individual distributed software engines 218. The distributed software engine 218 may be a 3D engine decomposed into multiple dynamic and independent software modules, each programmed to perform specific engine tasks. The individual distributed software engines 218 may communicate with each other, for example, via a distributed message exchange platform 216. Each individual distributed 3D engine 218 can use the message exchange platform 216 to invoke service functions within other individual distributed software engines 218. Such calls may be performed indirectly (e.g., through a generic proxy function invoked by the distributed message exchange platform 216) or directly (e.g., by the distributed message exchange platform 216 maintaining direct communication with each individual distributed 3D engine 218).

[0054] In some embodiments, the one or more processors 202 may be one or more dedicated hardware engines 220. In other embodiments, the one or more processors 202 are general-purpose processing units or dedicated processing units supported by a general-purpose processing unit. In some embodiments, the dedicated processing unit is a processing unit tailored to the processing of one or more individual distributed software engines 218 to perform engine tasks. Thus, the dedicated hardware engines 220 may be individually designed and configured to process operations related to one or more of 3D graphics, artificial intelligence operations, physics operations, 3D structure processing operations, simulation operations, tracking and positioning, or communication thereof. In some embodiments, each hardware engine 220 may be coupled to one or more electronic chips or systems-on-chips with one or more specific hardware processing cores designated to process each specific type of operation. The processing cores of the one or more electronic chips may be considered functional units or modules of the electronic circuitry of the electronic chip designed to physically execute the required algorithms of each individual distributed software engine 218. For example, at least one of the one or more processing cores may be realized as an FPGA or any other form of integrated circuitry implementing dedicated functions. Thus, one or more electronic chips include at least one integrated circuit that realizes at least some of the one or more processing cores, at least one of which implements a different specialized hardware engine 220. In some embodiments, the electronic chip may further comprise general-purpose processing cores, such as a CPU and / or GPU, or may connect to an external CPU and / or GPU to provide additional support for the respective engine tasks.By providing dedicated hardware engines 220, each individual distributed 3D engine 218 can obtain resources from one or more dedicated hardware cores from one or more hardware engines 220, allowing each individual distributed software engine 218 to pull more resources to perform a greater number of tasks within the persistent virtual world system.

[0055] Processor 202 may refer to a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. Furthermore, the explicit use of the term "processor" should not be construed to refer only to hardware capable of executing software, but may implicitly include, but is not limited to, digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), FPGAs, etc.

[0056] Memory 204 may be any suitable type of memory capable of storing information accessible by processor 202, including computer-readable media or other media that store data readable with the aid of electronic devices, such as hard drives, memory cards, flash drives, ROM, RAM, DVDs or other optical disks, and other writable and read-only memories. Memory may include temporary storage as well as persistent storage. Instructions may be executed directly (e.g., machine code) or indirectly (e.g., scripts) by processor 202. Instructions may be stored in object code format for direct processing by processor 202, or in any other computer language, including scripts or collections of independent source code modules that can be interpreted on demand or pre-compiled. Data may be retrieved, stored, or modified by processor 202 according to the instructions. Data may be stored in computer registers in a relational database, for example, as a table with multiple distinct fields and records, an XML document, or a flat file. Data may also be formatted in any computer-readable format.

[0057] In some embodiments, interactions with and between virtual objects 110 may be defined through the use of computer code contained in computer scripts and computer programs and enabled by applications, smart contracts, or a combination thereof provisioned within persistent virtual world system 112 where the interactions and interaction applications take place. Thus, the applications, smart contracts, or a combination thereof allow for digitally codifying and representing all possible desired interactions and interaction interfaces between all elements of the real world via persistent virtual world system 112. Interactions may be experienced by users in the form of interactive game-like applications or through game-like interactions.

[0058] FIG. 3 is a schematic diagram of a system 300 for allocating resources via a distributed message exchange platform 216 of a resource manager, according to one embodiment.

[0059] The resource allocation system 300 allocates resources 302 to the distributed message exchange platform 216, where the resources 302 include computing power 304, memory 306, and a network 308, and the resources 302 may be obtained from one or more server computers 102 and / or client devices 114. The distributed message exchange platform 216 receives requests from one or more virtual objects 110 residing in the cells 108 and performs the corresponding allocations by publishing to the cells 108. In some embodiments, to perform these resource allocations, the distributed message exchange platform 216 utilizes a publish / subscribe model, whereby one or more virtual objects 110 subscribe to one or more cells 108 to which the resources 302 are published. For example, a virtual object representing a user, or user virtual avatar, may be active in a particular cell 108, triggering a resource manager to publish resources needed by the user virtual avatar to the corresponding cell. The publishing is performed via the distributed message exchange platform 216. Thus, resources 302 are published once to the distributed message exchange platform 216 and distributed to each virtual object 110 that subscribes 310 to the resources 302. This contrasts with typical 3D engine systems developed to provision virtual worlds, which output a complete data update packet for each client device in the map, but tailors each packet to the client device so that the client device receives only the data that may be relevant to the client device and for objects in its field of view, thereby allowing the 3D engine to send the same data multiple times, once to each client device that can see the object.

[0060] The publish / subscribe model is a model of communication based on content production and consumption, which differs from typical client / server-based communication models in which communication typically occurs between two nodes (i.e., a client and a server). Communication via such a publish / subscribe model can be anonymous, meaning that the server computer 102 and / or client device 114 do not need to know who has subscribed to which cell 108 to obtain resources 302, and similarly, the virtual object 110 obtaining the resource does not need to know the source of the resource 302. A typical publish / subscribe model enables information exchange in which anyone can publish and receive information, allowing for greater flexibility than a typical client / server model. However, the use of a publish / subscribe model can also raise inefficiency concerns because every publication is typically conveyed to every participant, who must analyze the information to determine its relevance. In contrast, the publish / subscribe model of the present disclosure allows for a dynamic concentration of resources to the cells 108 that need them most (e.g., higher resource-demanding cells 108 to which virtual objects 110 can subscribe). Also, the amount of resources published takes into account the amount of virtual objects 110, the level of interaction within the portion of the persistent virtual world system visible to the user, and other parameters, so that each virtual object 110 obtains only the resources it needs from each subscribed cell during its respective simulation.

[0061] In some embodiments, the computing power 304 of the resource 302 includes engine services 314 obtained from one or more individual distributed 3D engines (e.g., individual distributed software engines 218 of FIG. 2 ). In further embodiments, the memory 306 of the resource 302 stores a world state that is further published to a corresponding cell 108 to provide an updated version of the real world through the virtual world to client devices subscribed to the cell 108. In further embodiments, the distributed message exchange platform 308 is configured as part of a network resource that is optionally shared. In some embodiments, multiple distributed software engines 218 can be used simultaneously or sequentially within the network 308 to create synergistic effects with each other.

[0062] As used herein, the term "engine services" refers to operations performed and provided by one or more 3D engines to accomplish one or more tasks, such as games or other application-related tasks. 3D engines typically include software components, such as libraries, software development kits (SDKs), objects, etc., required to perform such tasks. Engine services include, for example, services related to 3D rendering, graphics, physics, audio, artificial intelligence, positioning, and communications. Each of the engine services may be performed by an individual distributed software engine provisioned by a hardware engine, which may be a dedicated hardware engine, a general-purpose hardware engine, or a combination thereof, with each hardware engine comprising dedicated circuitry for executing instructions from functions called by the individual distributed 3D engine.

[0063] The term "world state," as used herein, refers to the state of each of the virtual objects in a persistent virtual world system and may be updated in time through multiple connected devices, including sensors, that provide sensor data to the persistent virtual world system. The connected devices may be, for example, Internet of Things (IoT) devices that may communicate with each other and with a server over a network and continuously capture multi-source data from the real world to update the virtual replica of the persistent virtual world system. Thus, state updates may result from sensor data. However, world state updates may also result from user input or from computations (e.g., artificial intelligence inference) from one or more 3D engines.

[0064] In some embodiments, allocation of virtual objects 110 by publishing to cells 108 may be based on current load and corresponding calculated and ranked demand. Referring to FIG. 2 , such demand calculation and ranking may be performed, for example, by the resource manager 214 of the distributed 3D engine 212. The ranking may assist in determining the amount of resources 302 to publish to each cell 108, such that cells with lower demand ranks may obtain fewer resources 302 than cells with higher demand ranks. In some embodiments, demand considers the amount of virtual objects 110 within the portion of the persistent virtual world system visible to a user via a corresponding client device in the cell. The portion of the persistent virtual world system visible to a user avatar may be defined by the avatar's field of view or field of view within the cell. For example, when a user views a portion of a world in augmented reality or virtual reality, a corresponding avatar may be available in the persistent virtual world system, and thus the user avatar's view frustum may define such field of view. In other embodiments, when a user is using an avatar through a "flat" user interface, such as a computer or mobile phone screen, the actual user avatar may be visible within the computer screen, and the user avatar may view and interact within a portion of the persistent virtual world system. In such an example, the user avatar may view the persistent virtual world system from a first-person perspective, and the first-person avatar's viewing frustum may define the field of view. In embodiments where the user avatar is viewed from above, the field of view may be defined by a radius centered on the user avatar. In all cases, the field of view may include multiple virtual objects 110 with which the user may have the potential to interact via the user avatar. Thus, the more virtual objects within the user avatar's field of view, the higher the resource demand for that particular cell, and therefore the higher the ranking demand score. In other embodiments, demand also takes into account the level of interaction within the portion of the persistent virtual world system visible to the user.For example, two user avatars may exist in two different regions and two different cells of a persistent virtual world system, both with five virtual objects 110 available for interaction. If a first user decides to simultaneously interact with all virtual objects 110 (e.g., by participating in a group activity, conversation, battle, etc.), and a second user avatar does not interact with any of the virtual objects 110 or has a lower level of interaction with such virtual objects 110, such cell may have a higher ranked demand and therefore be allocated more resources than the second user avatar. In this example, the ranked demand is higher for the virtual cell in which the first user avatar is located than for the virtual cell in which the second user avatar is located.

[0065] In other embodiments, demand further factors in other parameters, such as the distance between the client device and the network equipment (e.g., the server and / or antenna), the type of application being used by the user, or the type of entitlement the user may have depending on their individual user profile or contract. As an example of the distance between the client device and the network equipment, since each cell is assigned a specific network equipment, the farther a user is from the network equipment in a particular cell of the persistent virtual world system, the more network equipment may be simultaneously activated to compensate for a potentially lower quality of service (QoS), resulting in a higher demand ranking for that cell. As an example of the application being used by the user, if the application consumes more resources, such as bandwidth or computing power, because it requires a higher real-time rendering rate or a better level of detail and clarity, this indicates an increase in the resources required for that cell, and thus a correspondingly higher ranked demand. As an example of the type of entitlement, a user may be registered as a premium account user to obtain a higher quality of service (QoS) and a higher data rate than other users, resulting in a higher resource allocation than other users. Additionally, all of the above factors can be calculated in combination with each other to determine the ranked demand for each cell.

[0066] 4A and 4B are schematic diagrams 400a and 400b of engine service allocation via a distributed message exchange platform 216 according to one embodiment.

[0067] The schematic diagram 400a of Figure 4A includes multiple individual distributed software engines 218 hosted on one or more servers, such as server computer 102 of Figure 1. Each individual distributed software engine 218 provides a particular engine service that can be published to cells 108 via a distributed message exchange platform 216. Virtual objects 110, such as virtual objects A-D, can then obtain their engine services by subscribing to the cells 108, such as cells 1-2, in which they are located.

[0068] 4A , each distributed software engine 218 may be a plurality of dynamic, independent software modules designed for one or more specific types of tasks. Each distributed software engine 218 is an independent entity that provides the functionality for which it is programmed. Thus, each distributed software engine 218 may be, for example, an artificial intelligence engine 402 a, a 3D graphics engine 402 b, a simulation engine 402 c, a communications engine 402 d, an audio engine 402 e, a 3D structure engine 402 f, a position engine 402 g, and a rendering engine 402 h. Communication between each distributed software engine 218 is implemented via function calls managed by the distributed message exchange platform 216.

[0069] In some embodiments, multiple distributed software engines 218 can be used simultaneously or sequentially within a network to create synergistic effects. For example, multiple 3D graphics engines 402b hosted on the same or different server computers can allocate engine services sequentially or simultaneously via the distributed message exchange platform 216 to complement each other's engine services for the realization of one or more specific tasks. In one example, different 3D graphics engines 402b calculate 3D models of different locations, where the locations may be within the same or different cells 108. In another example, different 3D graphics engines 402b can calculate different applications located in the same area, with at least some of the applications configured in an AR layer and at least some of the applications configured in a VR layer. As an example, the same location in the same cell is calculated by Unity® in the VR layer, while any extensions, such as pure virtual objects and applications, configured in the AR layer are partially calculated by CryEngine® and UnrealEngine®. Furthermore, in this example, adjacent locations within the same or different cells 108 may be calculated in the VR layer by CryEngine®, while the augmentation in the AR layer is partially calculated by UnrealEngine® and Unity®.

[0070] 4A , virtual objects A-B are currently located within cell 1 of the persistent virtual world system and are receiving multiple engine services, such as AI service 404 provisioned by AI engine 402a, rendering service 406 provisioned by rendering engine 402h, and simulation service 408 provisioned by simulation engine 402c, which may be necessary for the processing and interaction of the current application and virtual objects within current cell 1. Meanwhile, virtual objects C-D are currently located within cell 2 of the persistent virtual world system and are receiving multiple engine services, such as audio service 410 provisioned by audio engine 402e, communication service 412 provisioned by communication engine 402d, and simulation service 408 provisioned by simulation engine 402c, which may be necessary for the processing and interaction of the current application and virtual objects within current cell 2. These engine services may be provisioned by the same server computer hosting the individual distributed software engines 218, by multiple server computers hosting the individual distributed software engines 218, or by a combination thereof.

[0071] FIG. 4B is a schematic diagram 400b illustrating the relationships between individual distributed software engines 218, hardware engines 220, and distributed 3D engines 212 of one or more servers 102. Upon receiving a request from one or more virtual objects located within a cell 108, a resource manager 214 hosted on one or more server computers 102 receives the request, evaluates the demand, and requests engine services from one or more server computers 102 hosting the software engines and hardware engines 218-220, respectively, via a distributed message exchange platform 216. Each of the engine services may be provided by an individual distributed software engine 218, such as 3D Engines 1-N, provisioned by a respective hardware engine 220, such as Hardware Engines 1-N, which may be a dedicated hardware engine, a general-purpose hardware engine, or a combination thereof. The distributed message exchange platform 216 then receives the respective engine services and publishes them to one or more cells 108 that require them. The service-requesting virtual object then subscribes to the respective cell via the distributed message exchange platform 216 to receive the required engine services.

[0072] 5A-5C are schematic diagrams 500a-500c of the relationships between sub-universes (also referred to herein as "verses" for short), streams, and other elements of a persistent virtual world system, according to one embodiment.

[0073] Referring to the schematic diagram 500a in FIG. 5A , multiple connected virtual objects 110 that can be used for a target-specific simulation can constitute a stream 502, with each stream 502 configured to obtain data from a corresponding virtual object 110 and perform a target-specific simulation. Each stream 502 can connect to a root node 504 or root simulation object, which generates multiple stream-related virtual objects 110. The stream 502 can further connect to an import function 506 that defines the virtual objects 110 the stream 502 needs to perform the required simulation. For example, there can be a traffic stream 502 that includes multiple virtual replicas of vehicles, people, traffic lights, and even weather conditions. In other words, the stream 502 can include any number of virtual objects 110, such as virtual replicas, that can be related to specific features of the stream 502. In this example, the import function 506 can define which virtual objects 110 should be considered for the traffic simulation. The stream 502 can then read the behavior of each of the corresponding virtual objects 110 by performing a target-specific simulation of a traffic scenario using a combination of virtual and / or real sensors. For example, a goal may be to optimize traffic streams, perform at a predetermined average speed, or reduce the likelihood of traffic jams. Such simulations may be used, for example, to operate autonomous vehicles or drones. Another example of stream 502 may be energy consumption, which may include a virtual replica of a building including power consumption data and specific energy-consuming virtual replicas of machines, computers, appliances, etc.

[0074] 5A , multiple streams 502 containing all or most of the virtual objects 110 in a given environment form sub-universes referred to herein as berths 508. For example, a city berth 508 may represent all or most of a city's buildings, trees, people, streets, traffic lights, vehicles, etc. In some embodiments, there may also be berths 508 within berths 508, such as residential berths within a city berth, which may include 3D designs of the exterior and interior of a home, and all or most of the home's appliances, people, walls, energy consumption data, etc.

[0075] 5B, a network of berths 508, such as berths 1-N, can form a persistent virtual world system 112. For example, a city berth can connect to other city berths to form a country berth. If a country berth does not connect to any other berths 508, the country berth can represent a universe of the persistent virtual world system 112.

[0076] Referring to diagram 500c of FIG. 5C, a spatial view of a berth 508 is shown that includes multiple streams 502, such as streams 1-4, each containing multiple stream-related virtual objects 110.

[0077] 6A-6B are schematic diagrams 600a-600b of a division of a berth 508 into multiple cells 108, according to one embodiment.

[0078] 6A , an undivided verse 602 representing a region of the world may be divided into divided verses 604 containing multiple cells 606 as a resource manager requests resources based on demand from virtual objects 110. More resource-intensive cells 608, such as cells containing a greater number of virtual objects 110, may be further divided into a greater number of cells 606. Similarly, cells 606 with fewer virtual objects 110 may be divided into fewer cells 606. Thus, each cell 606 is allocated resources, with the more resource-intensive cells 608 receiving more resources than the less resource-intensive cells 606. For example, some cells 606 within the more resource-intensive cells 608 contain more virtual objects 110 than the remaining cells 606. However, it should be understood that the amount of resources allocated to each cell 606 is related not only to the number of virtual objects 110 in a particular cell 606, but also to the level of interaction within the portion of the persistent virtual world system visible to the user, the distance between the user and network equipment such as servers and / or antennas, the type of applications being used by the user, the type of entitlements the user may have depending on their individual user profile or contract, or a combination thereof. However, for simplicity and illustrative purposes, Figures 6A-6B use only virtual objects 110 as parameters for assessing demand and managing allocation to one or more cells.

[0079] In some embodiments, the data structure representing the real world into cells is an octree data structure, with each cell represented as a voxel within the octree data structure. In some embodiments, the voxels utilized within the octree data structure are sparse voxels for use in arranging larger portions of the real world. In other embodiments, the voxels utilized within the octree data structure are dense voxels for use in arranging smaller portions of the real world. Octrees and selected sparse or dense voxels are suitable data structures for representing three-dimensional space in virtual worlds, such as persistent virtual world systems, that represent the real world. Octrees enable fast and easy combination of multiple objects, implement relatively simple rendering algorithms, enable rapid spatial searches, and enable model generation through virtualization of real objects (e.g., through LIDAR or other image scanning methods). For virtual worlds, including 3D virtual worlds that represent at least a portion of the real world, such 3D partitioning of the world enables three-dimensional-based engine services and cloud resource allocation over areas where users may be viewing and / or interacting. However, quadtrees may also be utilized when dealing with 2D world cases. In other embodiments, other suitable data structures may be utilized, such as BSP trees, sparse voxel octrees, 3D arrays, kD trees, point clouds, wireframes, boundary representations (B-Rep), spatial domain structure trees (CSG trees), binary trees, and hexagonal structures.

[0080] In some embodiments, the data structure includes a collection of data values, relationships, and functions or operations that can be applied to data, the data defining at least the geometry and coordinates of virtual objects within the persistent virtual world system. For example, the selection of the exact data structure depends on the origin of the data, the precision of the geometry required during rendering, whether the rendering is done in real time or pre-rendered, whether the rendering is performed via a cloud server, a user device, a fog device, or a combination thereof, the particular application in which the persistent virtual world system will be used (e.g., medical or scientific applications may require a higher level of clarity than other types of applications), the storage capacity and therefore the desired memory consumption from the server and user device, etc.

[0081] 6A, after completing the events associated with one or more requests, the resources are returned to the original source or sources. The split berth 604 is then consolidated back into the original number of cells 606, as represented by the curved arrow 610 pointing from the split berth 604 back to the unsplit berth 602. In this way, the system achieves efficient and dynamic allocation of resources by selecting areas of interest with higher demand and returning resources to those sources when they are no longer needed.

[0082] 6B illustrates an octree data structure 600b. The octree data structure 600b directly represents a volume within a data structure, such as a volume representing at least a portion of the real world. The main advantages of the octree data structure 600b are that it allows for fast and easy combination of several objects, simple rendering implementation, performing rapid spatial searches, and model generation by digitizing real objects (e.g., LIDAR or other image scans).

[0083] The octree data structure 600b is a tree data structure in which each internal node has eight child nodes, or octants. As the bath is further divided, additional levels of child nodes are created, and more resources are allocated to further divided cells that require them. Thus, in the example of FIG. 6B, the octree data structure 600b generates three levels 612 of child nodes, i.e., levels 1-3, and a root node 614 representing the entire bath. Areas of the octree data structure 600b shown in bold represent areas with virtual objects 110. Higher levels of the bath's octree data structure 600b, for example, represent areas requiring fewer divisions and therefore fewer resources because fewer virtual objects 110 are present. In contrast, lower levels containing a greater number of virtual objects 110 require more divisions and therefore more resources.

[0084] FIG. 7 is a schematic diagram 700 of another embodiment of dividing a berth into multiple cells, according to one embodiment.

[0085] The schematic diagram 700 shows a split verse 604 comprised of three streams 702, Streams 1-3. Each stream 702 contains one or more stream-specific virtual objects 110, and each stream 702 is programmed to be enabled or disabled for viewing and interaction on a client device. In a further embodiment, each stream 702 is associated with one or more user applications. Each stream 702 may be configured to retrieve data from the corresponding virtual objects 110 and perform thematic and / or goal-specific simulations. Each stream 702 may connect to a root node or root simulation object that generates multiple stream-related virtual objects 110. The streams 702 may further connect to an import function that defines the virtual objects 110 the stream needs to perform the simulation.

[0086] For example, if a viewer is using a restaurant application, the user may optionally select to enable the restaurant stream (e.g., Stream 1) of the split verse 704, thereby showing or highlighting only the restaurant to the user. Similarly, a traffic-related application may be configured to enable a traffic stream (e.g., Stream 2) to show or highlight traffic to the user. In another example, a game-related application that includes multiple purely virtual objects may activate a game stream (e.g., Stream 3) to show or highlight the purely virtual objects associated with the game application. In this example, Stream 3 may require a greater amount of resources than the other Streams 1-2 because it has more virtual objects 110. Thus, combining all Streams 1-3 results in Split Verse 704. In some embodiments, depending on which streams 702 are activated, the activated streams 702 are viewed by the user as rendered or unrendered media streams output on the client device, where the streams 702 include 3D image data, 3D geometry, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, text data, time data, position data, orientation data, haptic data, lighting data, etc., or a combination of such media content. Thus, each stream 702 may be computed and optionally rendered by one or more individual distributed software engines provisioned by one or more hardware engines. In some embodiments, one or more of the individual distributed software engines include instructions that, when executed by the one or more hardware engines, trigger end-to-end encryption (E2EE) of the stream 702 so that only users interested in the particular stream 702 can receive and decrypt the media content of the stream 702.Such an embodiment may be used, for example, during a video call using an end-to-end streaming service.

[0087] Figure 8 is a schematic diagram of a client device 114, according to one embodiment. Some elements of Figure 9 may be similar to elements of Figures 1-8, and therefore similar or identical reference numbers may be used to indicate those elements.

[0088] The client device 114 of the present disclosure may include operational components such as an input / output (I / O) module 802, a power supply 804, a memory 806, a sensing mechanism 808 and transceiver 810 forming a tracking module 812, and a network interface 814, all of which are operably connected to a processor 816.

[0089] The I / O module 802 is implemented as computing hardware and software configured to interact with a user and provide user input data to one or more other system components. For example, the I / O module 802 may be configured to interact with a user, generate user input data based on the interaction, and provide the user input data to the processor 816 before being transferred over a network to another processing system, such as a server. In another example, the I / O module 802 is implemented as an external computing pointing device (e.g., a touchscreen, a mouse, a 3D control, a joystick, a gamepad, etc.) and / or a text input device (e.g., a keyboard, a dictation tool, etc.) configured to interact with the client device 114. In still other embodiments, the I / O module 802 may provide additional, fewer, or different functionality than those described above.

[0090] The power supply 804 is implemented as computing hardware and software configured to provide power to the client device 114. In one embodiment, the power supply 804 may be a battery. The power supply 804 may be built into the client device 114 or may be removable from the client device 114, and may be rechargeable or non-rechargeable. In one embodiment, the client device 114 may be repowered by swapping one power supply 804 for another. In another embodiment, the power supply 804 may be recharged by a cable attached to a charging source, such as a Universal Serial Bus (“USB”), FIREWIRE®, Ethernet, Thunderbolt, or headphone cable, attached to a personal computer. In yet another embodiment, the power supply 804 may be recharged by inductive charging, where an electromagnetic field is used to transfer energy from an inductive charger to the power supply 804, when the two are brought into close proximity but do not need to be plugged into each other via a cable. In another embodiment, a docking station may be used to facilitate charging.

[0091] The memory 806 may be implemented as computing hardware and software adapted to store application program instructions and to store sensor data captured by the plurality of sensing mechanisms 808. The memory 806 may be any suitable type of memory capable of storing information accessible by the processor 816, including a computer-readable medium or other medium that stores data that can be read with the aid of an electronic device, such as a hard drive, a memory card, a flash drive, a ROM, a RAM, a DVD or other optical disk, and other writable and read-only memory. The memory 806 may include temporary storage as well as persistent storage.

[0092] The sensing mechanism 808 may be implemented as computing hardware and software adapted to obtain various sensor data from the real world and determine / track the position and orientation of the client device 114. The sensing mechanism 808 may include one or more, including but not limited to, one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors (e.g., gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, chromatic sensors, humidity sensors, water sensors, electrical sensors, or combinations thereof. In particular, the sensing mechanism 808 includes one or more inertial measurement units (IMUs), accelerometers, and gyroscopes. The IMU is configured to measure and report the velocity, acceleration, angular momentum, translational velocity, rotational velocity, and other telemetry metadata of the client device 114 by using a combination of accelerometers and gyroscopes. The accelerometer, located within and / or configured separate from the IMU, may be configured to measure the acceleration of the interacting device, including acceleration due to the Earth's gravitational field. In one embodiment, the accelerometer includes a tri-axis accelerometer capable of measuring acceleration in three orthogonal directions.

[0093] The transceiver 810 may be implemented as computing hardware and software configured to enable the client device 114 to receive radio waves from an antenna and transmit data back to the antenna. In some embodiments, a millimeter wave transceiver may be employed that may be configured to receive millimeter wave signals from the antenna and transmit data back to the antenna when interacting with immersive content. The transceiver 810 may be a two-way communication transceiver 810.

[0094] In one embodiment, tracking module 812 may be implemented by combining the capabilities of an IMU, accelerometer, and gyroscope with the position tracking provided by transceiver 810, and the precise tracking, low latency, and high QOS capabilities provided by the millimeter wave-based antenna may enable sub-centimeter or sub-millimeter position and orientation tracking, which may increase accuracy when tracking the real-time position and orientation of client device 114. In an alternative embodiment, sensing mechanism 808 and transceiver 810 may be combined within a single tracking module device.

[0095] The network interface 814 may be implemented as computing software and hardware for communicatively connecting to a network, receiving computer-readable program instructions from the network transmitted by the server or client device 114, and transferring the computer-readable program instructions to the memory 806 for storage in the memory 806 for execution by the processor 816.

[0096] The processor 816 may be implemented as computing hardware and software configured to receive and process sensor data. For example, the processor 816 may be configured to send imaging requests, receive imaging data, process imaging data into environmental data or other data, process user input data and / or imaging data to generate user interaction data, perform edge-based (on-device) machine learning training and inference, send server requests, receive server responses, and / or provide user interaction data, environmental data, and content object data to one or more other system components. For example, the processor 816 may receive user input data from the I / O module 802 and implement application programs stored in the memory 806, respectively. In other examples, the processor 816 may receive sensor data captured from the real world from the sensing mechanism 808 or may receive the precise position and orientation of the client device 114 via the tracking module 812 and prepare a portion of the data before sending it to the server for further processing. By way of example, the processor 816 may implement some of the steps required during data preparation, including analog or digital signal processing algorithms, such as raw data reduction or filtering of the sensor data, before sending it to the server.

[0097] FIG. 9 is a block diagram of a computer-implemented method 900 for enabling dynamic load balancing through a distributed 3D engine, according to one embodiment.

[0098] A method 900 for enabling multidimensional 3D engine computing and virtualization-based dynamic load balancing of a virtual or real world begins in step 902 by providing, in a memory of a server computer, a data structure representing at least a portion of the real world to a plurality of cells that store virtual objects forming a persistent virtual world system, where at least some of the virtual objects are virtual replicas of corresponding real-world elements. The method continues in step 904 by providing, in a memory of at least one server computer, a distributed 3D engine including a resource manager and a plurality of individual distributed software engines implemented in a distributed deployment.

[0099] The method then proceeds to step 906, where the resource manager calculates the demand for each individual cell based on the current load. The demand further considers one or more of the amount of virtual objects and the level of interaction within the portion of the persistent virtual world system visible to the user avatar. In other embodiments, the demand considers other parameters, such as the distance between the user and the network equipment (e.g., server and / or antenna), the type of application being used by the user, or the type of entitlement the user may have depending on their individual user profile or contract. Based on the demand, the method continues to rank the cells according to demand in step 908. Thereafter, based on the ranked demand, the method continues to dynamically allocate resources to one or more more resource-intensive cells in step 910.

[0100] FIG. 10 is a block diagram of a computer-implemented method 1000 that further includes steps of the method 900 for enabling dynamic load balancing through a distributed 3D engine, according to one embodiment.

[0101] According to one embodiment, method 1000 begins in step 1002 by dividing a more resource-intensive region of interest of at least a portion of the world into a larger number of cells based on the calculated demand. Method 1000 then continues in step 1004 by allocating corresponding resources from one or more sources to each cell. After terminating events associated with one or more demands within the more resource-intensive region of interest, method 1000 continues in step 1006 by returning the resources to the original source or sources. Finally, method 1000 concludes in step 1008 by consolidating the berths back into the original number of cells.

[0102] FIG. 11 is a block diagram of a method 1100 illustrating resource allocation steps via a distributed message exchange platform according to one embodiment.

[0103] Method 1100 may begin with publishing resources corresponding to corresponding cells via a distributed message exchange platform of a resource manager using a publish / subscribe model at step 1102. Method 1100 then concludes at step 1104 with subscribing to cells of interest by one or more virtual objects to obtain the required resources via the distributed message exchange platform.

[0104] In some embodiments, method 1200 further includes sharing, via a distributed message exchange platform, a dynamically updated state of at least a portion of the world to which one or more client devices and / or server computers may subscribe to obtain an updated version of the world with the client devices, the server computer, or a combination thereof. In still further embodiments, the state of at least a portion of the world is obtained by a plurality of connected devices including sensors that provide sensor data to the persistent virtual world system, by user input, by server computation, or a combination thereof.

[0105] While particular embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are exemplary only and do not limit the broad disclosure, and that the disclosure is not limited to the specific configurations and arrangements shown and described, since various other variations may occur to those skilled in the art. Accordingly, the description is to be regarded as illustrative and not limiting.

Claims

1. 1. A system comprising one or more server computers, The one or more server computers include a memory and at least one processor, the memory comprising: a data structure that virtualizes at least a portion of the virtual or real world into a plurality of cells that store virtual objects that form a persistent virtual world system; a distributed 3D engine including a plurality of distributed software engines; Equipped with resources are dynamically allocated to the plurality of cells based on a current load of the plurality of cells via a distributed deployment; an original number of cells of the plurality of cells representing one or more regions of at least a portion of the virtual world or the real world is divided into a larger number of cells; The resource is restored based on one or more requests; the persistent virtual world system includes a low-fidelity simulation used for demand assessment and load balancing, and a high-fidelity simulation used to improve the user experience; system.

2. The system of claim 1 , wherein at least a portion of the virtual or real world is integrated into the original number of cells.

3. The system of claim 1 , wherein at least one of the virtual objects of the persistent virtual world system includes self-computing capabilities and autonomous behavior.

4. The system of claim 1 , wherein the distributed software engines are used simultaneously or sequentially across the distributed deployment to complement each other's engine services for the accomplishment of at least one task.

5. the data structure is an octree data structure; At least one of the plurality of cells is represented as a voxel in the octree data structure; and The system of claim 1 , wherein the voxels representing at least one of the plurality of cells are selected from among sparse voxels and dense voxels based on a size of the at least one of the plurality of cells.

6. 2. The system of claim 1, wherein the data structure comprises at least one of a BSP tree, a sparse voxel octree, a 3D array, a kD tree, a point cloud, a wireframe, a boundary representation (B-Rep), a spatial domain construction tree (CSG tree), a binary tree, a hexagonal structure, and combinations thereof.

7. a resource manager of the distributed 3D engine performing allocation of the resources via a distributed message exchange platform; the distributed messaging platform utilizes a publish / subscribe model; and at least one of the virtual objects subscribes to at least one of the cells in which the resource is published; The system of claim 1 .

8. the distributed message exchange platform shares the dynamically updated state of at least a portion of the virtual world or the real world stored in the memory with one or more client devices or servers; the dynamically updated state is modified via data obtained by one or more of a plurality of connected devices, by user input, by server calculation, or a combination thereof; The plurality of connected devices includes sensors that provide sensor data to the persistent virtual world system. The system of claim 7.

9. each cell of the plurality of cells includes one or more streams, each stream including a plurality of stream-specific virtual objects, each stream programmed to be enabled or disabled for viewing and interaction on a client device; and Each stream is associated with one or more applications. The system of claim 1 .

10. providing, in the memory of at least one server computer, a data structure that virtualizes at least a portion of a virtual or real world into a plurality of cells that store virtual objects that form a persistent virtual world system, at least one of the virtual objects being a virtual replica of a corresponding real-world element; providing a distributed 3D engine in the memory of the at least one server computer, the distributed 3D engine comprising a plurality of distributed software engines; c) dynamically allocating resources to individual cells of said plurality of cells based on the current load of said cells; A method comprising: an original number of cells of the plurality of cells representing one or more regions of at least a portion of the virtual world or the real world is divided into a larger number of cells; The resource is restored based on one or more requests; the persistent virtual world system includes a low-fidelity simulation used for demand assessment and load balancing, and a high-fidelity simulation used to improve the user experience; method.

11. The method of claim 10 , wherein at least a portion of the virtual or real world is aggregated into the original number of cells.

12. The method of claim 10 , wherein the distributed software engines are used simultaneously or sequentially via a distributed deployment to complement each other's engine services for the accomplishment of at least one task.

13. providing an octree data structure as the data structure representing the virtual world or the real world; at least one of the plurality of cells represents a voxel; The voxel representing at least one of the plurality of cells is selected from among sparse voxels and dense voxels based on a size of at least one of the plurality of cells. The method of claim 10.

14. dividing the one or more regions of at least a portion of the virtual world or the real world into additional cells based on demand for the plurality of cells; allocating resources corresponding to said additional cells; The method of claim 10 further comprising:

15. The step of allocating resources is performed by the following steps: publishing the resources to corresponding cells via a distributed message exchange platform of a resource manager using a publish / subscribe model; and subscribing by the one or more virtual objects to at least one of the plurality of cells to obtain required resources; 11. The method according to claim 10.

16. each of the plurality of cells includes at least one stream, the at least one stream including a plurality of stream-specific virtual objects, and the at least one stream configured to be enabled or disabled for viewing and interaction on a client device; and The method of claim 10 , wherein the at least one stream is associated with one or more user applications.

17. One or more non-transitory computer-readable media having stored thereon instructions that, when executed by one or more computers, cause the one or more computers to perform the following steps: providing, in the memory of at least one server computer, a data structure that virtualizes at least a portion of a virtual or real world into a plurality of cells that store virtual objects that form a persistent virtual world system, at least one of the virtual objects being a virtual replica of a corresponding real-world element; b. providing a distributed 3D engine in the memory of the at least one server computer, the distributed 3D engine comprising a plurality of distributed software engines; and c) dynamically allocating resources to individual cells of said plurality of cells based on the current load of said plurality of cells; configured to run an original number of cells of the plurality of cells representing one or more regions of at least a portion of the virtual world or the real world is divided into a larger number of cells; The resource is restored based on one or more requests. the persistent virtual world system includes a low-fidelity simulation used for demand assessment and load balancing, and a high-fidelity simulation used to improve the user experience; 1. A non-transitory computer-readable medium comprising:

Images