Multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of virtual or real worlds

JP2024123206A5Active Publication Date: 2024-11-22TMRW FOUNDATION IP SARL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024102308
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-12-30
Filing Date
2024-06-25
Publication Date
2024-11-22
Estimated Expiration
2040-12-23

AI Technical Summary

Technical Problem

Traditional virtual world systems, particularly those used in massively multiplayer online games (MMOGs), face limitations in scaling to support large numbers of players or entities due to resource constraints, leading to degraded quality of service and user experience, and existing 3D engines are monolithic, centralized, and difficult to scale up.

Method used

A distributed 3D engine system with a resource manager and individual software engines, dynamically allocating resources based on load and demand across multiple cells, utilizing a publish/subscribe model for efficient resource distribution and virtualization of virtual objects, including real-world replicas, to provide continuous and optimized user experiences.

Benefits of technology

The system achieves efficient load balancing and optimized user experiences by dynamically allocating resources to regions of interest, ensuring high-quality, low-latency interactions in virtual or real-world environments, even with varying levels of detail and fidelity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To provide a system, method and non-transitory computer-readable medium that enable a distributed 3D engine for performing dynamic load balancing through virtual worlds.SOLUTION: A system 100 comprises one or more servers comprising memory and at least one processor. The memory stores: a data structure representing at least one portion of a virtual or real world as a plurality of cells storing virtual objects; and a distributed 3D engine comprising a resource manager implemented in a distributed deployment and a plurality of individual software engines. Resources are dynamically allocated via the distributed deployment to one or more cells based on a current load and a corresponding computed and ranked demand. In some embodiments, the demand further considers one or more of the amount of virtual objects and the level of interactions within the portion of the persistent virtual world system 112 visible to a user avatar.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of U.S. Provisional Application No. 62 / 955,247, filed December 30, 2019, the contents of which are incorporated herein by reference. [Background technology]

[0002] Conventional systems designed to run virtual worlds, especially massively multiplayer online games (MMOGs), have limitations in how they can scale to support a very large number of players or entities within the virtual world. Because these virtual worlds typically run on a single physical computer and rely on a single instance of the game or application, the virtual world is supported by the limited bandwidth of the system and the processor capacity of the server hosting the virtual world. The server hosting the game or simulation engine must provide simulation capabilities for all objects and players within the virtual world. However, as the number of players and entities increases, resource requirements increase as well, degrading the quality of service (QOS) and user experience within the virtual world.

[0003] Other approaches include splitting the map and distributing it evenly, running parts of the map on multiple servers, with each server responsible for a corresponding part of the map. Although more resources are available to serve the entire map, this method still has some drawbacks. For example, congestion in one zone can make this type of geographic division inefficient for use in virtual worlds, and transitions from one zone to another can provide a discrete view, as opposed to a continuous view, to users crossing zones due to the required server switches.

[0004] In addition, current 3D engines used in the development and processing of virtual worlds have several shortcomings that limit the capabilities of virtual world provisioning. For example, typical current 3D engines tend to be monolithic, centralized, can be difficult to scale up, and can be platform dependent. This, in addition to the aforementioned shortcomings, prevents the ability to maximize the capabilities of each of the specific modules of the 3D engine during the development and execution of the virtual world. Summary of the Invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are described in more detail below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] The present disclosure relates generally to computer systems and methods, and more particularly to systems and methods that enable virtual or real world multi-dimensional 3D engine computing and virtualization-based dynamic load balancing.

[0007] The system of the present disclosure comprises one or more server computers with a memory and at least one processor, the memory storing a data structure that virtualizes at least a portion of a virtual or real world into a plurality of cells that store virtual objects forming a persistent virtual world system. In some embodiments, at least some of the virtual objects are virtual replicas of corresponding real world elements. The memory further stores a distributed 3D engine implemented in a distributed deployment, the distributed 3D engine including a resource manager and a plurality of individual distributed software engines. Resources are dynamically allocated by the distributed deployment to one or more cells based on a current load and corresponding calculated and ranked demand of the one or more cells. In some embodiments, the demand is based at least in part on an amount of virtual objects in the one or more cells, or a level of interaction within a portion of the persistent virtual world system visible to a user avatar in the one or more cells, or a combination thereof. In some embodiments, the resources include computing power, memory, network resources (e.g., bandwidth), or a combination thereof.

[0008] In further embodiments, the demand is further based on the distance between a user device accessing one or more cells and network equipment (e.g., servers and network antennas), the type of application being used by the user, or the type of entitlement a user accessing one or more cells may have (e.g., based on the user's individual user profile or contractual terms), or a combination thereof.

[0009] In some embodiments, the virtual objects further include purely virtual objects that are not available in the real world, virtual avatars of real people, and applications.

[0010] In some embodiments, the persistent virtual world system includes multiple simulations of the same virtual or real world, where the simulations vary from high fidelity to low fidelity, with the low fidelity simulations being used for demand assessment and load balancing, and the high fidelity simulations being used to improve the user experience. This allows different simulations to be performed depending on the case or situation. LOD management also allows for an optimized user experience to be provided depending on the specific requirements and context. Thus, the same virtual or real world can be simulated at different LODs to facilitate the load balancing method of the present disclosure while providing a suitable experience to the user.

[0011] In some embodiments, the virtual objects of the persistent virtual world system include data and models that refer to any graphical, mathematical, or logical representation of aspects of the virtual object or the corresponding real object. In some embodiments, the data and models provide the entities of the persistent virtual world system with self-computing capabilities and autonomous behavior. In some embodiments, suitable models include one or more of 3D models, geometric models, dynamic models, and machine learning models. The virtual objects can be fed with data obtained from a simulation, data entered by a user via a user device, or data captured by a sensor device such as an Internet of Things device (e.g., in the case of a persistent virtual world system based on the real world). A continuous flow of data can keep the persistent virtual world system updated periodically.

[0012] In some embodiments, the persistent virtual world system may comprise a virtual world layer that may be separated into an augmented reality (AR) layer and a virtual reality (VR) layer. The separate layers may allow access to the persistent virtual world system in either augmented reality or virtual reality and combine elements stored in each layer, and may be activated via a client device connected to at least one server computer via a network whenever accessing one or the other type of reality. Each of the layers may include augmentations of real and virtual objects that may be unique to each layer.

[0013] In some embodiments, each distributed software engine may be a distributed 3D engine decomposed into multiple dynamic and independent software modules that may communicate with each other when configured locally or with remotely located individual software modules. In some embodiments, multiple distributed software engines may be used simultaneously or sequentially in a network to create synergy with each other. For example, multiple 3D graphics engines hosted on the same or different server computers may allocate engine services sequentially or simultaneously via a distributed message exchange platform to complement each other's engine services for the realization of one or more specific tasks.

[0014] In some embodiments, the one or more processors are dedicated processing devices, general-purpose processing devices, or a combination thereof configured to process tasks from the corresponding individual distributed software engines in hardware. In one embodiment, in contrast to a typical system that may employ one or more of a standard central processing unit (CPU), a graphics processing unit (GPU), each dedicated hardware engine is tailored for processing an individual distributed software engine used in at least one of 3D graphics, artificial intelligence operations, physics operations, 3D structure processing operations, simulation operations, tracking and positioning, and communications. In some embodiments, each hardware engine may be combined as a chip or system on an electronic chip with one or more specific hardware processing cores designated to process each specific type of operation. The processing cores of the one or more electronic chips may be considered as functional units or modules of electronic circuits of the electronic chip designed to physically execute the required algorithms of each 3D engine. For example, at least one of the one or more processing cores may be realized as an FPGA or any other form of integrated circuitry implementing dedicated functions. Thus, the electronic chip includes at least one integrated circuit that realizes at least some of the one or more processing cores, and at least one of the processing cores implements a different dedicated hardware engine. These electronic chips may be located, for example, in one or more external servers or in the main server or both. In some embodiments, the electronic chips may further comprise general-purpose processing cores, such as a CPU and / or GPU, or may be connected to an external CPU and / or GPU to provide additional support for each engine task. Providing a dedicated hardware engine for each of the sources and configuring each of the 3D engine modules in a distributed and independent manner can provide additional resources to the system as a whole, in addition to allowing each 3D engine module to have one or more dedicated hardware cores from one or more hardware engines.

[0015] In some embodiments, the data structure representing the virtual or real world into cells is an octree data structure, with each cell represented as a voxel in the octree data structure. In some embodiments, the voxels utilized in the octree data structure are sparse voxels for use in arranging larger portions of the real world. In other embodiments, the voxels utilized in the octree data structure are dense voxels for use in arranging smaller portions of the real world. The octree and selected sparse or dense voxels are suitable data structures for representing three-dimensional space in a virtual world, such as a persistent virtual world system, that represents the real world. The octree allows for fast and easy combination of multiple objects, implements relatively simple rendering algorithms, allows for rapid spatial searches to be performed, and enables model generation through virtualization of real objects (e.g., through LIDAR or other image scanning methods). For virtual worlds, including 3D virtual worlds that represent at least a portion of the real world, such 3D partitioning of the world enables three-dimensional based engine services and cloud resource allocation over areas that a user may be viewing and / or interacting with.

[0016] In some embodiments, cells representing more resource-intensive areas of interest from at least a portion of the world are further divided into a larger number of cells. In further embodiments, each highly resource-intensive cell is allocated resources from one or more sources accordingly, with smaller cells being allocated more resources. In yet another embodiment, after completing an event associated with one or more requests, resources are returned from the multiple sources to the original one or more sources. At least a portion of the world is then consolidated back into the original number of cells. In this manner, the system achieves efficient and dynamic allocation of resources by selecting areas of interest with higher demand.

[0017] In some embodiments, the resource manager performs the allocation via a distributed message exchange platform. In further embodiments, the distributed message exchange platform utilizes a publish / subscribe model, where one or more virtual objects subscribe to one or more cells in which resources are published. In further embodiments, the distributed message exchange platform shares the dynamically updated state of at least a portion of the world stored in memory with one or more of the client devices and the server. Thus, the resource manager of the distributed 3D engine of the present disclosure publishes each resource once to the distributed message exchange platform, which is delivered to each virtual object that needs the resource by subscribing to the cell that has the published resource. This is in contrast to a typical 3D engine system tailored to provision a virtual world, which outputs a complete data update packet for each client device in the map, and tailors each packet to the client device so that the client device receives only data that may be relevant to the client device and data for objects in view, thereby enabling the game or simulation engine to send the same data multiple times, once for each client device where the object is visible.

[0018] According to one embodiment, the state of at least a portion of the world is further updated via a plurality of connected devices including sensors that provide sensor data to the persistent virtual world system. The connected devices may be, for example, Internet of Things (IoT) devices that may communicate with each other and with a server over a network and may continuously capture multi-source sensor data from the real world to update the virtual replica of the persistent virtual world system. Thus, state updates may result from sensor data. However, state updates may also result from user input or from computations (e.g., artificial intelligence inference) by one or more 3D engines. The world state updates may then be published on a distributed message exchange platform for all client devices and sources to subscribe to and view on demand.

[0019] In some embodiments, each cell of the plurality of cells includes one or more streams, each stream including one or more of the stream-specific virtual objects, and configured to be enabled or disabled for viewing and interacting on a client device. In further embodiments, each stream is associated with one or more user applications. Each stream may be configured to obtain data from a corresponding virtual object and perform a thematic and / or goal-specific simulation. Each stream may connect to a root node or root simulation object that generates a plurality of stream-related virtual objects. The stream may further connect to an import function that defines the virtual replica in which the stream needs to perform the simulation. In some embodiments, a plurality of streams including the whole or most of the virtual replicas in a given environment, such as a cell or a plurality of cells, their contexts, and the relationships and interactions between each virtual replica form a universe or sub-universe. A network of sub-universes may form the universe of a persistent virtual world system. For example, a sub-universe of a city may be called a city verse, which may represent all of the city's buildings, trees, people, streets, traffic lights, vehicles, etc., and their relationships, interactions, and contexts. There may be a sub-universe within another sub-universe, such as a residential verse within a city verse. A residential universe may contain a 3D design of the exterior and interior of the house, all the appliances in the house, people, walls, energy consumption data, etc. The network of all the sub-universes may form a universe of a persistent virtual world system that can be viewed in virtual reality or augmented reality. Each of these sub-universes may be dynamically divided into multiple cells where resources are dynamically allocated.

[0020] In an embodiment of the present disclosure, the system may be implemented in a cloud-to-edge infrastructure that may display distributed computing capabilities, including the adoption of cloudlets / edge systems such as public or private clouds, fog servers, peer-to-peer cloud server systems, distributed ledger-based infrastructures, enterprise systems, mobile platforms, and user devices. The distributed ledger-based infrastructure may be a decentralized immutable blockchain or distributed ledger network that facilitates the transfer and storage of data necessary to widely distribute the persistent virtual world system, including virtual replicas, pure virtual objects, applications, and any events or interactions within the persistent virtual world system. Resources, including physical servers and network equipment, through the cloud-to-edge infrastructure enable shared storage and computing that can be dynamically allocated depending on factors such as the distance of users to the resources and network, and the computational demands from users to each other and to the location where they are located, and command them to compute more or less depending on the situation. In some embodiments, the dynamic allocation of resources is codified into smart contracts using the distributed ledger infrastructure to automate the transaction logic (e.g., the computer logic behind each allocation of storage and computing) and the conclusion or outcome of such allocation.

[0021] According to one embodiment, a method for enabling a distributed 3D engine to perform dynamic load balancing through a virtual world includes virtualizing, in a memory of at least one server computer, a data structure representing at least a portion of the virtual or real world into a plurality of cells that store virtual objects forming a persistent virtual world system, at least some of the virtual objects being virtual replicas of corresponding real world elements. The method then provides, in a memory of at least one server computer, a distributed 3D engine implemented in a distributed deployment, the distributed 3D engine comprising a resource manager and a plurality of individual distributed software engines. The method then continues by calculating, by the resource manager, a demand for each of the plurality of cells based on a current load, the demand being based at least in part on an amount of virtual objects in the each of the cells or a level of interaction within a portion of the persistent virtual world system visible to a user avatar in the each of the cells. The method then continues by ranking the each of the cells according to the demand. Finally, based on the ranked demand, the method ends with dynamically allocating resources to the each of the cells (e.g., one or more more resource-intensive cells).

[0022] According to one embodiment, the method further includes dividing at least a portion of an area of ​​interest of the virtual or real world into additional cells based on the calculated demand and allocating corresponding resources to the additional cells (e.g., each cell representing a resource-intensive area of ​​interest). In one embodiment, the method further includes returning resources to the original one or more sources after completing an event associated with one or more demands in the more resource-intensive area of ​​interest and consolidating at least a portion of the world in the persistent virtual world system back to the original number of cells.

[0023] In some embodiments, the allocation is performed by publishing resources to corresponding cells via a distributed message exchange platform of the resource manager using a publish / subscribe model, and by one or more virtual objects subscribing to cells of interest to obtain the required resources.

[0024] In some embodiments, the method further includes sharing, via a distributed message exchange platform, a dynamically updated state of at least a portion of the world to which one or more client devices and / or server computers subscribe to obtain an updated version of the world. In still further embodiments, the state of at least a portion of the world is obtained by a plurality of connected devices including sensors providing sensor data to the persistent virtual world system, by user input, by server computation, or by a combination thereof.

[0025] The above summary does not include an exhaustive list of all aspects of the present disclosure. The present disclosure is intended to include all systems and methods that may be implemented from all appropriate combinations of the various aspects summarized above, as well as those systems and methods disclosed in the following "Description of the Preferred Embodiments" and particularly pointed out in the claims filed herewith. Such combinations have certain advantages not specifically recited in the above summary. Other features and advantages of the present disclosure will become apparent from the accompanying drawings and the following detailed description.

[0026] Specific features, aspects, and advantages of the present disclosure will become better understood with reference to the following description and accompanying drawings. [Brief description of the drawings]

[0027] [Figure 1]FIG. 1 is a schematic diagram of a system for enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of a virtual or real world according to one embodiment. [Diagram 2] FIG. 1 is a schematic diagram of a server of a system for enabling virtual or real world multi-dimensional 3D engine computing and virtualization-based dynamic load balancing according to one embodiment. [Diagram 3] FIG. 2 is a schematic diagram of resource allocation via a distributed message exchange platform according to one embodiment. [Figure 4A-4B] FIG. 2 is a schematic diagram of engine service allocation via a distributed message exchange platform according to one embodiment. [Figure 5A-5C] 1 is a schematic diagram of relationships between elements of a persistent virtual world system according to one embodiment. [Figure 6A-6B] FIG. 2 is a schematic diagram of a division into multiple cells according to one embodiment. [Figure 7] FIG. 2 is a schematic diagram of another embodiment of a division into multiple cells according to an embodiment. [Figure 8] 2 is a schematic diagram of a client device according to one embodiment. [Figure 9] FIG. 1 is a block diagram of a computer-implemented method for enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of a virtual or real world according to one embodiment. [Figure 10] FIG. 11 is a block diagram including further steps of a computer-implemented method for enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of a virtual or real world according to one embodiment. [Figure 11] FIG. 2 is a block diagram illustrating a resource allocation step via a distributed message exchange platform according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] In the following description, reference is made to the drawings which illustrate various embodiments. The various embodiments are also described below with reference to several examples. It is to be understood that the embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.

[0029] FIG. 1 is a schematic diagram of a system 100 for enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of virtual or real worlds according to one embodiment.

[0030] The system 100 comprises a number of server computers 102 (e.g., external servers A-C) implementing a distributed 3D engine including a resource manager in a distributed deployment 106 via a network 104. The distributed deployment 106 refers to a software implementation of the distributed 3D engine in the number of server computers 102 using a distributed architecture such as the architecture of the system 100. The server computers 102 dynamically allocate resources to one or more of a number of cells 108 (e.g., cells 1-3) via a resource manager using a publish / subscribe model. Each cell 108 virtually represents one or more different areas of the real or virtual world. The allocation is based on the current load and the corresponding calculated and ranked demand. In some embodiments, the demand further considers one or more of the amount of virtual objects 110 and the level of interaction within the portion of the persistent virtual world system visible to the user avatar, as described in more detail below.

[0031] One or more of the plurality of cells 108 include virtual objects 110, and a plurality of those virtual objects 110 within the cells 108 form a persistent virtual world system 112. In some embodiments, at least some of the virtual objects 110 are virtual replicas of corresponding real-world elements. In some embodiments, the demand for an individual cell takes into account the amount of virtual objects 110 in the individual cell, the level of interaction within the portion of the persistent virtual world system visible to the user via the respective client device 114, or a combination thereof, as described in further detail below. In further embodiments, the demand takes into account other parameters, such as the distance between the client device 114 and network equipment such as a server and / or antenna, the type of application being used by the user, or the type of entitlement the user may have depending on the individual user profile or contractual content, as described in further detail below.

[0032] In this disclosure, the term "persistent" is used to characterize a state of a system that can continue to exist without continuously running processes or network connections. For example, the term "persistent" may be used to characterize a virtual world system in which the virtual world system and all of the objects contained therein continue to exist independent of a user connected to the virtual world system after the process used to create the virtual objects has terminated. Thus, the virtual world system is stored in a non-volatile storage location (e.g., a server). In this manner, virtual replicas, pure virtual objects, and applications can interact and collaborate with each other when configured to accomplish a particular goal, even when the user is not connected to the server.

[0033] In some embodiments, to reduce hardware and network demands, contribute to reduced network latency, and improve the general mixed reality experience, the system may be connected through a network 104 including millimeter wave (mmW) or a combination of mmW and sub-6 GHz communication systems, for example, through fifth generation wireless system communications (5G). In other embodiments, the system may be connected through a wireless local area network (Wi-Fi). The communication system provided may enable low (e.g., about 1 to about 5 ms) end-to-end (E2E) latency and high (e.g., 1 to 10 Gbps) downlink speeds to endpoints in the field, depending on the parameters typically required to run highly interactive digital reality applications or other high demand applications. The result is high quality, low latency, real-time digital application content streaming. In other embodiments, the system may be communicatively connected through fourth generation wireless system communications (4G), may be supported by a 4G communication system, or may include other wired or wireless communication systems.

[0034] In other embodiments, a Global Navigation Satellite System (GNSS), which refers to satellite navigation systems such as GPS, BDS, Glonass, QZSS, Galileo, and IRNSS, may be used to enable positioning of the device. Using signals from a sufficient number of satellites and techniques such as triangulation and trilateration, the GNSS can calculate the device's position, velocity, altitude, and time. In one embodiment, the external positioning system is extended by Assisted GNSS (AGNSS) through the architecture of an existing cellular communication network, where the existing architecture includes 5G. In other embodiments, the AGNSS tracking system is further supported by a 4G cellular communication network. In indoor embodiments, the GNSS is further extended through a wireless local area network, such as, but not limited to, Wi-Fi, preferably providing data at 60 GHz. In alternative embodiments, the GNSS is extended through other technologies known in the art, such as Differential GPS (DGPS), Satellite-Based Augmentation System (SBAS), and Real-Time Kinematic (RTK) systems. In some embodiments, tracking of the device is implemented by a combination of AGNSS and inertial sensors within the device.

[0035] In some embodiments of the present disclosure, the system 100 may be implemented within a cloud-to-edge infrastructure that may represent a distributed computing capability employing computing devices and systems such as public or private clouds, fog servers, cloudlets, peer-to-peer cloud server systems, distributed ledger-based infrastructures, and edge devices / systems such as enterprise systems, mobile platforms, and user devices (all of which may connect via the network 104). Using the cloud-to-edge computing network, access to computing power, computer infrastructure (e.g., via so-called infrastructure as a service, or IaaS), applications, and business processes may be delivered on demand as a service to users via the client devices 114. In this manner, resources including physical servers and network equipment allow for shared storage and computing that may be dynamically allocated. In some embodiments, the client devices 114 include user devices and other devices. The user devices may comprise, for example, mobile devices, personal computers, game consoles, media centers, head-mounted displays, and see-through devices (e.g., digital reality contact lenses). The other devices may be configured to, among other things, capture multi-source sensor data from real-world objects via sensor mechanisms to update and enrich the virtual replicas of the corresponding real-world objects. According to one embodiment, the other devices are one or more surveillance cameras, vehicles, traffic lights, buildings, roads, railroad tracks, household appliances, etc.

[0036] FIG. 2 is a schematic diagram of a server 102 of a system 100 for enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of virtual or real worlds, according to one embodiment.

[0037] The one or more servers 102 comprise one or more processors 202 and a memory 204 that stores data structures that virtualize at least a portion of the virtual or real world as a number of cells (e.g., cells 108 in FIG. 1), each cell including a virtual object 110, forming a persistent virtual world system 112. In an embodiment where the persistent virtual world system 112 is based on the real world, some of the virtual objects 110 are virtual replicas 206 of corresponding real world elements. In other embodiments, the virtual objects 110 further include applications 208 and pure virtual objects 210 that do not exist in the real world. The memory 204 further stores a distributed 3D engine 212 that includes a resource manager 214 that manages resources. The resource manager 214 utilizes a distributed message exchange platform 216 that uses a publish / subscribe model to allocate resources including computing power, memory, and network by publishing resources on one or more cells to which the virtual objects can subscribe. The resource manager 214 may act as a virtual layer that interfaces the hardware and software components that manage the resources. Management actions may include assessment, provisioning, scaling, and replication of resources via the distributed message exchange platform 216.

[0038] In this disclosure, the term "virtual replica" refers to an accurate and persistent virtual representation of a real-world element. In one embodiment, the virtual replica 206 of the present disclosure refers to a virtual version or twin of a real-world element or a reality twin, which may be mapped or modeled by computer-aided drawing (CAD) or computer-aided engineering (CAE) methods, input of a computer model of the real object (e.g., Building Information Model (BIM)), or other suitable prior art methods, and may reflect not only the appearance but also the behavior of the reality twin. Some of the real-world objects may further include sensors that can provide multi-source input data to the virtual replica 206 to enrich and synchronize the virtual replica 206 with their respective real-world counterparts.

[0039] In some embodiments, the virtual replica 206 includes one or more of 3D world / building data, such as SLAM or differential mapping based data, 3D geometry data, 3D point cloud data, or geographic information system data representing real-world structural characteristics that may be useful for modeling 3D structures for an application.

[0040] In this disclosure, the term "virtualization" refers to the act of creating an abstracted virtual version of at least a portion of a virtual or real world or a persistent virtual world system 112. In the case of the real world, virtualization refers to creating virtual versions of elements found in the real world, as well as virtual versions of any resources (e.g., computing power, memory, or network) and associated equipment of the real world, and allowing a user (e.g., a human user, a computer program, or a machine) to interact with the real world elements through the persistent virtual world system 112. In this way, the user can feel as if the interaction took place directly with the real world elements. For example, a human user can interact with elements of a real city through a virtual city in augmented reality or virtual reality through a user interface of a user device (e.g., a mobile phone or a head-mounted display device), where multiple elements of the real city are virtualized within the persistent virtual world system 112. For physical manipulation of the real world through the virtual replica, electromechanical devices can be installed on the real world objects to translate remote commands sent to the real world objects through the virtual replica into physical actions from the real world elements. Regardless of whether the persistent virtual world system 112 is based in a virtual world or a real world, a virtual object may further include a model, which refers to any graphical, mathematical, or logical representation of an aspect of a virtual object or a corresponding real object. In some embodiments, such a model provides entities of the persistent virtual world system 112 with self-computing capabilities and autonomous behavior. In some embodiments, suitable models include one or more of a 3D model, a geometric model, a dynamic model, and a machine learning model.

[0041] The distributed 3D engine 212 of the present disclosure refers to a distributed version of a 3D engine implemented in a distributed deployment (e.g., multiple server computers or client computers). The distributed 3D engine 212 is configured to virtually pool, allocate, provision, dynamically load balance and allocate resources (e.g., computing power, memory and network) via the resource manager 214. The connection and access of the distributed 3D engine 212 to the persistent virtual world system 112 allows load balancing to be performed precisely by using spatial computing principles to evaluate demand in each of the cells representing at least a portion of the real or virtual world. As part of resource provisioning, the distributed 3D engine 212 provides multiple engine services (e.g., 3D rendering, graphics, physics, audio, artificial intelligence, positioning, and communication) on demand via multiple individual distributed software engines. The hardware operations performed to provide each of the above-mentioned engine services may be provided by multiple corresponding hardware engines. This pooling of resources is implemented through virtualization as described above. Thus, the concept of a distributed 3D engine 212 in this disclosure should not be interpreted simply as a collection of abstracted functions in a library that can be reused across different applications (e.g., video games), but rather as a distributed hardware and software implementation that encompasses engine operations that are pooled, allocated, provisioned, and load-balanced via a resource manager 214 using spatial computing principles that employ a persistent virtual world system as the basis for said operations.

[0042] In this disclosure, the term "spatial computing" refers to the use of physical space to send input and receive output from a computer. Thus, spatial computing uses the geometric properties of virtual objects as well as their positions and orientations to perform required computing and dynamic load balancing.

[0043] The virtual objects 110 may be fed with data obtained from a simulation, data input by a user through a user device, or data captured by a sensor device such as an Internet of Things device (e.g., in the case of a persistent virtual world system based on the real world). A continuous flow of data can keep the persistent virtual world system 112 periodically updated. Thus, the persistent virtual world system 112 from at least a portion of the real world is not only a visual 3D representation of the real world, as is the case in many current virtual world systems, but also a multi-dimensional representation of the real world, as the virtual objects 110 include models and data from other dimensions or aspects of the real world. For example, multiple virtual objects 110 may include data related to temperature, fuel consumption, speed, power, pressure, mass, sound, etc., allowing for connections and communication between each virtual object as well.

[0044] In some embodiments, virtualization of a virtual or real world involves running multiple simulations of the same world at various levels of detail (LOD). LOD management can associate different physics models from low to high fidelity models to the virtual replica, thus increasing the efficiency of computing processes such as the rendering process by reducing the workload in graphics pipeline usage, typically vertex transformation, or by enhancing physics simulation. This allows different simulations to be performed depending on the case or situation. LOD management also allows for an optimized user experience depending on the specific requirements and context. Thus, the same virtual or real world can be simulated at different LODs to facilitate the load balancing method of the present disclosure while providing a suitable experience to the user. For example, the real world simulation in the persistent virtual world system 112 may include a high fidelity simulation of at least a portion of the real world intended for visualization of a human user, and a low fidelity simulation to evaluate the demand of each cell and facilitate load balancing of the high fidelity simulation. Thus, the low fidelity simulation may facilitate viewing which areas are overloaded and which are not, while utilizing fewer resources. In another example, virtualization of a manufacturing plant may enable accurate management of the manufacturing plant, including demand assessment and corresponding resource allocation via the distributed 3D engine 212, with accurate spatial representation and relationships between objects, as well as data and models related to the current energy consumption, production rate, efficiency, etc., of each machine in the manufacturing plant. The manufacturing plant may be simulated in high fidelity for management of the manufacturing plant (e.g., control of the manufacturing process and all elements in the plant) and further in low fidelity for managing the load balancing of the manufacturing plant. The same LOD principles described above can be applied to the persistent virtual world system 112 based on a virtual world, where a low fidelity version of the virtual world may be simulated within the persistent virtual world system 112 for low load balancing of the virtual world and a high fidelity version of the virtual world may be simulated to provide a better user experience.

[0045] "Self-computing capability", also referred to as "self-management capability", herein refers to the capability of applying artificial intelligence algorithms to autonomously manage computing resources. In one embodiment, a virtual replica with self-computing capability in a persistent virtual world system can autonomously manage computing resources to adapt to changes in the environment of the corresponding real-world element or changes in the real-world element itself. The self-management rules and conditions can be further managed by the use of smart contracts implemented in blockchain or distributed ledger technology to further codify the rules and conditions in a decentralized and transparent manner. Thus, in one embodiment, each virtual object, such as a virtual replica of a real object, can exhibit autonomous behavior that operates autonomously in response to real-world conditions reflected in the persistent virtual world system (e.g., by allocating required resources, autonomously sending and executing commands, and / or generating events required by the respective situation). To achieve this type of behavior, it may be necessary to train the virtual object with artificial intelligence algorithms during modeling of the virtual object.

[0046] A virtual replica may obtain data from one or more sources (e.g., one or more real-world objects, environmental sensors, computing devices, etc.). As used herein, the term "multi-source data" refers to data that may be obtained from multiple sources. A virtual replica may be updated based on the multi-source data, for example, by updating known parameters or features, enriching the virtual replica with additional parameters or features.

[0047] In this disclosure, the term "enriching" is used to describe the act of providing a virtual replica with additional characteristics based on multi-source data. For example, enriching a virtual replica may refer to providing real-world data captured from sensor mechanisms on a client device, where the additional real-world data includes video data, temperature data, real-time energy consumption data, real-time water consumption data, speed or acceleration data, etc.

[0048] In some embodiments, some of the virtual replicas 206 may be virtual avatars of real users who access the persistent virtual world system 112 through their client devices. In these embodiments, resources are published on the corresponding cells in which the user's virtual avatar is active, for example, corresponding to the position and orientation that the user can view through his client device, so that resources are received by the user through the corresponding virtual avatar subscribing to the corresponding cell, through the user client device. Similarly, when applications 208 and / or pure virtual objects 210 are active in a cell, these virtual objects 110 subscribe to that cell and trigger the resource manager 214 to evaluate demand and accordingly publish more or less resources that the virtual objects 110 can obtain by subscribing to the corresponding cell.

[0049] For example, a particular traffic light in a city may be linked to a road or street and also associated with other objects such as sensors or other traffic lights. These relationships can trigger events that may generate new interactions within the persistent virtual world system 112, for example, an emergency vehicle in a street may automatically trigger a priority event that changes the traffic light pattern to avoid stops at intersections based on a simulation of the emergency vehicle's trajectory and the actual traffic conditions in the area. The management of resources during such events, including provisioning of devices (e.g., automobiles) involved in such events, is performed by the resource manager 212. In some embodiments, such conditions, events, and outcomes are programmed as smart contracts bound to a traffic-specific distributed ledger, for example, a traffic sidechain with many smart contracts that define various codified traffic rules and expected behaviors, thus automating and recording the occurrence of such events.

[0050] In some embodiments, a pure virtual object 210 refers to an object that exists only within the persistent virtual world system 112 and therefore does not have a corresponding object in the real world.

[0051] In some embodiments, the application 208 may be one or more traditional applications, distributed applications, or decentralized applications. Traditional applications are typically based on a traditional client / server model and run on dedicated servers in a static infrastructure. Decentralized applications are applications that are primarily stored on a cloud computing platform, such as the cloud servers of the present disclosure, and may run simultaneously on multiple systems and devices on the same network, or may run on a blockchain or distributed ledger-based distributed database. Decentralized applications primarily run on a decentralized infrastructure, such as a blockchain or distributed ledger-based distributed database.

[0052] In some embodiments, the persistent virtual world system 112 may comprise a virtual world layer that may be separated into an augmented reality (AR) layer and a virtual reality (VR) layer. The separate layers may allow accessing the persistent virtual world system in either augmented reality or virtual reality and combining elements stored in each layer, and may be activated via a client device connected to at least one server computer 102 via a network whenever accessing one or the other type of reality. Each of the layers may include augmentations of real objects and virtual objects 110 that may be unique to each layer. For example, the AR layer may include only the virtual replicas 206, applications 208, and pure virtual objects 210 that may only be visible when accessing the persistent virtual world system 112 in augmented reality. Similarly, the VR layer may include only the virtual replicas 206, applications 208, and pure virtual objects 210 that may only be visible when accessing the persistent virtual world system 112 in virtual reality. However, the user may decide to activate both layers or adjust the levels of each layer so that at least some objects of one layer are visible in the other layer. For example, when accessing a virtual reality scene, it may be possible to activate an augmented reality layer that allows viewing an extension of the reality object virtualized in the VR layer, such as seeing an additional description of a building or a tourist attraction.

[0053] In some embodiments, the distributed 3D engine 212 further comprises one or more individual distributed software engines 218. The distributed software engine 218 may be a 3D engine decomposed into multiple dynamic and independent software modules, each software module being programmed to perform a specific engine task. The individual distributed software engines 218 may communicate with each other, for example, via the distributed message exchange platform 216. Each individual distributed 3D engine 218 may use the message exchange platform 216 to invoke service functions in other individual distributed software engines 218. Such invocations may be performed indirectly (e.g., through a generic proxy function invoked by the distributed message exchange platform 216) or directly (e.g., by the distributed message exchange platform 216 maintaining direct communication with each individual distributed 3D engine 218).

[0054] In some embodiments, the one or more processors 202 may be one or more dedicated hardware engines 220. In other embodiments, the one or more processors 202 are general-purpose processing units or dedicated processing units supported by a general-purpose processing unit. In some embodiments, the dedicated processing units are processing units tailored for the processing of one or more individual distributed software engines 218 to perform engine tasks. Thus, the dedicated hardware engines 220 may be individually designed and configured for processing operations related to one or more of 3D graphics, artificial intelligence operations, physics operations, 3D structure processing operations, simulation operations, tracking and positioning, or communication thereof. In some embodiments, each hardware engine 220 may be coupled to one or more electronic chips or systems on a chip with one or more specific hardware processing cores designated to process each specific type of operation. The processing cores of the one or more electronic chips may be considered as functional units or modules of electronic circuits of the electronic chip designed to physically execute the required algorithms of each individual distributed software engine 218. For example, at least one of the one or more processing cores may be realized as an FPGA or any other form of integrated circuitry implementing a dedicated function. Thus, the one or more electronic chips include at least one integrated circuit that realizes at least some of the one or more processing cores, at least one of which implements a different dedicated hardware engine 220. In some embodiments, the electronic chip may further comprise general-purpose processing cores, such as a CPU and / or GPU, or may connect to an external CPU and / or GPU to provide additional support for the respective engine tasks.By providing dedicated hardware engines 220, each individual distributed 3D engine 218 can obtain resources from one or more dedicated hardware cores from one or more hardware engines 220, allowing each individual distributed software engine 218 to pull more resources to perform a greater number of tasks within the persistent virtual world system.

[0055] Processor 202 may refer to a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. Furthermore, explicit use of the term "processor" should not be construed to refer only to hardware capable of executing software, but may implicitly include, but is not limited to, digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), FPGAs, and the like.

[0056] The memory 204 may be any suitable type of memory capable of storing information accessible by the processor 202, including computer-readable media or other media that store data that can be read with the aid of electronic devices such as hard drives, memory cards, flash drives, ROM, RAM, DVDs or other optical disks, and other writable and read-only memories. The memory may include temporary storage in addition to persistent storage. The instructions may be executed directly by the processor 202 (e.g., machine code) or indirectly (e.g., script). The instructions may be stored in object code format for direct processing by the processor 202, or in any other computer language including scripts or collections of independent source code modules that may be interpreted on demand or pre-compiled. Data may be retrieved, stored, or modified by the processor 202 according to the instructions. The data may be stored in computer registers in a relational database, for example, as a table with multiple different fields and records, an XML document, or a flat file, etc. The data may also be formatted in any computer-readable format.

[0057] In some embodiments, interactions with and between virtual objects 110 may be defined through the use of computer code contained in computer scripts and computer programs and enabled by applications, smart contracts, or combinations thereof provisioned within the persistent virtual world system 112 where the interactions and interaction applications take place. Thus, the applications, smart contracts, or combinations thereof allow for digitally codifying and expressing all possible desired interactions and interaction interfaces between all elements of the real world via the persistent virtual world system 112. The interactions may be experienced by the user in the form of applications such as interactive games or through game-like interaction mechanisms.

[0058] FIG. 3 is a schematic diagram of a system 300 for allocating resources via a distributed message exchange platform 216 of a resource manager according to one embodiment.

[0059] The resource allocation system 300 allocates resources 302 to the distributed message exchange platform 216, where the resources 302 include computing power 304, memory 306, and network 308, and where the resources 302 may be obtained from one or more server computers 102 and / or client devices 114. The distributed message exchange platform 216 receives requests from one or more virtual objects 110 present in the cells 108 and performs the corresponding allocations by publishing to the cells 108. In some embodiments, to perform these resource allocations, the distributed message exchange platform 216 utilizes a publish / subscribe model, whereby one or more virtual objects 110 subscribe to one or more cells 108 in which the resources 302 are published. For example, a virtual object representing a user, or user virtual avatar, may be active in a particular cell 108, triggering a resource manager to publish resources required by the user virtual avatar to the corresponding cell. The publishing is performed via the distributed message exchange platform 216. Thus, resources 302 are published once to the distributed message exchange platform 216 and distributed to each virtual object 110 that subscribes 310 to the resources 302. This is in contrast to typical 3D engine systems developed to provision virtual worlds, which output a complete data update packet for each client device in the map, but tailor each packet to the client device such that the client device receives only the data that may be relevant to the client device and for objects in its field of view, thereby allowing the 3D engine to send the same data multiple times, once to each client device that has the object visible.

[0060] The publish / subscribe model is a model of communication based on content generation and consumption, which differs from the typical client / server-based communication model, where communication is usually between two nodes (i.e., between a client and a server). Communication via such a publish / subscribe model can be anonymous, i.e., the server computer 102 and / or the client device 114 do not need to know who has subscribed to which cell 108 to obtain the resource 302, and similarly, the virtual object 110 that obtains the resource does not need to know the source of the resource 302. A typical publish / subscribe model allows for information exchange in which anyone can publish and receive information, allowing for a higher degree of freedom than a typical client / server model. However, the use of a publish / service model can also bring about inefficiency concerns, because typically every publication is conveyed to every participant, who must analyze the information to determine its relevance. On the other hand, the publish / subscribe model of the present disclosure allows for a greater amount of resources to be dynamically concentrated in the cells 108 that need them most (e.g., in the more resource-demanding cells 108 to which virtual objects 110 can subscribe), and the amount of resources published takes into account the amount of virtual objects 110, the level of interaction within the portion of the persistent virtual world system visible to the user, and other parameters, so that each virtual object 110 obtains only the resources it needs from each subscribed cell during its respective simulation.

[0061] In some embodiments, the computing power 304 of the resource 302 includes engine services 314 obtained from one or more individual distributed 3D engines (e.g., individual distributed software engines 218 of FIG. 2). In further embodiments, the memory 306 of the resource 302 stores a world state that is further published to a corresponding cell 108 to provide an updated version of the real world through the virtual world to client devices subscribing to the cell 108. In further embodiments, the distributed message exchange platform 308 is configured as part of a network resource that is optionally shared. In some embodiments, multiple distributed software engines 218 can be used simultaneously or sequentially in the network 308 to create synergy with each other.

[0062] The term "engine services" as used herein refers to operations performed and provided by one or more 3D engines to accomplish one or more tasks, such as a game or other application-related task. A 3D engine typically includes software components, such as libraries, software development kits (SDKs), objects, etc., required to perform such tasks. Engine services include, for example, services related to 3D rendering, graphics, physics, audio, artificial intelligence, positioning, and communications. Each of the engine services may be performed by an individual distributed software engine provisioned by a hardware engine, which may be a dedicated hardware engine, a general-purpose hardware engine, or a combination thereof, with each hardware engine comprising dedicated circuitry for executing instructions from functions called by the individual distributed 3D engine.

[0063] The term "world state" as used herein refers to the state of each of the virtual objects in the persistent virtual world system, which may be updated in time through multiple connected devices, including sensors that provide sensor data to the persistent virtual world system. The connected devices may be, for example, Internet of Things (IoT) devices that may communicate with each other and with a server over a network, and may continuously capture multi-source data from the real world to update the virtual replica of the persistent virtual world system. Thus, state updates may result from sensor data. However, world state updates may also result from user input or from computations (e.g., artificial intelligence inference) from one or more 3D engines.

[0064] In some embodiments, allocation to virtual objects 110 by publishing to cells 108 may be based on current load and corresponding calculated and ranked demand. With reference to FIG. 2, such calculation and ranking of demand may be performed, for example, by the resource manager 214 of the distributed 3D engine 212. The ranking may aid in determining the amount of resources 302 to publish to each cell 108, such that cells with lower demand ranks may obtain fewer resources 302 than cells with higher demand ranks. In some embodiments, the demand considers the amount of virtual objects 110 within the portion of the persistent virtual world system visible to a user via a corresponding client device in the cell. The portion of the persistent virtual world system visible to a user avatar may be defined by the avatar's field of view or field in the cell. For example, when a user views a portion of the world in augmented reality or virtual reality, a corresponding avatar may be available in the persistent virtual world system, and thus the user avatar's view frustum may define such field of view. In other embodiments, when a user is using an avatar via a "flat" user interface, such as a computer or mobile phone screen, the actual user avatar may be visible within the computer screen, and the user avatar may view and interact within a portion of the persistent virtual world system. In such an example, the user avatar may view the persistent virtual world system from a first-person perspective, and the first-person avatar's viewing frustum may define the field of view. In embodiments where the user avatar is viewed from a top-down perspective, the field of view may be defined by a radius centered on the user avatar. In all cases, the field of view may include multiple virtual objects 110 with which the user may have the possibility to interact via the user avatar. Thus, the more virtual objects in the user avatar's field of view, the higher the resource demand for that particular cell, and therefore the higher the ranking demand score. In other embodiments, the demand further considers the level of interaction within the portion of the persistent virtual world system visible to the user.For example, there may be two user avatars in two different regions and two different cells of a persistent virtual world system, both having five virtual objects 110 available for interaction. If a first user decides to interact with all virtual objects 110 simultaneously (e.g., by participating in a group activity, conversation, battle, etc.), and a second user avatar does not interact with any of the virtual objects 110 or has a lower level of interaction with such virtual objects 110, such a cell may have a higher ranked demand and thus be allocated more resources than the second user avatar. In this example, the ranked demand is higher for the virtual cell in which the first user avatar is located than for the virtual cell in which the second user avatar is located.

[0065] In other embodiments, the demand further factors in other parameters such as the distance between the client device and the network equipment (e.g., server and / or antenna), the type of application being used by the user, or the type of entitlement the user may have depending on the individual user profile or contract. As an example of the distance between the client device and the network equipment, since each cell is assigned a specific network equipment, the further the user is from the network equipment in a particular cell of the persistent virtual world system, the more network equipment may be simultaneously activated to compensate for a potentially lower quality of service (QoS), and the higher the demand ranking for that cell. As an example of the application being used by the user, if the application consumes more resources such as bandwidth or computing power because it requires a higher real-time rendering rate or a better level of detail and clarity, it indicates that the resources required for that cell will increase, and thus the ranked demand will be correspondingly higher. As an example of the type of entitlement, the user may be registered as a premium account user to obtain a higher quality of service (QoS) and higher data transfer rate than other users, resulting in a higher resource allocation than other users. Furthermore, all of the above factors may be calculated in combination with each other to determine the ranked demand of each cell.

[0066] 4A and 4B are schematic diagrams 400a and 400b of engine service allocation via the distributed message exchange platform 216 according to one embodiment.

[0067] The schematic diagram 400a of Figure 4A includes multiple individual distributed software engines 218 hosted on one or more servers, such as server computer 102 of Figure 1. Each individual distributed software engine 218 provides a particular engine service that may be published to cells 108 via a distributed message exchange platform 216. Virtual objects 110, such as virtual objects A-D, may then obtain their engine services by subscribing to the cells 108 in which they are located, such as cells 1-2.

[0068] As seen in FIG. 4A, each distributed software engine 218 may be a plurality of dynamic and independent software modules designed for one or more specific types of tasks. Each distributed software engine 218 is an independent entity that provides the functionality for which the individual distributed software engine 218 is programmed. Thus, each distributed software engine 218 may be, for example, an artificial intelligence engine 402a, a 3D graphics engine 402b, a simulation engine 402c, a communication engine 402d, an audio engine 402e, a 3D structure engine 402f, a position engine 402g, and a rendering engine 402h. Communication between each distributed software engine 218 is implemented via function calls managed by the distributed message exchange platform 216.

[0069] In some embodiments, multiple distributed software engines 218 can be used simultaneously or sequentially in the network to create synergy with each other. For example, multiple 3D graphics engines 402b hosted on the same or different server computers can allocate engine services sequentially or simultaneously via the distributed message exchange platform 216 to complement each other's engine services for the realization of one or more specific tasks. In one example, different 3D graphics engines 402b calculate 3D models of different locations, where the locations can be in the same or different cells 108. In another example, different 3D graphics engines 402b can calculate different applications located in the same area, where at least some of the applications are configured in the AR layer and at least some of the applications are configured in the VR layer. As an example, the same location in the same cell is calculated by Unity® in the VR layer, while any extensions such as pure virtual objects and applications configured in the AR layer are partially calculated by CryEngine® and UnrealEngine®. Further, in this example, adjacent locations in the same or different cells 108 may be calculated in the VR layer by CryEngine®, while the augmentations in the AR layer are partially calculated by UnrealEngine® and Unity®.

[0070] 4A, virtual objects A-B are currently located within cell 1 of the persistent virtual world system and are receiving multiple engine services, such as AI service 404 provisioned by AI engine 402a, rendering service 406 provisioned by rendering engine 402h, and simulation service 408 provisioned by simulation engine 402c, which may be necessary for the processing and interaction of the current application and virtual objects in current cell 1. Meanwhile, virtual objects C-D are currently located within cell 2 of the persistent virtual world system and are receiving multiple engine services, such as audio service 410 provisioned by audio engine 402e, communication service 412 provisioned by communication engine 402d, and simulation service 408 provisioned by simulation engine 402c, which may be necessary for the processing and interaction of the current application and virtual objects in current cell 2. These engine services may be provisioned by the same server computer hosting the individual distributed software engines 218, by multiple server computers hosting the individual distributed software engines 218, or by a combination thereof.

[0071] FIG. 4B is a schematic diagram 400b illustrating the relationship between the individual distributed software engines 218, hardware engines 220, and distributed 3D engines 212 of one or more servers 102. Upon receiving a request from one or more virtual objects located within a cell 108, a resource manager 214 hosted on one or more server computers 102 receives the request, evaluates the demand, and requests engine services from one or more server computers 102 hosting the software engines and hardware engines 218-220, respectively, via a distributed message exchange platform 216. Each of the engine services may be provided by an individual distributed software engine 218, such as 3D engines 1-N, provisioned by a respective hardware engine 220, such as hardware engines 1-N, which may be a dedicated hardware engine, a general-purpose hardware engine, or a combination thereof. The distributed message exchange platform 216 then receives the respective engine service and publishes the engine service to one or more cells 108 that require the engine service. The virtual object requesting the service then subscribes to the respective cell via the distributed message exchange platform 216 to receive the required engine services.

[0072] 5A-5C are schematic diagrams 500a-500c of relationships between sub-universes (also referred to herein as "verses" for short), streams, and other elements of a persistent virtual world system according to one embodiment.

[0073] Referring to the schematic diagram 500a of FIG. 5A, multiple connected virtual objects 110 that may be used for target-specific simulation may constitute a stream 502, where each stream 502 is configured to obtain data from a corresponding virtual object 110 and perform a target-specific simulation. Each stream 502 may connect to a root node 504 or a root simulation object that generates multiple stream-related virtual objects 110. The stream 502 may further connect to an import function 506 that defines the virtual objects 110 that the stream 502 requires to perform the required simulation. For example, there may be a traffic stream 502 that includes multiple virtual replicas of vehicles, people, traffic lights, and even weather conditions. In other words, the stream 502 includes any number of virtual objects 110 such as virtual replicas that may be related to a particular function of the stream 502. In this example, the import function 506 may define which virtual objects 110 should be considered for the traffic simulation. The stream 502 may then read the behavior of each of the corresponding virtual objects 110 by performing a target-specific simulation of a traffic scenario using a combination of virtual and / or real sensors. For example, the goal may be to optimize traffic streams, perform at a given average speed, or reduce the likelihood of traffic jams. Such simulations may be used, for example, to drive autonomous vehicles or drones. Another example of stream 502 may be energy consumption, which may include a virtual replica of a building that includes power consumption data, and specific energy consuming virtual replicas of machines, computers, appliances, etc.

[0074] 5A , the multiple streams 502 that contain all or most of the virtual objects 110 in a given environment form a sub-universe referred to herein as a berth 508. For example, a city berth 508 may represent all or most of a city's buildings, trees, people, streets, traffic lights, vehicles, etc. In some embodiments, there may also be berths 508 within berths 508, such as residential berths within a city berth, which may include exterior and interior 3D designs of homes, and all or most of the home's appliances, people, walls, energy consumption data, etc.

[0075] 5B, a network of berths 508, such as berths 1-N, can form a persistent virtual world system 112. For example, a city berth can connect to other city berths to form a country berth. If the country berth does not connect to any other berths 508, the country berth can represent a universe of the persistent virtual world system 112.

[0076] Referring to diagram 500c of FIG. 5C, a spatial view of a berth 508 including multiple streams 502, such as streams 1-4, each including multiple stream-related virtual objects 110, is shown.

[0077] 6A-6B are schematic diagrams 600a-600b of a division of a berth 508 into multiple cells 108 according to one embodiment.

[0078] As seen in FIG. 6A, an undivided verse 602 representing a region of the world may be divided into a divided verse 604 including a number of cells 606 as the resource manager requests resources based on requests from the virtual objects 110. More resource-intensive cells 608, such as cells that include a greater number of virtual objects 110, may be further divided into a greater number of cells 606. Similarly, cells 606 with a smaller number of virtual objects 110 may be divided into a smaller number of cells 606. Thus, each cell 606 is assigned resources, with the more resource-intensive cells 608 receiving more resources than the less resource-intensive cells 606. For example, some cells 606 within the more resource-intensive cells 608 include more virtual objects 110 than the remaining cells 606. However, it should be understood that the amount of resources allocated to each cell 606 is related not only to the number of virtual objects 110 in a particular cell 606, but also to the level of interaction within the portion of the persistent virtual world system visible to the user, the distance between the user and network equipment such as servers and / or antennas, the type of application being used by the user, the type of entitlements the user may have depending on their individual user profile or contract, or a combination thereof. However, for simplicity and illustrative purposes, Figures 6A-6B use only virtual objects 110 as a parameter for assessing demand and managing allocation to one or more cells.

[0079] In some embodiments, the data structure representing the real world into cells is an octree data structure, with each cell represented as a voxel in the octree data structure. In some embodiments, the voxels utilized in the octree data structure are sparse voxels for use in arranging larger portions of the real world. In other embodiments, the voxels utilized in the octree data structure are dense voxels for use in arranging smaller portions of the real world. The octree and selected sparse or dense voxels are suitable data structures for representing three-dimensional space in a virtual world, such as a persistent virtual world system, that represents the real world. The octree allows for fast and easy combination of multiple objects, implements relatively simple rendering algorithms, allows for rapid spatial searches to be performed, and enables model generation through virtualization of real objects (e.g., through LIDAR or other image scanning methods). For virtual worlds, including 3D virtual worlds that represent at least a portion of the real world, such 3D partitioning of the world enables three-dimensional based engine services and cloud resource allocation over areas that a user may be viewing and / or interacting with. However, quadtrees may also be utilized when dealing with 2D world cases. In other embodiments, other suitable data structures may be utilized, such as BSP trees, sparse voxel octrees, 3D arrays, kD trees, point clouds, wireframes, boundary representations (B-Rep), spatial domain structure trees (CSG trees), binary trees, and hexagonal structures.

[0080] In some embodiments, the data structure includes a collection of data values, relationships, and functions or operations that can be applied to the data, which defines at least the geometry and coordinates of virtual objects within the persistent virtual world system. For example, the selection of the exact data structure depends on the origin of the data, the precision of the geometry required during rendering, whether the rendering is done in real time or pre-rendered, whether the rendering is performed via a cloud server, a user device, a fog device, or a combination thereof, the particular application in which the persistent virtual world system will be used (e.g., a medical or scientific application may require a higher level of clarity than other types of applications), the storage capacity and therefore the desired memory consumption from the server and user device, etc.

[0081] 6A, after completing the event associated with one or more demands, the resources are returned to the original source or sources. The split berth 604 is then consolidated back to the original number of cells 606, as represented by the curved arrow 610 pointing from the split berth 604 back to the unsplit berth 602. In this manner, the system achieves efficient and dynamic allocation of resources by selecting areas of interest with higher demand and returning resources to their sources when they are no longer needed.

[0082] 6B illustrates an octree data structure 600b. The octree data structure 600b directly represents a volume within a data structure, such as a volume that represents at least a portion of the real world. The main advantages of the octree data structure 600b are that it allows for fast and easy combination of several objects, implements simple rendering, performs quick spatial searches, and allows for model generation by digitizing real objects (e.g., LIDAR or other image scanning).

[0083] The octree data structure 600b is a tree data structure with each internal node having eight child nodes, or octants. As the bath is further divided, further levels of child nodes are generated and more resources are allocated to the further divided cells that require resources. Thus, in the example of FIG. 6B, the octree data structure 600b has generated three levels 612 of child nodes, namely levels 1-3, and a root node 614 that represents the entire bath. The areas of the octree data structure 600b shown in bold represent areas that have virtual objects 110. The higher levels of the bath's octree data structure 600b represent areas that require fewer divisions and therefore fewer resources, for example because there are fewer virtual objects 110 present. In contrast, lower levels that include a greater number of virtual objects 110 require more divisions and therefore more resources.

[0084] FIG. 7 is a schematic diagram 700 of another embodiment of a division of a berth into multiple cells according to one embodiment.

[0085] The schematic diagram 700 shows a split verse 604 made up of three streams 702, streams 1-3, where each stream 702 contains one or more stream-specific virtual objects 110, and each stream 702 is programmed to be enabled or disabled for viewing and interaction on a client device. In a further embodiment, each stream 702 is associated with one or more user applications. Each stream 702 may be configured to obtain data from the corresponding virtual object 110 and perform a thematic and / or goal-specific simulation. Each stream 702 may connect to a root node or root simulation object that generates multiple stream-related virtual objects 110. The streams 702 may further connect to an import function that defines the virtual objects 110 that the stream needs to perform the simulation.

[0086] For example, if a viewer is using a restaurant application, the user may optionally select to enable the restaurant stream (e.g., stream 1) of the split verse 704, such that only the restaurant is shown or highlighted to the user. Similarly, a traffic-related application may be configured to enable a traffic stream (e.g., stream 2) to show or highlight the traffic to the user. In another example, a gaming-related application that includes multiple pure virtual objects may activate a gaming stream (e.g., stream 3) to show or highlight the pure virtual objects associated with the gaming application. In this example, stream 3 may require a larger amount of resources than the other streams 1-2 because it has more virtual objects 110. Thus, by combining all streams 1-3 together, the split verse 704 is obtained. In some embodiments, depending on which streams 702 are activated, the activated streams 702 are viewed by the user as rendered or unrendered media streams output on the client device, where the streams 702 include 3D image data, 3D geometry, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, text data, time data, position data, orientation data, haptic data, lighting data, etc., or a combination of such media content. Thus, each stream 702 may be computed and optionally rendered by one or more of the individual distributed software engines that are provisioned by one or more hardware engines. In some embodiments, one or more of the individual distributed software engines include instructions that, when executed by one or more hardware engines, trigger end-to-end encryption (E2EE) of the stream 702 such that only users interested in a particular stream 702 can receive and decrypt the media content of the stream 702.Such an embodiment may be used, for example, during a video call utilizing an end-to-end streaming service.

[0087] Figure 8 is a schematic diagram of a client device 114, according to one embodiment. Some elements of Figure 9 may be similar to elements of Figures 1-8, and therefore similar or identical reference numbers may be used to indicate those elements.

[0088] The client device 114 of the present disclosure may include operational components such as an input / output (I / O) module 802, a power supply 804, memory 806, a sensing mechanism 808 and transceiver 810 forming a tracking module 812, and a network interface 814, all of which are operatively connected to a processor 816.

[0089] The I / O module 802 is implemented as computing hardware and software configured to interact with a user and provide user input data to one or more other system components. For example, the I / O module 802 may be configured to interact with a user, generate user input data based on the interaction, and provide the user input data to the processor 816 before being transferred over a network to another processing system, such as a server. In another example, the I / O module 802 is implemented as an external computing pointing device (e.g., touch screen, mouse, 3D control, joystick, gamepad, etc.) and / or text input device (e.g., keyboard, dictation tool, etc.) configured to interact with the client device 114. In still other embodiments, the I / O module 802 may provide additional, less, or different functionality to those described above.

[0090] The power source 804 is implemented as computing hardware and software configured to provide power to the client device 114. In one embodiment, the power source 804 may be a battery. The power source 804 may be built into the client device 114 or may be removable from the client device 114, and may be rechargeable or non-rechargeable. In one embodiment, the client device 114 may be repowered by swapping one power source 804 for another. In another embodiment, the power source 804 may be recharged by a cable attached to a charging source, such as a Universal Serial Bus ("USB"), FIREWIRE, Ethernet, Thunderbolt, or headphone cable attached to a personal computer. In yet another embodiment, the power source 804 may be recharged by inductive charging, where an electromagnetic field is used to transfer energy from an inductive charger to the power source 804, where the two are brought into close proximity but do not need to be plugged into each other via a cable. In another embodiment, a docking station may be used to facilitate charging.

[0091] The memory 806 may be implemented as computing hardware and software adapted to store application program instructions and store sensor data captured by the plurality of sensing mechanisms 808. The memory 806 may be any suitable type of memory capable of storing information accessible by the processor 816, including a computer-readable medium or other medium that stores data that can be read with the aid of electronic devices, such as hard drives, memory cards, flash drives, ROM, RAM, DVDs or other optical disks, and other writable and read-only memories. The memory 806 may include temporary storage in addition to persistent storage.

[0092] The sensing mechanism 808 may be implemented as computing hardware and software adapted to obtain various sensor data from the real world and determine / track the position and orientation of the client device 114. The sensing mechanism 808 may include one or more including, but not limited to, one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors (e.g., gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, chromatic sensors, humidity sensors, water sensors, electrical sensors, or combinations thereof. In particular, the sensing mechanism 808 includes one or more inertial measurement units (IMUs), accelerometers, and gyroscopes. The IMU is configured to measure and report the velocity, acceleration, angular momentum, translational velocity, rotational velocity, and other telemetry metadata of the client device 114 by using a combination of the accelerometers and gyroscopes. The accelerometers configured within and / or separate from the IMU may be configured to measure the acceleration of the interacting device, including acceleration due to the Earth's gravitational field. In one embodiment, the accelerometer includes a tri-axis accelerometer capable of measuring acceleration in three orthogonal directions.

[0093] The transceiver 810 may be implemented as computing hardware and software configured to enable the client device 114 to receive radio waves from an antenna and transmit data back to the antenna. In some embodiments, a millimeter wave transceiver may be employed that may be configured to receive millimeter wave signals from the antenna and transmit data back to the antenna when interacting with the immersive content. The transceiver 810 may be a two-way communication transceiver 810.

[0094] In one embodiment, the tracking module 812 may be implemented by combining the capabilities of the IMU, accelerometer, and gyroscope with the position tracking provided by the transceiver 810, and the precise tracking, low latency, and high QOS capabilities provided by the mmWave based antenna may enable sub-centimeter or sub-millimeter position and orientation tracking, which may increase accuracy when tracking the real-time position and orientation of the client device 114. In an alternative embodiment, the sensing mechanism 808 and the transceiver 810 may be combined within a single tracking module device.

[0095] The network interface 814 may be implemented as computing software and hardware for communicatively connecting to a network, receiving computer-readable program instructions from the network transmitted by the server or client device 114, and forwarding the computer-readable program instructions to the memory 806 for storage in the memory 806 for execution by the processor 816.

[0096] The processor 816 may be implemented as computing hardware and software configured to receive and process sensor data. For example, the processor 816 may be configured to send imaging requests, receive imaging data, process imaging data into environmental data or other data, process user input data and / or imaging data to generate user interaction data, perform edge-based (on-device) machine learning training and inference, send server requests, receive server responses, and / or provide user interaction data, environmental data, and content object data to one or more other system components. For example, the processor 816 may receive user input data from the I / O module 802 and may implement application programs stored in the memory 806, respectively. In other examples, the processor 816 may receive sensor data captured from the real world from the sensing mechanism 808 or may receive the precise location and orientation of the client device 114 via the tracking module 812 and may prepare a portion of the data before sending the data to the server for further processing. As an example, the processor 816 may realize some of the steps required during data preparation, including analog or digital signal processing algorithms, such as raw data reduction or filtering of the sensor data before sending the data to the server.

[0097] FIG. 9 is a block diagram of a computer-implemented method 900 for enabling dynamic load balancing across a distributed 3D engine according to one embodiment.

[0098] A method 900 for enabling multi-dimensional 3D engine computing and virtualization-based dynamic load balancing of a virtual or real world begins at step 902 with providing, in a memory of a server computer, a data structure representing at least a portion of the real world to a plurality of cells that store virtual objects forming a persistent virtual world system, where at least some of the virtual objects are virtual replicas of corresponding real-world elements. The method continues at step 904 with providing, in a memory of at least one server computer, a distributed 3D engine including a resource manager and a plurality of individual distributed software engines implemented in a distributed deployment.

[0099] The method then proceeds to step 906, where the demand of each individual cell is calculated by the resource manager based on the current load. The demand further considers one or more of the amount of virtual objects and the level of interaction within the portion of the persistent virtual world system visible to the user avatar. In other embodiments, the demand considers other parameters such as the distance between the user and the network equipment (e.g., server and / or antenna), the type of application being used by the user, or the type of entitlement the user may have depending on the individual user profile or contractual content. Based on the demand, the method continues to rank the cells according to the demand in step 908. Then, based on the ranked demand, the method continues to dynamically allocate resources to one or more more resource-intensive cells in step 910.

[0100] FIG. 10 is a block diagram of a computer-implemented method 1000 further including steps of the method 900 for enabling dynamic load balancing via a distributed 3D engine according to one embodiment.

[0101] According to one embodiment, method 1000 begins in step 1002 by dividing at least a portion of the world's more resource-intensive regions of interest into a greater number of cells based on the calculated demand. Method 1000 then continues in step 1004 by allocating corresponding resources from one or more sources to each cell. After terminating an event associated with one or more demands in the more resource-intensive regions of interest, method 1000 continues in step 1006 by returning the resources to the original source or sources. Finally, method 1000 ends in step 1008 by consolidating the berths back into the original number of cells.

[0102] FIG. 11 is a block diagram of a method 1100 illustrating resource allocation steps via a distributed message exchange platform according to one embodiment.

[0103] The method 1100 may begin with publishing resources corresponding to corresponding cells via a distributed message exchange platform of a resource manager using a publish / subscribe model, at step 1102. The method 1100 then ends with subscribing to cells of interest by one or more virtual objects to obtain the required resources via the distributed message exchange platform, at step 1104.

[0104] In some embodiments, the method 1200 further includes sharing, via a distributed message exchange platform, a dynamically updated state of at least a portion of the world to which one or more client devices and / or server computers may subscribe to obtain an updated version of the world with said client devices, server computers, or a combination thereof. In still further embodiments, the state of at least a portion of the world is obtained by a plurality of connected devices including sensors providing sensor data to the persistent virtual world system, by user input, by server computation, or a combination thereof.

[0105] While particular embodiments have been described and illustrated in the accompanying drawings, it is to be understood that such embodiments are merely illustrative and do not limit the broad disclosure, and that the disclosure is not limited to the specific configurations and arrangements shown and described, since those skilled in the art may recognize various other variations. Thus, the description is to be regarded as illustrative and not limiting.

Claims

1. A system comprising one or more server computers, The one or more server computers include a memory and at least one processor, the memory comprising: a data structure for virtualizing at least a portion of the virtual or real world into a number of cells for storing virtual objects forming a persistent virtual world system; a distributed 3D engine including a plurality of distributed software engines; Equipped with resources are dynamically allocated to the plurality of cells based on a current load of the plurality of cells via a distributed deployment; an original number of cells of the plurality of cells representing one or more regions of at least a portion of the virtual or real world is divided into a larger number of cells; and The resources are restored based on one or more requests. system.

2. The system described in claim 1, wherein at least a portion of the virtual world or real world is integrated into the original number of cells.

3. The system of claim 1, wherein the persistent virtual world system includes a low-fidelity simulation used for demand assessment and load balancing, and a high-fidelity simulation used to improve user experience.

4. The system described in claim 1, wherein at least one of the virtual objects of the persistent virtual world system includes self-computing capabilities and autonomous behavior.

5. The system described in claim 1, wherein the distributed software engines are used simultaneously or sequentially via the distributed deployment to complement each other's engine services for the accomplishment of at least one task.

6. the data structure is an octree data structure, At least one of the plurality of cells is represented as a voxel in the octree data structure; and The system of claim 1 , wherein the voxels representing at least one of the plurality of cells are selected from among sparse voxels and dense voxels based on a size of the at least one of the plurality of cells.

7. The system of claim 1, wherein the data structure includes at least one of a BSP tree, a sparse voxel octree, a 3D array, a kD tree, a point cloud, a wireframe, a boundary representation (B-Rep), a spatial domain construction tree (CSG tree), a binary tree, a hexagonal structure, and combinations thereof.

8. A resource manager of the distributed 3D engine performs the allocation of the resources via a distributed message exchange platform; the distributed messaging platform utilizes a publish / subscribe model; and At least one of the virtual objects subscribes to at least one of the cells in which the resource is published. The system of claim 1 .

9. The distributed message exchange platform shares a dynamically updated state of at least a portion of the virtual world or the real world stored in the memory with one or more client devices or servers; the dynamically updated state is modified via data obtained by one or more of a plurality of connected devices, by user input, by server calculation, or a combination thereof; The plurality of connected devices includes sensors that provide sensor data to the persistent virtual world system. The system of claim 8.

10. Each cell of the plurality of cells includes one or more streams, each stream includes a plurality of stream-specific virtual objects, each stream is programmed to be enabled or disabled for viewing and interaction on a client device; and Each stream is associated with one or more applications. The system of claim 1 .

11. A method of providing, in a memory of at least one server computer, a data structure that virtualizes at least a portion of a virtual or real world into a plurality of cells that store virtual objects that form a persistent virtual world system, at least one of the virtual objects being a virtual replica of a corresponding real world element; b. providing a distributed 3D engine in said memory of said at least one server computer, said distributed 3D engine comprising a plurality of distributed software engines; and c) dynamically allocating resources to individual cells of said plurality of cells based on a current load of said plurality of cells; A method comprising: an original number of cells of the plurality of cells representing one or more regions of at least a portion of the virtual or real world is divided into a larger number of cells; and The resources are restored based on one or more requests. method.

12. The method of claim 11, wherein at least a portion of the virtual world or real world is integrated into the original number of cells.

13. The method of claim 11, wherein the persistent virtual world system includes a low-fidelity simulation used for demand assessment and load balancing, and a high-fidelity simulation used to improve user experience.

14. The method described in claim 11, wherein the distributed software engines are used simultaneously or sequentially via the distributed deployment to complement each other's engine services for the accomplishment of at least one task.

15. The method according to claim 1, further comprising providing an octree data structure as the data structure representing the virtual world or the real world; At least one of the plurality of cells represents a voxel; The voxel representing at least one of the plurality of cells is selected from among sparse voxels and dense voxels based on a size of the at least one of the plurality of cells. The method of claim 11. dividing at least a portion of the one or more regions of the virtual world or the real world into additional cells based on demand for the plurality of cells; allocating resources corresponding to said additional cells; The method of claim 11 further comprising:

17. The step of allocating resources is performed by the following steps: publishing said resources to corresponding cells via a distributed message exchange platform of a resource manager using a publish / subscribe model; and subscribing by said one or more virtual objects to at least one of said plurality of cells to obtain required resources; 12. The method according to claim 11 .

18. A method for providing a virtual object visualization system comprising: viewing and interacting with a plurality of cells, the plurality of cells including a plurality of streams, the plurality of streams including a plurality of stream-specific virtual objects, the plurality of streams being configured to be enabled or disabled for viewing and interacting with a client device; and The method of claim 11 , wherein the at least one stream is associated with one or more user applications.

19. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform the following steps: providing, in a memory of at least one server computer, a data structure virtualizing at least a portion of a virtual or real world into a plurality of cells storing virtual objects forming a persistent virtual world system, at least one of said virtual objects being a virtual replica of a corresponding real world element; b. providing a distributed 3D engine in the memory of the at least one server computer, the distributed 3D engine including a plurality of distributed software engines; and c) dynamically allocating resources to individual cells of said plurality of cells based on a current load of said plurality of cells; configured to run an original number of cells of the plurality of cells representing one or more regions of at least a portion of the virtual or real world is divided into a larger number of cells; and The resources are restored based on one or more requests. A non-transitory computer-readable medium comprising: