Multidimensional 3D engine calculations in virtual or real worlds and virtualization-based dynamic load balancing

Abstract

Provided is a system that enables a distributed 3D engine to perform dynamic load balancing through a virtual world. The system includes one or more server computers, the server computers including a memory and at least one processor, the memory storing a data structure representing at least a portion of a virtual or real world as a plurality of units storing virtual objects. The memory further stores a distributed 3D engine, which includes a resource manager and a plurality of individual software engines implemented in a distributed deployment. Resources are dynamically allocated to one or more units via the distributed deployment based on current load and corresponding computing and sorting requirements. In some embodiments, the requirements also take into account one or more of the number of virtual objects and the level of interaction within a portion of the persistent virtual world system visible to a user avatar. A method thereof is also provided.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 955,247, filed on December 30, 2019, which is incorporated herein by reference. Background Art

[0003] Conventional systems designed to run virtual worlds, especially large - scale multi - player online games (MMOGs), are limited in the way they support a large number of players or entities within the virtual world. Since these virtual worlds typically run on a single physical computer and rely on a single instance of the game or application, the virtual world is supported by the limited system bandwidth and the processor capacity of the server hosting the virtual world. The server hosting the game or simulation engine needs to provide simulation capabilities for each object and player in the virtual world. However, as the number of players and entities increases, the resource requirements also increase, thus reducing the quality of service (QOS) and user experience within the virtual world.

[0004] Other methods include splitting and evenly distributing the map and running a portion of the map on multiple servers, with each server responsible for a corresponding part of the map. Although more resources are available to provide the entire map, this method still has some drawbacks. For example, overcrowding in one area can make this type of geographical partitioning inefficient in a virtual world, and migrations from one area to another provide a discrete view rather than a continuous view for cross - area users due to the need for server switching.

[0005] In addition, current 3D engines used in the development and processing of virtual worlds have many drawbacks that limit the supply capacity of virtual worlds. For example, current typical 3D engines tend to be monolithic, may be centralized and difficult to scale up, and may be platform - dependent. This hinders the possibility of fully utilizing the capabilities of each specific module of the 3D engine during the development and execution of virtual worlds, further increasing the aforementioned drawbacks. Summary of the Invention

[0006] This Summary of the Invention is provided to introduce a selection of concepts in a simplified form that will be further described in the Detailed Description below. This Summary of the Invention is not intended to identify key features of the claimed subject matter nor is it intended to be used to help determine the scope of the claimed subject matter.

[0007] The present disclosure generally relates to computer systems and methods, and more particularly to systems and methods for implementing multi - dimensional 3D engine computations and virtualization - based dynamic load balancing of virtual or real worlds.

[0008] The system of the present disclosure includes one or more server computers, the server computers including a memory and at least one processor, the memory storing a data structure that virtualizes at least one part of a virtual or real world into a plurality of units that store virtual objects that form a persistent virtual world system. In some embodiments, at least some of the virtual objects are virtual copies of corresponding real-world elements. The memory further stores a distributed 3D engine implemented in a distributed deployment, the distributed 3D engine including a resource manager and a plurality of individual distributed software engines. Resources are dynamically allocated to one or more units via the distributed deployment based on the current load and the corresponding computing and sorting requirements of the one or more units. In some embodiments, the requirements are at least partially based on the number of virtual objects within the one or more units or the level of interaction within the part of the persistent virtual world system visible to a user avatar within the one or more units, or a combination thereof. In some embodiments, resources include computing power, memory, network resources (e.g., bandwidth), or a combination thereof.

[0009] In other embodiments, the requirements are further based on the distance between a user device accessing the one or more units and network devices (e.g., servers and network antennas), the type of application the user is using, or the type of rights the user accessing the one or more units may have, e.g., according to their personal user profile or contract details, or a combination thereof.

[0010] In some embodiments, the virtual objects further include purely virtual objects not available in the real world, virtual avatars of real people, and applications.

[0011] In some embodiments, the persistent virtual world system includes a plurality of simulations of the same virtual or real world, the simulations varying from high fidelity to low fidelity, where the low-fidelity simulations are used for demand assessment and load balancing, and where the high-fidelity simulations are used to enhance the user experience. This enables different simulations to be performed depending on the situation and circumstances. LOD management also allows for an optimized user experience to be provided based on specific requirements and context. Thus, the same virtual or real world can be simulated at different LODs, contributing to the load balancing method of the present disclosure while providing an appropriate experience to the user.

[0012] In some embodiments, virtual objects of the persistent virtual world system include data and models, where the data and models refer to any graphical, mathematical, or logical representation of aspects of the virtual object or the corresponding real object. In some embodiments, the data and models provide self-computation capabilities and autonomous behavior for entities of the persistent virtual world system. In some embodiments, suitable models include one or more of 3D models, geometric models, dynamic models, and machine learning models. Virtual objects can be fed data obtained from simulations, data input by users via user devices, or data captured by sensor devices (e.g., for a real-world-based persistent virtual world system), such as data captured by Internet of Things devices. A continuous data stream can maintain periodic updates of the persistent virtual world system.

[0013] In some embodiments, the persistent virtual world system can include a virtual world layer, which can be separated into an augmented reality (AR) layer and a virtual reality (VR) layer. The separate layers can enable access to the persistent virtual world system in either augmented reality or virtual reality, combine elements stored in each layer, and can be enabled via a client device connected to at least one server computer over a network when accessing one or the other type of reality. Each layer can include enhancements of reality and virtual objects, which can be specific to each layer.

[0014] In some embodiments, a separate distributed software engine can be a distributed 3D engine that is decomposed into multiple dynamic and independent software modules that can communicate with each other when configured locally or with separate software modules located remotely. In some embodiments, multiple distributed software engines can be used in parallel or sequentially within a network to create synergies with each other. For example, multiple 3D graphics engines hosted on the same or different server computers can allocate engine services sequentially or in parallel via a distributed message exchange platform to complement each other's engine services for implementing one or more specific tasks.

[0015] In some embodiments, the one or more processors are specialized processing units, general-purpose processing units, or a combination thereof, configured to process tasks from corresponding individual distributed software engines in hardware. In an embodiment, contrary to typical systems that employ one or more standard central processing units (CPUs) and graphics processing units (GPUs), each of the specialized hardware engines is customized to process a separate distributed software engine for at least one of 3D graphics, artificial intelligence operations, physical operations, 3D structure processing operations, simulation operations, tracking and positioning, and communication. In some embodiments, each hardware engine may be coupled as a chip or a system-level electronic chip, the chip including one or more specific hardware processing cores designated for processing each specific type of operation. The processing cores of one or more electronic chips may be regarded as functional units or modules of the electronic circuits of the electronic chip, which is designed to physically execute the algorithms required for each 3D engine. For example, at least one of the one or more processing cores may be implemented as an FPGA or any other form of integrated circuit implementing specialized functionality. Thus, the electronic chip includes at least one integrated circuit implementing at least some of the one or more processing cores, wherein at least one of the processing cores implements different specialized hardware engines. These electronic chips may be installed, for example, on any one or both of one or more external servers or a primary server. In some embodiments, the electronic chip may additionally include general-purpose processing cores, such as CPUs and / or GPUs, or may be connected to external CPUs and / or GPUs to provide them with additional support for each engine task. In addition to enabling each 3D engine module to have one or more dedicated hardware cores from one or more hardware engines, providing dedicated hardware engines at each source and configuring each 3D engine module in a distributed and independent manner can provide more resources to the entire system.

[0016] In some embodiments, the data structure representing the virtual or real world as units is an octree data structure, where each unit is represented as a voxel within the octree data structure. In some embodiments, the voxels used in the octree data structure are sparse voxels for the layout of a larger portion of the real world. In other embodiments, the voxels used in the octree data structure are dense voxels for the layout of a smaller portion of the real world. The octree and the selected sparse or dense voxels are suitable data structures for representing three-dimensional space in a virtual world, such as in a persistent virtual world system representing the real world. The octree can quickly and easily combine multiple objects, implement relatively simple rendering algorithms, perform fast spatial searches, and generate models through the virtualization of real objects (e.g., via LIDAR or other image scanning methods). For a virtual world including a 3D virtual world representing at least a portion of the real world, such 3D partitioning of the world enables three-dimensional-based engine services and cloud resource allocation on the areas where the user may be viewing and / or interacting.

[0017] In some embodiments, the units representing higher resource-intensive regions of interest from the at least one portion of the world are further divided into a greater number of units. In other embodiments, each high-resource-intensive unit is correspondingly assigned resources from one or more sources, where the smaller units are allocated a greater amount of resources. In another embodiment, after an event associated with one or more requests is completed, the resources are restored from multiple sources to the original one or more sources. Subsequently, at least one portion of the world is merged back into the original number of units. In this way, the system achieves efficient and dynamic allocation of resources by selecting regions of interest with higher demand.

[0018] In some embodiments, the resource manager performs the allocation through a distributed message exchange platform. In other embodiments, the distributed message exchange platform utilizes a publish-subscribe model, where one or more virtual objects subscribe to one or more units in which resources are published. In yet another embodiment, the distributed message exchange platform shares with one or more of the client devices and the server the dynamic update state of at least one portion of the world stored in the memory. Thus, the resource manager of the distributed 3D engine of the present disclosure publishes each resource once to the distributed message exchange platform and is delivered to each virtual object that needs them by subscribing to the units with the published resources. This is in contrast to typical 3D engine systems customized for supplying virtual worlds, which output a complete data update package for each client device within the map, customize each package for the client device such that the client device only receives data that may be relevant to the client device and the visible objects, resulting in the game or simulation engine possibly sending the same data multiple times for each client device where the object is visible.

[0019] According to an embodiment, the state of at least a portion of the world is further updated by a plurality of connection devices, the connection devices including sensors that provide sensor data to the persistent virtual world system. The connection devices can be, for example, Internet of Things (IoT) devices that can communicate with each other and with a server over a network and can continuously capture multi-source sensor data from the real world in order to update a virtual copy of the persistent virtual world system. Thus, the state update may be caused by sensor data. However, the state update can also be caused by user input or by computations (e.g., artificial intelligence inference) from one or more 3D engines. Then, the state update of the world can be published on a distributed message exchange platform for all client devices and sources to subscribe to and view on demand.

[0020] In some embodiments, each of the plurality of units includes one or more streams, each stream including one or more stream-specific virtual objects and being configured to be enabled or disabled for viewing and interaction on the client device. In another embodiment, each stream is associated with one or more user applications. Each stream can be configured to obtain data from the corresponding virtual objects and perform simulations specific to a theme and / or a target. Each stream can be connected to a root node or a root simulation object that generates a plurality of stream-related virtual objects. The stream can additionally be connected to an import function that defines the virtual copy required for the stream to perform the simulation. In some embodiments, in a predetermined environment, such as in one unit or multiple units, the plurality of streams including all or most of the virtual copies, their context, and the relationships and interactions between each virtual copy form a universe or a sub-universe. The network of sub-universes can form the universe of the persistent virtual world system. For example, the sub-universe of a city can be called a city universe, which can represent all the buildings, trees, people, streets, traffic lights, vehicles, etc. of a city and the relationships, interactions, and environment between them. There may also be another sub-universe within the sub-universe, such as a house universe within the city universe. The house universe can include the 3D design of the exterior and interior of the house, all the household appliances, people, walls, energy consumption data, etc. of the house. The network of all sub-universes can form the universe of the persistent virtual world system, which can be viewed in virtual or augmented reality. Each of these sub-universes can be dynamically divided into a plurality of units in which resources are dynamically allocated.

[0021] In embodiments of the present disclosure, the system can be implemented in a cloud-to-edge infrastructure that can display distributed computing capabilities, including using public or private clouds, fog servers, peer-to-peer cloud server systems, distributed ledger-based infrastructures, micro-clouds, and edge systems such as enterprise systems, mobile platforms, and user devices. The distributed ledger-based infrastructure can be a decentralized and immutable blockchain or distributed ledger network that facilitates the transfer and storage of data necessary for a widely distributed persistent virtual world system, including virtual copies, purely virtual objects, applications, and any events or interactions within the persistent virtual world system. Through the cloud-to-edge infrastructure, resources including physical servers and network devices achieve shared storage and computing, and the shared storage and computing can allow for its dynamic allocation based on factors such as the distance of users to resources and the network, and the computing requirements of users relative to each other and their locations, thereby commanding more or less computing accordingly. In some embodiments, the dynamic allocation of resources is incorporated into smart contracts using the distributed ledger infrastructure, automating transaction logic (e.g., the computer logic behind each storage and computing allocation) and the consequences or results of such allocations.

[0022] According to an embodiment, a method for enabling a distributed 3D engine to perform dynamic load balancing through a virtual world includes virtualizing a data structure in the memory of at least one server computer, the data structure representing at least a portion of the virtual or real world as a plurality of units storing virtual objects that form a persistent virtual world system, where at least some of the virtual objects are virtual copies of corresponding real-world elements. The method continues by providing, in the memory of at least one server computer, a distributed 3D engine implemented in a distributed deployment, the distributed 3D engine including a resource manager and a plurality of individual distributed software engines. Subsequently, the method continues by the resource manager calculating the demand for individual units among the plurality of units based on the current load, where the demand is at least partially based on the number of virtual objects within the individual unit or the level of interaction within the portion of the persistent virtual world system visible to user avatars within the individual unit. Then, the method continues by sorting the individual units according to the demand. Finally, based on the sorted demand, the method ends by dynamically allocating resources to the individual units (e.g., one or more higher resource-intensive units).

[0023] According to an embodiment, the method further includes: dividing an area of interest of at least a part of a virtual or real world into additional units based on a calculated demand; and assigning corresponding resources to the additional units (e.g., each unit representing a resource-intensive area of interest). In an embodiment, the method further includes, after an event associated with one or more requests within a higher resource-intensive area of interest ends, restoring the resources to the original one or more sources; and merging at least a part of the world within the persistent virtual world system back into the original number of units.

[0024] In some embodiments, the allocation is performed by publishing resources to the corresponding units using a publish-subscribe model through a distributed message exchange platform of a resource manager; and one or more virtual objects subscribe to the units of interest to obtain the required resources.

[0025] In some embodiments, the method further includes sharing a dynamic update status of at least a part of the world with one or more client devices and / or server computers through a distributed message exchange platform, where the client devices and / or server computers have subscribed to the dynamic update status to obtain an updated version of the world. In yet another embodiment, the status of at least a part of the world is obtained through multiple connected devices, through user input, through server computing, or a combination thereof, where the connected devices include sensors that provide sensor data to the persistent virtual world system.

[0026] The above summary of the invention does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be practiced with all suitable combinations of the aspects outlined above, as well as those systems and methods disclosed in the following detailed description and specifically pointed out in the claims filed together with this application. Such combinations have special advantages not specifically recited in the above summary of the invention. Other features and advantages of the present disclosure will be apparent from the drawings and from the detailed description below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The specific features, aspects, and advantages of the present disclosure will be better understood with reference to the following description and drawings, in which:

[0028] Figure 1 A schematic diagram of a system for implementing multi-dimensional 3D engine calculations and virtualization-based dynamic load balancing of a virtual or real world according to an embodiment is shown.

[0029] Figure 2 A schematic diagram of a server of a system for implementing multi-dimensional 3D engine calculations and virtualization-based dynamic load balancing of a virtual or real world according to an embodiment is shown.

[0030] Figure 3Shows a schematic diagram of resource allocation through a distributed message exchange platform according to an embodiment.

[0031] Figures 4A - 4B Shows a schematic diagram of engine service allocation through a distributed message exchange platform according to an embodiment.

[0032] Figures 5A - 5C Shows a schematic diagram of the relationships between elements of a persistent virtual world system according to an embodiment.

[0033] Figures 6A - 6B Shows a schematic diagram of partitioning into multiple units according to an embodiment.

[0034] Figure 7 Shows a schematic diagram of another embodiment of partitioning into multiple units according to an embodiment.

[0035] Figure 8 Shows a schematic diagram of a client device according to an embodiment.

[0036] Figure 9 Shows a block diagram of a computer-implemented method for implementing multi-dimensional 3D engine calculations in a virtual or real world and virtualization-based dynamic load balancing according to an embodiment.

[0037] Figure 10 Shows a block diagram of other steps of a computer-implemented method including implementing multi-dimensional 3D engine calculations in a virtual or real world and virtualization-based dynamic load balancing according to an embodiment.

[0038] Figure 11 Shows a block diagram of the steps of resource allocation through a distributed message exchange platform according to an embodiment. Detailed Embodiments

[0039] In the following description, reference is made to the accompanying drawings which illustrate various embodiments by way of illustration. Moreover, various embodiments will be described by referring to several examples. It should be understood that embodiments may include design and structural changes without departing from the scope of the claimed subject matter.

[0040] Figure 1 Shows a schematic diagram of system 100 for implementing multi-dimensional 3D engine calculations in a virtual or real world and virtualization-based dynamic load balancing according to an embodiment.

[0041] System 100 includes multiple server computers 102 (e.g., external servers A - C) that implement a distributed 3D engine via a network 104. The distributed 3D engine includes a resource manager in a distributed deployment 106. The distributed deployment 106 is a software implementation of the distributed 3D engine across multiple server computers 102 using a distributed architecture such as that of system 100. The server computers 102 use a publish - subscribe model to dynamically allocate resources via the resource manager to one or more of multiple units 108 (e.g., units 1 - 3), where each unit 108 virtually represents one or more different regions of the real world or a virtual world. The allocation is based on the current load and the corresponding computing and sorting requirements. In some embodiments, the requirements also consider one or more of the number of virtual objects 110 and the level of interaction within the portion of the persistent virtual world system visible to the user avatar, as described in more detail below.

[0042] One or more of the multiple units 108 include virtual objects 110, and the multiple virtual objects 110 within a unit 108 form a persistent virtual world system 112. In some embodiments, at least some of the virtual objects 110 are virtual copies of corresponding real - world elements. In some embodiments, the requirements for an individual unit consider the number of virtual objects 110 in the individual unit, the level of interaction within the portion of the persistent virtual world system visible to the user via the corresponding client device 114, or a combination thereof, as described in more detail below. In other embodiments, the requests consider other parameters, such as the distance between the client device 114 and network devices such as servers and / or antennas, the type of application the user is using, or the type of rights the user may have according to their personal user profile or contract details, as described in more detail below.

[0043] In the present disclosure, the term "persistent" is used to characterize the state of a system that can continue to exist without a continuously executing process or network connection. For example, the term "persistent" can be used to characterize a virtual world system where the virtual world system and all the objects included therein continue to exist after the process used to create the virtual objects has stopped and independently of the users connected to the virtual world system. Thus, the virtual world system is stored in a non - volatile storage location (e.g., on a server). In this way, virtual copies, purely virtual objects, and applications can interact and cooperate with each other when configured to achieve a specific goal, even when the user is not connected to the server.

[0044] In some embodiments, to reduce hardware and network requirements, help reduce network latency, and improve the general mixed reality experience, the system can be connected via network 104, including a combination of millimeter wave (mmW) or mmW and sub-6 GHz communication systems, such as via fifth-generation wireless system communication (5G). In other embodiments, the system can be connected via a wireless local area network (Wi-Fi). The provided communication system can allow for low (e.g., about 1 to about 5 milliseconds) latency to the in-field endpoint and high (e.g., 1 - 10 Gbps) downlink speeds, meeting the parameters necessary to execute typical highly interactive digital reality applications or other high-demand applications. This results in high-quality, low-latency, real-time streaming of digital application content. In other embodiments, the system can communicate via fourth-generation wireless system communication (4G), can be supported by a 4G communication system, or can include other wired or wireless communication systems.

[0045] In other embodiments, the Global Navigation Satellite System (GNSS) can be used to implement the positioning of the device. The Global Navigation Satellite System refers to satellite-based navigation systems such as GPS, BDS, Glonass, QZSS, Galileo, and IRNSS. Using signals from a sufficient number of satellites and techniques such as triangulation and trilateration, GNSS can calculate the position, speed, altitude, and time of the device. In embodiments, the external positioning system is enhanced by Assisted GNSS (AGNSS) through the architecture of an existing cellular communication network, where the existing architecture includes 5G. In other embodiments, the AGNSS tracking system is further supported by a 4G cellular communication network. In indoor embodiments, GNSS is further enhanced via radio wireless local area networks such as Wi-Fi, preferably but not limited to providing 60 GHz data. In alternative embodiments, GNSS is enhanced by other techniques known in the art, such as via Differential GPS (DGPS), Satellite-Based Augmentation System (SBAS), Real-Time Kinematic (RTK) systems. In some embodiments, the tracking of the device is achieved through a combination of AGNSS and inertial sensors in the device.

[0046] In some embodiments of the present disclosure, system 100 may be implemented in a cloud-to-edge infrastructure that may use computing devices and systems to exhibit distributed computing capabilities, such as public or private clouds, fog servers, micro-clouds, peer-to-peer cloud server systems, distributed ledger-based infrastructures, and edge devices and systems, such as enterprise systems, mobile platforms, and user devices, all of which may be connected via network 104. Using a cloud-to-edge computing network, access to computing capabilities, computer infrastructure (e.g., via so-called infrastructure as a service, or IaaS), applications, and business processes may be delivered as a service to users on demand via client device 114. In this way, resources, including physical servers and network devices, can enable shared storage and computing that can be dynamically allocated. In some embodiments, client device 114 includes user devices and other devices. User devices may include, for example, mobile devices, personal computers, gaming consoles, media centers, head-mounted displays, and see-through devices (e.g., digital reality contact lenses). Other devices may be specifically configured to capture multi-source sensor data from real-world objects via sensor mechanisms to update and enrich virtual copies of the corresponding real-world objects. According to embodiments, the other devices are one or more surveillance cameras, vehicles, traffic lights, buildings, streets, train tracks, household appliances, etc.

[0047] Figure 2 A schematic diagram of server 102 of system 100 implementing multi-dimensional 3D engine calculations and virtualization-based dynamic load balancing of a virtual or real world is shown according to an embodiment.

[0048] One or more servers 102 include one or more processors 202 and a memory 204 that stores a data structure that virtualizes at least a portion of a virtual or real world into a plurality of units (e.g., Figure 1 units 108), each unit including a virtual object 110, forming a persistent virtual world system 112. In persistent virtual world system 112 based on real-world embodiments, some virtual objects 110 are virtual copies 206 of corresponding real-world elements. In other embodiments, virtual objects 110 also include applications 208 and pure virtual objects 210 that do not exist in the real world. Memory 204 also stores a distributed 3D engine 212 that includes a resource manager 214 that manages resources. Resource manager 214 allocates resources, including computing power, memory, and network, by publishing resources on one or more units to which virtual objects may subscribe using a distributed message exchange platform 216 that uses a publish-subscribe model. Resource manager 214 may act as a virtual layer that interfaces with hardware and software components that manage resources. Management actions may include the evaluation, provisioning, scaling, and replication of resources via distributed message exchange platform 216.

[0049] In the current embodiment, the term "virtual copy" refers to an exact and persistent virtual representation of a real-world element. In an embodiment, the virtual copy 206 of the present disclosure refers to a virtual version or virtual twin of a real-world element or real twin, which can be mapped or modeled by computer-aided drawing (CAD) or computer-aided engineering (CAE) methods, input through a computer model of a real object (e.g., building information model - BIM) or other suitable existing technology methods, and which may reflect not only the appearance but also the behavior of the real twin. Some real-world objects may additionally include sensors that can provide multi-source input data to the virtual copy 206 for enriching the virtual copy 206 and synchronizing it with their respective real counterparts.

[0050] In some embodiments, the virtual copy 206 includes one or more of 3D world and building data, such as data based on SLAM or derivative mapping; 3D geometric data; 3D point cloud data; or geographic information system data representing the structural properties of the real world, which can be used to model 3D structures for applications.

[0051] In the present disclosure, the term "virtualization" refers to the action of creating an abstract virtual version or persistent virtual world system 112 of at least a part of the virtual or real world. For real-world situations, virtualization refers to creating virtual versions of elements found in the real world and any resources of the real world (e.g., computing power, memory, or network) and associated devices, such that a user (e.g., a human user, a computer program, or a machine) can interact with the real-world elements through the persistent virtual world system 112. In this way, the user can feel that the interaction occurs directly with the real-world elements. For example, a human user can interact through the user interface of a user device (e.g., a mobile phone or a head-mounted display device) with elements of a real city in a virtual city in augmented reality or virtual reality, where multiple elements of the real city have been virtualized in the persistent virtual world system 112. For physical manipulation of the real world through a virtual copy, electromechanical devices can be installed on real-world objects to convert remote commands sent to the real-world objects through the virtual copy into physical actions from the real-world elements. Regardless of whether the persistent virtual world system 112 is based on a virtual world or a real world, a virtual object may also include a model, which refers to any graphical, mathematical, or logical representation of an aspect of the virtual object or the corresponding real object. In some embodiments, such models provide self-computing capabilities and autonomous behaviors for entities of the persistent virtual world system 112. In some embodiments, suitable models include one or more of 3D models, geometric models, dynamic models, and machine learning models.

[0052] The distributed 3D engine 212 of the present disclosure refers to a distributed version of a 3D engine implemented in a distributed deployment (e.g., in multiple servers or client computers). The distributed 3D engine 212 is configured to virtually pool, allocate, provide, and dynamically load-balance and distribute resources (e.g., computing power, memory, and network) through a resource manager 214. The connection to and access of the distributed 3D engine 212 to the persistent virtual world system 112 enables accurate load balancing by evaluating the demand in each unit representing at least a portion of the real or virtual world using spatial computing principles. As part of resource provisioning, the distributed 3D engine 212 provides multiple engine services (e.g., 3D rendering, graphics, physics, audio, artificial intelligence, positioning, and communication) on demand through multiple individual distributed software engines. The hardware operations performed to provide each of the mentioned engine services may be provided by multiple corresponding hardware engines. As described above, such resource pooling is achieved through virtualization. Thus, the concept of the distributed 3D engine 212 of the present disclosure should not be simply construed as an aggregation of functionality abstracted in a library that can be reused between different applications (e.g., video games), but includes a distributed hardware and software implementation of engine operations that pool, allocate, provide, and load-balance using spatial computing principles through the resource manager 214 with the persistent virtual world system as the basis for such operations.

[0053] In the present disclosure, the term "spatial computing" refers to the use of physical space in order to send inputs and receive outputs from a computer. Thus, spatial computing uses the geometric properties of virtual objects and their position and orientation in order to perform the required computations and dynamic load balancing.

[0054] The virtual object 110 can be fed data obtained from simulations, data input by a user through a user device, or data captured by a sensor device (e.g., for a real-world-based persistent virtual world system), such as data captured by an Internet of Things device. A continuous data stream can maintain periodic updates of the persistent virtual world system 112. Thus, the persistent virtual world system 112 of at least a portion of the real world is not merely a visual 3D representation of the real world, as is the case with many current virtual world systems, but a multi-dimensional representation of the real world, since the virtual object 110 includes models and data from other dimensions or aspects of the real world. For example, multiple virtual objects 110 can include data related to temperature, fuel consumption, speed, power, pressure, mass, sound, etc., and also implement connections and communications between each virtual object.

[0055] In some embodiments, virtualization of the virtual or real world includes performing multiple simulations of the same world at multiple levels of detail (LOD). LOD management improves the efficiency of computational processes, such as the rendering process, by reducing the workload (usually vertex transformations) used by the graphics pipeline or by enhancing physical simulations, since different physical models can be associated with the virtual replicas, from low-fidelity models to high-fidelity models. This enables different simulations to be performed depending on the situation and circumstances. LOD management also allows for an optimized user experience to be provided based on specific requirements and context. Thus, the same virtual or real world can be simulated at different LODs, contributing to the load balancing method of the present disclosure while providing an appropriate experience to the user. For example, the real-world simulation in the persistent virtual world system 112 can include a high-fidelity simulation of at least a portion of the real world for human user visualization purposes, and a lower-fidelity simulation for assessing the demand of each unit and contributing to the load balancing of the high-fidelity simulation. Thus, the lower-fidelity simulation can help to see which areas may be overloaded and which areas are not, while using fewer resources. In another embodiment, in addition to data and models related to the current energy consumption, productivity, efficiency, etc. of each machine in the manufacturing plant, due to the precise spatial representation and relationships between objects, the virtualization of the manufacturing plant can achieve precise management of the manufacturing plant through a distributed 3D engine 212, including demand assessment and corresponding resource allocation. For the purpose of managing the manufacturing plant (e.g., controlling the production process and all elements in the factory), the manufacturing plant can be simulated at a high fidelity, and for the purpose of managing the load balancing of the manufacturing plant, it can also be simulated at a low fidelity. The same LOD principle described above can be applied to the persistent virtual world system 112 based on a virtual world, where a low-fidelity version of the virtual world can be simulated within the persistent virtual world system 112 for the purpose of load balancing of the virtual world, while a high-fidelity version of the virtual world can be simulated to provide a better user experience.

[0056] "Self-computing ability", also known as "self-management ability", herein refers to the ability to apply artificial intelligence algorithms to autonomously manage computing resources. In an implementation, a virtual replica with self-computing ability in a persistent virtual world system can autonomously manage computing resources to adapt to changes in the environment of a corresponding real-world element or the real-world element itself. The rules and conditions for self-management can be further controlled by using smart contracts running on a blockchain or distributed ledger technology, and the rules and conditions can be further encoded in a distributed and transparent manner. Thus, in an implementation, each virtual object, such as a virtual replica of a real object, can exhibit autonomous behavior and act autonomously according to the conditions in the real world reflected in the persistent virtual world system (e.g., by allocating required resources, autonomously sending and executing commands, and / or generating events required for each environment). Implementing this type of behavior may require training virtual objects with artificial intelligence algorithms during virtual object modeling.

[0057] The virtual replica can obtain data from one or more sources (e.g., from one or more real-world objects, environmental sensors, computing devices, etc.). As used in this disclosure, the term "multi-source data" refers to data that can be obtained from multiple sources. The virtual replica can be updated based on multi-source data, such as by updating known parameters or features, by enriching the virtual replica with additional parameters or features, etc.

[0058] In this disclosure, the term "enrich" is used to describe the action of providing additional properties to a virtual replica based on multi-source sensor data. For example, enriching a virtual replica can refer to providing real-world data captured from a sensor mechanism on a client device, where the other real-world data includes video data, temperature data, real-time energy consumption data, real-time water consumption data, speed or acceleration data, etc.

[0059] In some implementations, some virtual replicas 206 can be virtual avatars of real users accessing the persistent virtual world system 112 via a client device. In these implementations, resources are published on the corresponding units where the user's virtual avatar is active, e.g., corresponding to the location and orientation that the user may view through their client device, such that the resources are received by the user through the corresponding virtual avatar subscribing to the corresponding unit. Similarly, when an application 208 and / or a pure virtual object 210 is active in a unit, these virtual objects 110 subscribe to the unit, triggering the resource manager 214 to evaluate the requirements and publish more or fewer resources that the virtual objects 110 can obtain by subscribing to the corresponding unit accordingly.

[0060] For example, a particular traffic signal in a city is associated with a road or street and can also be associated with other objects, such as sensors or other traffic signals. These relationships can trigger events that can create new interactions in the persistent virtual world system 112. For example, an ambulance in the street can automatically trigger a priority event that will change the traffic signal pattern based on a simulation of the ambulance's trajectory and the actual traffic conditions in the area to avoid any stops at intersections. Resource management during such events, including the provisioning of devices (e.g., cars) involved in such events, is performed by the resource manager 212. In some embodiments, such conditions, events, and outcomes can be programmed as smart contracts bound to a traffic-specific distributed ledger, e.g., a traffic sidechain with many smart contracts that define various encoded traffic rules and expected behaviors, thus automating and recording the occurrence of such events.

[0061] In some embodiments, the pure virtual object 210 refers to an object that exists only in the persistent virtual world system 112 and thus has no corresponding object in the real world.

[0062] In some embodiments, the application 208 can be one or more traditional applications, distributed applications, or decentralized applications. Traditional applications are typically based on the traditional client-server model and run on dedicated servers in a static infrastructure. Distributed applications are applications that are primarily stored on a cloud computing platform, such as the cloud server of the present disclosure, and can run simultaneously on multiple systems and devices on the same network or can run on a blockchain or a distributed ledger-based distributed database. Decentralized applications primarily run on a decentralized infrastructure, such as a blockchain or a distributed ledger-based distributed database.

[0063] In some embodiments, the persistent virtual world system 112 may include a virtual world layer that may be separated into an augmented reality (AR) layer and a virtual reality (VR) layer. The separate layers may enable access to the persistent virtual world system in either augmented reality or virtual reality, combine elements stored in each layer, and may be enabled via a client device connected to at least one server computer 102 over a network when accessing one or the other type of reality. Each layer may include an augmentation of reality and virtual objects 110 that may be specific to each layer. For example, the AR layer may include only virtual replicas 206, applications 208, and purely virtual objects 210 that may be visible only when accessing the persistent virtual world system 112 in augmented reality. Similarly, the VR layer may include only virtual replicas 206, applications 208, and purely virtual objects 210 that may be visible only when accessing the persistent virtual world system 112 in virtual reality. However, a user may decide to enable both layers or adjust the levels of each layer such that at least some objects of one layer are visible in the other layer. For example, when accessing a scene in virtual reality, it may be possible to enable the augmented reality layer such that an augmentation of a real object virtualized in the VR layer can be viewed, such as viewing a further description of a building or a tourist location.

[0064] In some embodiments, the distributed 3D engine 212 also includes one or more separate distributed software engines 218. The distributed software engine 218 may be a 3D engine broken into multiple dynamic and independent software modules, each software module being programmed to perform a specific engine task. The separate distributed software engines 218 may communicate with each other via, for example, a distributed message exchange platform 216. Each individual distributed 3D engine 218 may use the message exchange platform 216 to invoke service functions in other separate distributed software engines 218. Such invocations may be performed indirectly (e.g., via a general proxy function invoked by the distributed message exchange platform 216) or directly (e.g., via the distributed message exchange platform 216 that maintains direct communication with each individual distributed 3D engine 218).

[0065] In some embodiments, one or more processors 202 can be one or more dedicated hardware engines 220. In other embodiments, one or more processors 202 are general-purpose processing units, or are dedicated processing units supported by general-purpose processing units. In some embodiments, the dedicated processing unit is a processing unit customized to process one or more individual distributed software engines 218 to perform engine tasks. The dedicated hardware engine 220 can thus be individually designed and configured to process operations related to one or more of 3D graphics, artificial intelligence operations, physical operations, 3D structure processing operations, simulation operations, tracking and positioning, or their communication. In some embodiments, each hardware engine 220 can be coupled on one or more electronic chips or system-on-chips, which include one or more specific hardware processing cores designated for processing each specific type of operation. The processing cores of one or more electronic chips can be regarded as functional units or modules of the electronic circuits of the electronic chips, which are designed to physically execute the algorithms required by each individual distributed software engine 218. For example, at least one of the one or more processing cores can be implemented as an FPGA or any other form of integrated circuit implementing dedicated functionality. Thus, one or more electronic chips can include at least one integrated circuit implementing at least some of the one or more processing cores, where at least one of the processing cores implements different dedicated hardware engines 220. In some embodiments, the electronic chips can additionally include general-purpose processing cores, such as CPUs and / or GPUs, or can be connected to external CPUs and / or GPUs to provide them with additional support for each engine task. Since each individual distributed 3D engine 218 can obtain resources from one or more dedicated hardware cores of one or more hardware engines 220, providing the dedicated hardware engines 220 enables each individual distributed software engine 218 to have greater resource attractiveness for performing a larger number of tasks within the persistent virtual world system.

[0066] The processor 202 can refer to a single dedicated processor, a single shared processor, or multiple individual processors, some of which can be shared. Additionally, the explicit use of the term "processor" should not be construed as specifically referring to hardware capable of executing software, and can implicitly include, but is not limited to, digital signal processor (DSP) hardware, network processors, application-specific integrated circuits (ASICs), and FPGAs, etc.

[0067] The memory 204 can be of any suitable type capable of storing information accessible by the processor 202, including computer-readable media, or other media that store data readable by an electronic device, such as a hard disk drive, a memory card, a flash drive, ROM, RAM, a DVD, or other optical discs, and other writable and read-only memories. In addition to persistent storage, the memory can include transient storage. Instructions can be executed directly by the processor 202 (e.g., machine code) or indirectly (e.g., scripts). Instructions can be stored in object code format for direct processing by the processor 202, or in any other computer language, including a collection of scripts or stand-alone source code modules that can be interpreted on demand or pre-compiled. The processor 202 can retrieve, store, or modify data according to the instructions. For example, data can be stored in computer registers, in a relational database, as a table with multiple different fields and records, an XML document, or a flat file, etc. Data can also be formatted in any computer-readable format.

[0068] In some embodiments, the interaction mechanisms between and with the virtual object 110 are defined by using computer code included in computer scripts and computer programs, and can be implemented by an application, a smart contract, or a combination thereof provided in the persistent virtual world system 112 where the interaction and interaction applications occur. Thus, the application, the smart contract, or a combination thereof enables the digital encoding and representation of the possible and desired interactions between all elements of the real world and the interaction interfaces through the persistent virtual world system 112. The user can experience the interaction in the form of a game-like interactive application or through a game-like interaction mechanism.

[0069] Figure 3 FIG. shows a schematic diagram of a resource allocation system 300 of a distributed message exchange platform 216 by a resource manager according to an embodiment.

[0070] The resource allocation system 300 allocates resources 302 to the distributed message exchange platform 216, where the resources 302 include computing power 304, memory 306, and network 308, and where the resources 302 can be obtained from one or more server computers 102 and / or client devices 114. The distributed message exchange platform 216 receives requests from one or more virtual objects 110 present in unit 108 and performs corresponding allocations by publishing to unit 108. In some embodiments, to perform these resource allocations, the distributed message exchange platform 216 utilizes a publish-subscribe model, whereby one or more virtual objects 110 subscribe to one or more units 108 to which resources 302 have been published. For example, a virtual object representing a user or a user avatar can be active in a particular unit 108, triggering the resource manager to publish the resources required by the user avatar to the corresponding unit, where the publishing is performed by the distributed message exchange platform 216. Thus, the resources 302 are published to the distributed message exchange platform 216 once and are delivered to each virtual object 110 that subscribes 310 to them. This is in contrast to typical 3D engine systems developed for providing virtual worlds, which output a complete data update packet for each client device within the map, customizing each packet for the client device such that the client device only receives data that may be relevant to the client device and visible objects, resulting in the 3D engine potentially sending the same data multiple times for each client device for which an object is visible.

[0071] The publish-subscribe model is a communication model based on content production and consumption, which is different from the commonly used client / server-based communication model, in which communication typically occurs between two nodes: the client and the server. Such communication via the publish-subscribe model can be anonymous, meaning that the server computer 102 and / or client device 114 do not need to know who subscribes to which unit 108 to obtain resources 302; similarly, the virtual objects 110 obtaining the resources do not need to know where the resources 302 come from. The typical publish-subscribe model enables information exchange where everyone can publish and receive information, providing greater flexibility than the typical client-server model. However, the use of the publish-service model also brings problems of inefficiency, as typically each publication is delivered to every participant, who must analyze the information to determine its relevance. On the other hand, the publish-subscribe model of the present disclosure enables a larger amount of resources to be dynamically concentrated in the units 108 that need them most, such as units 108 with higher resource requirements that virtual objects 110 can subscribe to. Moreover, since the amount of resources published takes into account the amount of virtual objects 110, the level of interaction within the part of the persistent virtual world system visible to the user, and other parameters, each virtual object 110 only obtains from each subscribed unit the resources required during the corresponding simulation.

[0072] In some embodiments, the computing power 304 of the resource 302 includes engine services 314 obtained from one or more separate distributed 3D engines (e.g., Figure 2 the separate distributed software engine 218 of Figure 2 ). In other embodiments, the memory 306 of the resource 302 stores world states, which are also published to the corresponding units 108 to provide an updated version of the real world to the client devices subscribing to the units 108 via the virtual world. In another embodiment, a distributed message exchange platform 308 is included as part of the corresponding shared network resource. In some embodiments, multiple distributed software engines 218 may be used in parallel or sequentially within the network 308 to create synergies with each other.

[0073] As used herein, the term "engine service" refers to operations performed and provided by one or more 3D engines to achieve one or more tasks, such as game or other application-related tasks. 3D engines typically include software components required to perform such tasks, such as libraries, software development kits (SDKs), objects, etc. Engine services include, for example, services related to 3D rendering, graphics, physics, audio, artificial intelligence, positioning, and communication. Each engine service may be performed by a separate distributed software engine provided by a hardware engine, which may be a dedicated hardware engine, a general-purpose hardware engine, or a combination thereof, and each hardware engine includes circuitry dedicated to executing instructions for performing functions called by the separate distributed 3D engine.

[0074] As used herein, the term "world state" refers to the state of each virtual object in a persistent virtual world system, which may be updated in a timely manner through multiple connected devices including sensors that provide sensor data to the persistent virtual world system. The connected devices may be, for example, Internet of Things (IoT) devices that can communicate with each other and with a server via a network and can continuously capture multi-source sensor data from the real world to update the virtual copy of the persistent virtual world system. Thus, state updates may be caused by sensor data. However, updates to the world state may also be caused by user input or by computations (e.g., artificial intelligence inferences) from one or more 3D engines.

[0075] In some embodiments, the assignment of virtual objects 110 to the units 108 by publishing may be based on the current load and the corresponding computing and sorting requirements. Refer to Figure 2, the calculation and sorting of such requirements can be performed by a resource manager 214, such as a distributed 3D engine 212. Sorting can help determine the amount of resources 302 to be released to each unit 108, such that units with lower demand sorting receive fewer resources 302 than units with higher demand sorting. In some embodiments, the requirements take into account the number of virtual objects 110 within a portion of the persistent virtual world system visible to the user through a corresponding client device within the unit. The portion of the persistent virtual world system visible to the user avatar can be defined by the field of view or realm of the avatar within the unit. For example, when a user views a portion of the world in augmented reality or virtual reality, the corresponding avatar may be available within the persistent virtual world system, and thus the frustum of the user avatar can define such a field of view. In other embodiments, when a user uses an avatar through a "flat" user interface (such as through a computer or mobile phone screen), the actual user avatar may be visible on the computer screen, where the user avatar views and interacts within a portion of the persistent virtual world system. In such embodiments, the user avatar can view the persistent virtual world system from a first-person perspective, and the frustum of the first-person avatar can define the field of view. In embodiments where the user avatar is seen from a top view, the field of view can be defined by a radius around the user avatar. In all cases, the field of view can include a plurality of virtual objects 110 with which the user may potentially interact through the user avatar. Thus, the more virtual objects within the field of view of the user avatar, the higher the resource requirements for a particular unit, and thus the higher the sorted demand score. In other embodiments, the requirements also take into account the level of interaction within the portion of the persistent virtual world system visible to the user. For example, there may be two user avatars in two different regions of the persistent virtual world system and within two different units, both having 5 virtual objects 110 available for interaction. If the first user decides to interact with all of the virtual objects 110 simultaneously (e.g., by participating in a group activity, conversation, battle, etc.), then if the second user avatar does not interact with any of the virtual objects 110, or interacts at a lower level with such virtual objects 110, such a unit can have a higher sorted demand than the second user avatar and thus be allocated more resources. In this example, the sorted demand of the virtual unit in which the first user avatar is located is higher than that of the virtual unit in which the second user avatar is located.

[0076] In other embodiments, the requirements further include factors among other parameters, such as the distance between the client device and the network device (e.g., server and / or antenna), the type of application the user is using, or the type of rights the user may have according to their personal user profile or contract details. As an example of the distance between the client device and the network device, the user may be far from the network device within a particular unit of the persistent virtual world system, and since each unit may be assigned a particular network device, more network devices may be activated simultaneously to compensate for the potentially lower quality of service (QoS), resulting in a higher-ranked requirement for that unit. As an example of the application the user is using, if the application consumes more resources, such as bandwidth or computing power, due to the need for a higher real-time rendering rate or better level of detail and clarity, then this would represent an increase in the resources required for that unit and thus a corresponding increase in the ranking requirement. As an example of the type of rights, the user may register as a premium account user to obtain a higher quality of service (QoS) and higher data transfer speed than other users, resulting in a higher resource allocation than other users. Additionally, all of the above factors can be combined and calculated with each other to determine the ranking requirement for each unit.

[0077] Figures 4A - 4B FIGS. 400a-b illustrate a schematic diagram of engine service allocation via a distributed message exchange platform 216 according to an embodiment.

[0078] Figure 4A Schematic diagram 400a includes a plurality of separate distributed software engines 218 hosted on one or more servers, such as Figure 1 server computer 102. Each separate distributed software engine 218 provides a specific engine service, which can be published to unit 108 via the distributed message exchange platform 216. Then, virtual objects 110, such as virtual objects A-D, can obtain those engine services by subscribing to the unit 108 in which they are located (e.g., units 1-2).

[0079] As Figure 4A shown, the separate distributed software engines 218 can be multiple dynamic and independent software modules designed for one or more specific types of tasks. The separate distributed software engines 218 are independent entities that provide the functionality for which the separate distributed software engines 218 have been programmed. Thus, the separate distributed software engines 218 can be, for example, an artificial intelligence engine 402a, a 3D graphics engine 402b, a simulation engine 402c, a communication engine 402d, an audio engine 402e, a 3D structure engine 402f, a location engine 402g, and a rendering engine 402h, etc. Communication between the separate distributed software engines 218 is achieved through function calls managed by the distributed message exchange platform 216.

[0080] In some embodiments, multiple distributed software engines 218 can be used in parallel or in sequence within a network to create synergies with each other. For example, multiple 3D graphics engines 402b hosted on the same or different server computers can allocate engine services sequentially or in parallel through a distributed message exchange platform 216 to supplement each other's engine services for implementing one or more specific tasks. In one embodiment, different 3D graphics engines 402b calculate 3D models at different locations, where the locations can be in the same or different units 108. In another embodiment, different 3D graphics engines 402b can calculate different applications located in the same area, where at least some applications are configured in the AR layer and at least some applications are configured in the VR layer. For example, the same location within the same unit is calculated by while any enhancements such as pure virtual objects and applications configured in the AR layer are partially calculated by and In addition, in this embodiment, consecutive locations within the same or different units 108 can be calculated by in the VR layer, while the enhancements in the AR layer are partially calculated by and

[0081] In addition, in Figure 4A , virtual object A - B is currently located in unit 1 of the persistent virtual world system and is receiving multiple engine services, such as AI service 404 provided by AI engine 402a, rendering service 406 provided by rendering engine 402h, and simulation service 408 provided by simulation engine 402c, which may be necessary for the current application and processing of virtual objects and interactions in the current unit 1. On the other hand, virtual object C - D is currently located in unit 2 from the persistent virtual world system and is receiving multiple engine services, such as audio service 410 provided by audio engine 402e, communication service 412 provided by communication engine 402d, and simulation service 408 provided by simulation engine 402c, which may be necessary for the current application and processing of virtual objects and interactions in the current unit 2. These engine services can be provided by the same server computer hosting separate distributed software engines 218, multiple server computers hosting separate distributed software engines 218, or a combination thereof.

[0082] Figure 4B ​FIG. 400b is a schematic diagram showing the relationship between a separate distributed software engine 218, a hardware engine 220, and a distributed 3D engine 212 of one or more servers 102. When receiving a request from one or more virtual objects located in unit 108, the resource manager 214 hosted on one or more server computers 102 receives the request, evaluates the requirements, and requests engine services from one or more server computers 102 that respectively host the software and hardware engines 218-220 through the distributed message exchange platform 216. Each engine service can be provided by a separate distributed software engine 218, such as 3D engines 1-N provided by corresponding hardware engines 220, such as hardware engines 1-N, which can be dedicated hardware engines, general-purpose hardware engines, or a combination thereof. The distributed message exchange platform 216 then receives the corresponding engine services and publishes them to one or more units 108 that need the engine services. Then, the virtual objects requesting the services subscribe to the corresponding units to receive the necessary engine services through the distributed message exchange platform 216.

[0083] Figures 5A - 5C FIGS. 500a-c are schematic diagrams showing the relationship between a sub-universe (also simply referred to as "universe" in this disclosure), a stream, and other elements of a persistent virtual world system according to an embodiment.

[0084] Referring Figure 5A to FIG. 500a, multiple connected virtual objects 110 that can be used for a specific target simulation can form a stream 502, where each stream 502 is configured to obtain data from the corresponding virtual object 110 and perform a specific target simulation. Each stream 502 can be connected to a root node 504 or a root simulation object that generates multiple stream-related virtual objects 110. The stream 502 can also be connected to an import function 506 that defines the virtual objects 110 required for the stream 502 to perform the required simulation. For example, there may be a traffic stream 502 that includes multiple virtual copies of vehicles, people, traffic lights, and even weather conditions. In other words, the stream 502 includes any number of virtual objects 110 that may be related to the specific function of the stream 502, such as virtual copies. In this embodiment, the import function 506 can define which virtual objects 110 are to be considered for the traffic simulation. The stream 502 can then read the behavior of each corresponding virtual object 110 by using a combination of virtual and / or real sensors and performing a specific target simulation of the traffic scenario. For example, the goal can be to optimize the traffic flow to run at a predetermined average speed or reduce the chance of traffic jams. For example, such simulations can be used for driving driverless cars or drones. Another embodiment of the stream 502 can be energy consumption, which can include virtual copies of buildings, including power consumption data, and specific energy consumption virtual copies, such as machines, computers, household appliances, etc.

[0085] Continuing to refer to Figure 5A the description of, a plurality of streams 502 of all or most of the virtual objects 110 in a predetermined environment form a sub-universe, referred to in the present disclosure as a universe 508. For example, a city universe 508 can represent all or most of the buildings, trees, people, streets, traffic lights, vehicles, etc. of a city. In some embodiments, there can also be universes 508 within a universe 508, such as a house universe within a city universe, which can include the 3D design of the exterior and interior of a house, as well as all or most of the household appliances, people, walls, energy consumption data, etc. of the house.

[0086] Referring to Figure 5B the schematic diagram 500b of, the network of the universe 508 (such as universe 1-N) can form a persistent virtual world system 112. For example, a city universe can be connected to other city universes to form a national universe. If a national universe is not connected to any other universe 508, the national universe can represent the universe of the persistent virtual world system 112.

[0087] Referring to Figure 5C the schematic diagram 500c of, which shows a spatial diagram of a universe 508 including a plurality of streams 502 (such as streams 1-4), each stream including a plurality of stream-related virtual objects 110.

[0088] Figures 6A - 6B Shows schematic diagrams 600a-b of dividing a universe 508 into a plurality of cells 108 according to an embodiment.

[0089] As Figure 6AAs shown, when the resource manager requests resources based on a request from the virtual object 110, the unpartitioned universe 602 representing a region of the world can be partitioned into a partitioned universe 604 including a plurality of cells 606. Higher resource-intensive cells 608, such as cells including a higher number of virtual objects 110, can be further partitioned into a greater number of cells 606. Similarly, cells 606 having a smaller number of virtual objects 110 can be partitioned into a smaller number of cells 606. Thus, each cell 606 is assigned resources, where higher resource-intensive cells 608 receive more resources than lower resource-intensive cells 606. For example, some cells 606 within the higher resource-intensive cell 608 include more virtual objects 110 than the remaining cells 606. However, it can be understood that the amount of resources allocated to each cell 606 is associated not only with the number of virtual objects 110 in a particular cell 606, but also with the following: the level of interaction within the part of the persistent virtual world system visible to the user; the distance between the user and network devices such as servers and / or antennas; the type of application the user is using; the type of rights the user may have, depending on his personal user profile or contract details; or a combination thereof. However, for simplicity and illustrative purposes, Figures 6A - 6B only the virtual object 110 is utilized as a parameter for evaluating demand and managing the allocation to one or more cells.

[0090] In some embodiments, the data structure representing the real world as cells is an octree data structure, where each cell is represented as a voxel within the octree data structure. In some embodiments, the voxels used in the octree data structure are sparse voxels for the arrangement of a larger part of the real world. In other embodiments, the voxels used in the octree data structure are dense voxels for the arrangement of a smaller part of the real world. The octree and the selected sparse or dense voxels are suitable data structures for representing three-dimensional space in a virtual world, such as in a persistent virtual world system representing the real world. The octree can quickly and easily combine multiple objects, implement a relatively simple rendering algorithm, be able to perform fast spatial searches, and be able to generate models through the virtualization of real objects (e.g., through LIDAR or other image scanning methods). For a virtual world including a 3D virtual world representing at least a part of the real world, such 3D partitioning of the world enables three-dimensional-based engine services and cloud resource allocation over the area where the user may be viewing and / or interacting. However, in the case of handling a 2D world, a quadtree can also be used. In other embodiments, other suitable data structures can be utilized, such as BSP trees, sparse voxel octrees, 3D arrays, kD trees, point clouds, wireframes, boundary representations (B-Reps), constructive solid geometry trees (CSG trees), binary trees, and hexagonal structures, etc.

[0091] In some embodiments, a data structure includes a collection of data values, relationships, and functions or operations that can be applied to data, where the data at least defines the geometry and coordinates of virtual objects in a persistent virtual world system. For example, the correct data structure selection depends on the source of the data, the geometric precision sought during rendering; whether the rendering is done in real-time or pre-rendered; whether the rendering is performed via a cloud server, via a user device, a fog device, or a combination thereof; the specific application using the persistent virtual world system, e.g., medical or scientific applications may require a higher level of clarity compared to other types of applications; the memory capacity from the server and from the user device, and thus the desired memory consumption; and others.

[0092] In addition, in Figure 6A , after an event associated with one or more requests has ended, the resources are restored to their original one or more sources. Subsequently, the partitioned universe 604 is merged back into the original number of cells 606, as shown by the curved arrow 610 returning from the partitioned universe 604 to the unpartitioned universe 602. In this way, the system achieves efficient and dynamic allocation of resources by selecting regions of interest with higher demand and returning resources to their sources when they are no longer needed.

[0093] Figure 6B An illustration of an octree data structure 600b is shown. The octree data structure 600b directly represents volumes in the data structure, such as volumes representing at least a portion of the real world. The main advantages of the octree data structure 600b are the quick and easy combination of several objects; the implementation of simple rendering; the ability to perform fast spatial searches; and the generation of models through the digitization of real objects (e.g., LIDAR or other image scans).

[0094] The octree data structure 600b is a tree data structure where each internal node has eight children or octants. The deeper the universe is partitioned, the deeper the level of the resulting children nodes, and the more resources are allocated to the cells in the deeper partitions that require resources. Thus, in Figure 6B 's embodiment, the octree data structure 600b has generated three levels 612 of children nodes, levels 1 - 3, plus the root node 614 representing the entire universe. The areas that have been bolded in the octree data structure 600b of the universe represent the areas with virtual objects 110. The upper levels of the octree data structure 600b of the universe represent regions that require fewer partitions and thus fewer resources because, for example, there are fewer virtual objects 110 located therein. In contrast, the lower levels that include a greater number of virtual objects 110 require more partitions and thus more resources.

[0095] Figure 7FIG. 700 is a schematic diagram showing another embodiment of dividing the universe into multiple units according to an embodiment.

[0096] Schematic diagram 700 shows a partitioned universe 604 including three streams 702 (Stream 1 - 3), where each stream 702 includes one or more stream - specific virtual objects 110, and where each stream 702 is programmed to be enabled or disabled for viewing and interaction on a client device. In other embodiments, each stream 702 is associated with one or more user applications. Each stream 702 can be configured to obtain data from the corresponding virtual object 110 and perform simulations specific to a theme and / or objective. Each stream 702 can be connected to a root node or root simulation object that generates multiple stream - related virtual objects 110. Stream 702 can additionally be connected to an import function that defines the virtual objects 110 required for the stream to perform the simulation.

[0097] For example, if the viewer is using a restaurant application, the user can choose to enable the restaurant stream (e.g., Stream 1) of the partitioned universe 704, whereby only the restaurant will be presented or highlighted to the user. Similarly, a traffic - related application can be configured to implement a traffic stream (e.g., Stream 2) that presents or highlights traffic to the user. In another example, a game - related application including multiple pure virtual objects can enable the game stream (e.g., Stream 3), presenting or highlighting the pure virtual objects associated with the game application. In the current embodiment, since there are more virtual objects 110, Stream 3 may require a greater amount of resources than the other streams 1 - 2. Thus, the sum of all streams 1 - 3 will result in the partitioned universe 704. In some embodiments, depending on which streams 702 are enabled, the enabled streams 702 are regarded by the user as rendered or unrendered media streams output on the client device, where the streams 702 include 3D image data, 3D geometry, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, text data, time data, location data, orientation data, tactile data, and lighting data, etc., or a combination of such media content. Thus, each stream 702 can be computed by one or more separate distributed software engines provided by one or more hardware engines and optionally rendered. In some embodiments, one or more separate distributed software engines include instructions that, once executed by one or more hardware engines, trigger end - to - end encryption (E2EE) of the stream 702, such that only users associated with a particular stream 702 can receive and decrypt the media content of the stream 702. For example, such embodiments can be used during a video call utilizing an end - to - end streaming service.

[0098] Figure 8 FIG. shows a schematic diagram of a client device 114 according to an embodiment. Figure 9 Some elements of can be similar to Figures 1 - 8elements, and thus like or identical reference numerals may be used to describe those elements.

[0099] The client device 114 of the present disclosure may include operating components such as an input / output (I / O) module 802; a power supply 804; a memory 806; a sensing mechanism 808 and a transceiver 810 that form a tracking module 812; and a network interface 814, all of which are operably connected to a processor 816.

[0100] The I / O module 802 is implemented as computing hardware and software that is configured to interact with a user and provide user input data to one or more other system components. For example, the I / O module 802 may be configured to interact with a user, generate user input data based on the interaction, and provide the user input data to the processor 816 before transmitting it via a network to other processing systems (e.g., a server). In another embodiment, the I / O module 802 is implemented as an external computing pointing device (e.g., a touch screen, a mouse, a 3D control, a joystick, a gamepad, etc.) and / or a text input device (e.g., a keyboard, a dictation tool, etc.) that is configured to interact with the client device 114. In other embodiments, the I / O module 802 may provide additional, less, or different functionality compared to that described above.

[0101] The power supply 804 is implemented as computing hardware and software that is configured to power the client device 114. In one embodiment, the power supply 804 may be a battery. The power supply 804 may be built into the client device 114 or removable from the client device 114, and may be rechargeable or non-rechargeable. In one embodiment, the client device 114 may be re-powered by replacing one power supply 804 with another power supply 804. In another embodiment, the power supply 804 may be recharged via a cable connected to a charging source, such as a universal serial bus (“USB”), FireWire, Ethernet, Thunderbolt, or headphone cable attached to a personal computer. In another embodiment, the power supply 804 may be recharged by inductive charging, where an electromagnetic field is used to transfer energy from an inductive charger to the power supply 804 when the two are in close proximity, but without the need to plug into each other via a cable. In another embodiment, a docking station may be used to facilitate charging.

[0102] The memory 806 can be implemented as computing hardware and software suitable for storing application instructions and sensor data captured by a plurality of sensing mechanisms 808. The memory 806 can be any suitable type capable of storing information accessible to the processor 816, including computer-readable media, or other media that store data readable by an electronic device, such as a hard disk drive, memory card, flash drive, ROM, RAM, DVD, or other optical disc, and other writable and read-only memories. In addition to the persistent storage, the memory 806 can include temporary storage.

[0103] The sensing mechanism 808 can be implemented as computing hardware and software suitable for obtaining various sensor data from the real world and determining / tracking the position and orientation of the client device 114. The sensing mechanism 808 can include, but is not limited to, one or more of the following, including: one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors (e.g., gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, chromaticity sensors, humidity sensors, water sensors, electrical sensors, or combinations thereof. In particular, the sensing mechanism 808 includes one or more inertial measurement units (IMUs), accelerometers, and gyroscopes. The IMU is configured to measure and report the speed, acceleration, angular momentum, translational speed, rotational speed, and other telemetry metadata of the client device 114 by using a combination of the accelerometer and the gyroscope. The accelerometer configured within and / or separate from the IMU can be configured to measure the acceleration of the interaction device, including the acceleration caused by the Earth's gravitational field. In one embodiment, the accelerometer includes a triaxial accelerometer capable of measuring acceleration in three orthogonal directions.

[0104] The transceiver 810 can be implemented as computing hardware and software configured to enable the client device 114 to receive wireless radio waves from the antenna and send data back to the antenna. In some embodiments, an mmW transceiver can be used, which can be configured to receive an mmW wave signal from the antenna and send data back to the antenna when interacting with immersive content. The transceiver 810 can be a bidirectional communication transceiver 810.

[0105] In an embodiment, the tracking module 812 can be implemented by combining the capabilities of the IMU, accelerometer, and gyroscope with the position tracking provided by the transceiver 810, and sub-centimeter or sub-millimeter position and orientation tracking can be achieved based on the precise tracking, low latency, and high QOS functionality provided by the mmW-based antenna, which can improve the accuracy when tracking the real-time position and orientation of the client device 114. In an alternative embodiment, the sensing mechanism 808 and the transceiver 810 can be coupled together in a single tracking module device.

[0106] The network interface 814 can be implemented as computing software and hardware to communicatively connect to a network, receive computer-readable program instructions sent by a server or by a client device 114 from the network, and forward the computer-readable program instructions for storage in the memory 806 for execution by the processor 816.

[0107] The processor 816 can be implemented as computing hardware and software configured to receive and process sensor data. For example, the processor 816 can be configured to provide imaging requests, receive imaging data, process the imaging data into environmental or other data, process user input data and / or imaging data to generate user interaction data, perform edge-based (on-device) machine learning training and inference, provide server requests, receive server responses, and / or provide user interaction data, environmental data, and content object data to one or more other system components. For example, the processor 816 can receive user input data from the I / O module 802 and can implement application programs stored in the memory 806, respectively. In other embodiments, the processor 816 can receive sensor data captured from the real world from the sensing mechanism 808, or can receive the precise location and orientation of the client device 114 through the tracking module 812, and can prepare some data before sending the data to the server for further processing. For example, the processor 816 can implement some of the steps required during data preparation, including analog or digital signal processing algorithms, such as performing raw data reduction or filtering sensor data before sending the data to the server.

[0108] Figure 9 A block diagram of a computer-implemented method 900 for achieving dynamic load balancing through a distributed 3D engine according to an embodiment is shown.

[0109] In step 902, the method 900 for implementing virtual or real-world multi-dimensional 3D engine computations and virtualization-based dynamic load balancing begins by providing a data structure in the memory of a server computer, the data structure representing at least a portion of the real world as a plurality of units storing virtual objects that form a persistent virtual world system, where at least some of the virtual objects are virtual copies of corresponding real-world elements. The method continues in step 904 in the memory of at least one server computer, where the distributed 3D engine includes a resource manager and a plurality of individual distributed software engines implemented in a distributed deployment.

[0110] Then, at step 906, the method continues to calculate the requirements for each individual unit by the resource manager based on the current load, where the requirements further take into account one or more of the number of virtual objects and the level of interaction within the portion of the persistent virtual world system visible to the user avatar. In other embodiments, the requirements take into account other parameters such as the distance between the user and the network devices (such as servers and / or antennas), the type of application the user is using, or the type of rights the user may have according to their personal user profile or contract details. Based on the requirements, the method continues at step 908 by sorting the units according to the requirements. Subsequently, based on the sorted requirements, the method continues at step 910 by dynamically allocating resources to one or more of the higher resource-intensive units.

[0111] Figure 10 FIG. shows a block diagram of a computer-implemented method 1000 according to an embodiment, the method including other steps of method 900 for implementing dynamic load balancing by a distributed 3D engine.

[0112] According to an embodiment, method 1000 starts at step 1002 by partitioning a higher resource-intensive region of interest of at least a portion of the world into a greater number of units based on the calculated requirements. Then, at step 1004, method 1000 continues by allocating the corresponding resources from one or more sources to each unit. After concluding an event associated with one or more requests within the higher resource-intensive region of interest, method 1000 continues at step 1006 by restoring the resources to the original one or more sources. Finally, at step 1008, method 1000 ends by merging the universe back into the original number of units.

[0113] Figure 11 FIG. shows a block diagram of a method 1100 according to an embodiment describing the resource allocation steps via a distributed message exchange platform.

[0114] Method 1100 may start at step 1102 by publishing the corresponding resources to the corresponding units using a publish-subscribe model by the distributed message exchange platform of the resource manager. Then, at step 1104, method 1100 ends by having one or more virtual objects subscribe to the units of interest in order to obtain the required resources via the distributed message exchange platform.

[0115] In some embodiments, method 1200 further includes sharing, via a distributed messaging platform, a dynamic updated state of at least a portion of the world with one or more client devices, server computers, or a combination thereof, where the client devices and / or server computers may subscribe to the dynamic updated state to obtain an updated version of the world. In yet another embodiment, the state of at least a portion of the world is obtained via a plurality of connected devices, via user input, via server computation, or a combination thereof, where the connected devices include sensors that provide sensor data to the persistent virtual world system.

[0116] Although certain embodiments have been described and shown in the drawings, it is to be understood that such embodiments are merely illustrative and not limiting of the broad disclosure, and the present disclosure is not limited to the specific constructions and arrangements shown and described, as various other modifications may occur to those of ordinary skill in the art. Accordingly, the description is to be regarded as illustrative rather than restrictive.

Claims

1. A system for performing dynamic load balancing, characterized in that Comprising: One or more server computers, said one or more server computers including a memory and at least one processor, the memory storing: A data structure that virtualizes at least a portion of a virtual or real world into multiple units storing virtual objects that form a persistent virtual world system; And A distributed 3D engine, implemented in a distributed deployment, the distributed 3D engine including a resource manager and multiple separate distributed software engines; Wherein, based on the current load and the corresponding computing and sorting requirements of one or more of the multiple units, resources are dynamically allocated to the one or more units via the distributed deployment, wherein the sorting requirements are based on the amount of virtual objects within the one or more units or the level of interaction within a portion of the persistent virtual world system visible to a user avatar within the one or more units, and The units representing the original quantity of a higher resource-intensive region of interest from the at least a portion of the virtual or real world are further divided into a greater number of units, wherein the smaller units are allocated a greater amount of resources, wherein the resources are restored after an event associated with one or more requests ends, and wherein the at least a portion of the virtual or real world is merged back into the original quantity of units.

2. The system according to claim 1, wherein The sorting requirements are further based on the distance between a client device accessing the one or more units and a network device, the type of application used by a user accessing the one or more units, or the type of rights of a user accessing the one or more units.

3. The system according to claim 1, wherein The persistent virtual world system includes a low-fidelity simulation for demand assessment and load balancing, and a high-fidelity simulation for enhancing the user experience.

4. The system according to claim 1, wherein At least some of the virtual objects of the persistent virtual world system include self-computing capabilities and autonomous behaviors.

5. The system according to claim 1, characterized in that, One or more specific tasks are achieved by sequentially or concurrently using the separate distributed software engines via the distributed deployment to supplement each other's engine services.

6. The system according to claim 1, wherein The data structure representing the virtual or real world as the multiple units is an octree data structure, and wherein each unit of the multiple units is represented as a voxel within the octree data structure, wherein the voxel is a sparse voxel in an arrangement for a larger portion of the virtual or real world or a dense voxel in an arrangement for a smaller portion of the virtual or real world.

7. The system according to claim 1, wherein The data structure representing the virtual or real world as the multiple units includes one or more of the following: BSP tree, sparse voxel octree, 3D array, kD tree, point cloud, wireframe, boundary representation (B-Rep), constructive solid geometry tree (CSG tree), binary tree, or hexagonal structure.

8. The system according to claim 1, wherein The resource manager performs the allocation via a distributed message exchange platform, wherein the distributed message exchange platform utilizes a publish-subscribe model, and wherein one or more virtual objects subscribe to one or more of the units that publish resources among the multiple units.

9. The system according to claim 8, wherein The distributed message exchange platform shares with one or more of the client device and the server a dynamically updated state of the at least one part of the virtual or real world stored in the memory, and wherein the virtual or real world state is modified by data obtained by one or more of the plurality of connection devices, by user input, by server calculations, or a combination thereof, the plurality of connection devices including sensors that provide sensor data to the persistent virtual world system.

10. The system according to claim 1, wherein Each of the plurality of units includes one or more streams, each stream including a plurality of stream-specific virtual objects that are programmed to be enabled or disabled for viewing and interaction on the client device, and wherein each of the streams is associated with one or more applications.

11. A method for performing dynamic load balancing, characterized in that, Comprising: Providing a data structure in a memory of at least one server computer, the data structure virtualizing at least one part of the virtual or real world into a plurality of units storing virtual objects forming a persistent virtual world system, wherein at least some of the virtual objects are virtual copies of corresponding real world elements; Providing in the memory of the at least one server computer a distributed 3D engine implemented in a distributed deployment, the distributed 3D engine including a resource manager and a plurality of separate distributed software engines; Calculating, by the resource manager, the requirements of individual units among the plurality of units based on the current load, wherein the requirements are based on the amount of virtual objects within the individual unit or the level of interaction within a part of the persistent virtual world system visible to a user avatar within the individual unit; Sorting the individual units according to the requirements; Dynamically allocating resources to the individual units based on the sorted requirements; Further dividing units among the plurality of units that represent an original amount of a higher resource-intensive region of interest from the virtual or real world into a greater number of units, wherein the smaller units are allocated a greater amount of resources; Restoring the resources after an event associated with one or more requests has ended; and Merging the at least one part of the virtual or real world back into the original amount of units.

12. The method according to claim 11, characterized in that, The requirements are further based on the distance between the client device accessing an individual unit among the plurality of units and a network device, the type of application used by a user accessing the individual unit, or the type of rights of the user accessing the individual unit.

13. The method according to claim 11, wherein The persistent virtual world system includes a low-fidelity simulation for demand assessment and load balancing, and a high-fidelity simulation for enhancing the user experience.

14. The method according to claim 11, wherein Using the separate distributed software engines sequentially or in parallel through the distributed deployment to complement each other's engine services to implement one or more specific tasks.

15. The method according to claim 11, wherein Further including providing an octree data structure as the data structure representing the virtual or real world, wherein each of the plurality of units is represented as a voxel, and the voxel is a sparse voxel in an arrangement for a larger part of the virtual or real world or a dense voxel in an arrangement for a smaller part of the virtual or real world.

16. The method according to claim 11, wherein Further including: Based on the calculated requirements, divide at least one region of interest of the at least one part of the virtual or real world into additional units; Allocate corresponding resources to the additional units.

17. The method according to claim 11, wherein The allocation is performed by: Using a publish-subscribe model, publish resources to one of the multiple units through the distributed message exchange platform of the resource manager; and Subscribe to one of the multiple units through one or more virtual objects to obtain the required resources.

18. The method according to claim 11, wherein Each of the multiple units includes one or more streams, each stream including a plurality of stream-specific virtual objects, the virtual objects being programmed to be enabled or disabled for viewing and interaction on a client device, and wherein each stream is associated with one or more user applications.

19. One or more non-transitory computer-readable media storing instructions thereon, characterized in that, The instructions are configured to cause the one or more computers to perform steps including: Provide a data structure that virtualizes at least one part of the virtual or real world into multiple units that store virtual objects forming a persistent virtual world system, wherein at least some of the virtual objects are virtual copies of corresponding real-world elements; Provide a distributed 3D engine implemented in a distributed deployment, the distributed 3D engine including a resource manager and a plurality of individual distributed software engines; Calculate, by the resource manager, the requirements of individual units among the multiple units based on the current load, wherein the requirements are based on the amount of virtual objects within the individual unit or the level of interaction within a part of the persistent virtual world system visible to a user avatar within the individual unit; Sort the individual units according to the requirements; Dynamically allocate resources to the individual units based on the sorted requirements; Further divide the units representing the original quantity of higher resource-intensive regions of interest from the at least one part of the virtual or real world into a greater number of units, wherein the smaller units are allocated a greater amount of resources; Recover the resources after an event associated with one or more requests ends; and Merge the at least one part of the virtual or real world back into the original quantity of units.

Images