Systems and methods for optimizing virtual world computations through an n-tier architecture

Abstract

Systems and methods are provided for optimizing virtual world computation through an n-tier architecture, the n-tier comprising at least three tiers. In an example three-tier architecture, a first tier comprises a client software engine configured to receive input data, send data to a second tier, and perform end user processing. The second tier is connected to the first and third tiers through a network and comprises client-specific modules that are dynamically instantiated and configured to prepare received data from the client software engine for subsequent processing or send the received data to the third tier. The third tier comprises virtual world processing modules configured to receive and process data from the second tier, generate world state updates, and dynamically instantiate client-specific modules to produce client-specific instances. The world state updates are sent to the respective client-specific module instances for further processing.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application 62 / 955239, filed December 30, 2019, which is incorporated herein by reference. Background Technology

[0003] Software architecture is a style and methodology for designing and building software systems. Software architecture can be divided into multiple layers, where each layer is a logical or functional division of the software architecture, defining a set of technologies that provide one or more services to its clients. Over time, system architecture has evolved from monolithic architectures to multi-layered or n-layered architectures. In monolithic architectures, presentation, application logic, and resource management are combined into a single layer, while in multi-layered architectures, logical or functional elements are divided into two or more layers. For example, a two-layer architecture separates the presentation layer from the application logic and resource management in another layer. This type of architecture offers portability, ease of maintenance, and high communication speeds due to direct client-server communication, but it introduces scalability issues due to applications requiring a large number of entities. The demand for greater scalability and flexibility led to the creation of three-layer architectures, which typically include a presentation layer, a business or data access layer, and a data layer. This architecture allows each component to participate in multiple applications simultaneously, exhibiting higher performance, greater scalability, reusability, flexibility, and manageability compared to previous architectures. This is the primary architecture used by most network and legacy applications today.

[0004] However, the three-tier architecture has certain limitations that hinder its use in resource-intensive applications, such as those related to virtual worlds. Traditional systems using a three-tier architecture to run virtual worlds, particularly massively multiplayer online games (MMOGs), are limited in how they scale to support a large number of players or entities within the virtual world. Because these virtual worlds typically run on a single physical computer and depend on a single instance of the game or application, they are supported by the system's limited bandwidth and the processor capacity of the server hosting the virtual world. The server hosting the game or simulation engine needs to provide simulation capabilities for every object and user in the virtual world. However, as the number of players and entities increases, so does the resource demand, degrading the Quality of Service (QoS) and user experience within the virtual world. As virtual worlds become larger and more complex, current implementations of the three-tier architecture have proven insufficient to achieve adequate scalability and flexibility.

[0005] In the case of MMOGs, which can be applied to any type of interactive application within a large virtual world, three types of real-time remote rendering and computing systems are typically used: 3D graphics streaming, video streaming, and video streaming with post-rendering or post-processing operations. In 3D graphics streaming, the server sends graphics-related commands to the client device, which then interprets and renders the corresponding scene. Therefore, the client device needs to be able to render streaming 3D graphics and process user input. In video streaming, the server renders 3D commands and sends the video stream to the client device. This can lead to low utilization of the client device's hardware capabilities, increased load on cloud servers and network traffic, and higher latency, but it can utilize less powerful client devices. In the third option, the heavy lifting of 3D graphics rendering is performed by the server, while some low-level processing is done on the client side via a thin client. Currently, a good balance between low latency, high scalability, and reusability has not yet been found in the n-tier architecture design. Summary of the Invention

[0006] This summary is intended to introduce, in a simplified form, some concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0007] This disclosure generally relates to computer systems and methods, and more specifically, to systems and methods for optimizing virtual world computing through an n-layer architecture.

[0008] Specific embodiments of the n-layer architecture disclosed herein include at least three layers: a first layer comprising a client software engine configured to receive input data, send the input data to a second layer, and perform end-user processing of the input data; a second layer connected to the first and third layers via a network, the second layer comprising a client-specific module dynamically instantiated by the third layer for each client device, and configured to prepare input data from the client software engine of the first layer for subsequent processing, or send the input data to the third layer, depending on the type of the input data; and a third layer comprising a virtual world processing module configured to receive and process input data from the second layer to generate world state updates, and dynamically instantiate client-specific modules to generate client-specific module instances. The world state updates are sent to the corresponding client-specific module instances for further processing.

[0009] In some embodiments, the client software engine is stored in the memory of at least one client device and executed by at least one processor of the at least one client device. In some embodiments, the client-specific module and the virtual world processing module are stored in the memory of one or more cloud server computers and executed by at least one processor of the one or more cloud server computers. For example, the client-specific module is stored in the memory of at least one cloud server computer and executed by at least one processor of the at least one cloud server computer, and the virtual world processing module is stored in the memory of the same or different cloud server computers and executed by at least one processor of the same or different cloud server computers.

[0010] In some embodiments, the n-tier architecture of this disclosure can be implemented in a cloud-to-edge infrastructure that exhibits distributed computing capabilities employing computing devices and systems, such as public or private clouds, fog servers, small clouds, peer-to-peer cloud server systems, and edge devices and systems, such as enterprise systems, mobile platforms, and user devices, all of which can be connected via a network. According to embodiments of this disclosure, different layers presented as part of a plurality of n-tier architectures can therefore be hosted (e.g., via so-called Infrastructure as a Service or IaaS) in at least some of the computing devices and systems within the cloud-to-edge infrastructure. In some embodiments, the n-tier architecture is implemented in a distributed ledger that enables decentralized and immutable data storage and processing.

[0011] In some embodiments, the client software engine is further configured to identify one or more portions of the input data for processing by different modules based on the type of input data, such as a client-specific module or a virtual world processing module. In one embodiment, the client software engine is further configured to identify data as high-interaction-frequency data or low-interaction-frequency data, wherein the high-interaction-frequency data is processed by the client-specific module, and wherein the low-interaction-frequency data is sent to the virtual world processing module for processing. In another embodiment, high-interaction-frequency data includes high-priority data, and low-interaction-frequency data includes low-priority data. Classifying data into, for example, high-interaction-frequency data or low-interaction-frequency data for processing by the client-specific module or the virtual world processing module enables the automatic and efficient offloading of only necessary data and tasks to appropriate layers.

[0012] In some embodiments, the client-dedicated module is implemented as instructions stored in the memory of the at least one cloud server computer. In another embodiment, the client-dedicated module is implemented as a client-dedicated virtual machine stored in the memory of the at least one cloud server computer. In these embodiments, when instantiated by the virtual world processing module, the client-dedicated module dedicates a specific amount of resources (e.g., memory, network, and computing power) and other server attributes (e.g., operating system) to the client device in question, and uses the client-dedicated instance to perform one or more data preparation operations on the data.

[0013] In some embodiments, the at least one processor of the at least one cloud server computer includes a general-purpose processing unit, a dedicated processing unit, or a combination thereof. The at least one processor can be allocated on demand during each client instantiation via the virtual world processing module. Allocating at least one hardware processor to an instance of the client-dedicated module can provide further resources that can then be used to process data, thereby delivering higher QoS and quality of experience (QOE) for end users.

[0014] In some embodiments, the input data received by the client software engine includes one or more of color data, geometric data, depth data, audio data, pose data, sensor data, and user input commands. In other embodiments, prepared data from the client-specific module includes color data, geometric data, depth data, and audio data streamed to the client software engine. In still other embodiments, some of the prepared data is data that has already been processed by the virtual world processing module. In yet other embodiments, per-client processing further includes data encoding. In still other embodiments, the geometric data includes screen-space meshes, proxy geometry, or combinations thereof.

[0015] In some embodiments, the client software engine is further configured to perform one or more of the following: generating intermediate frames capable of increasing the final image resolution; providing complementary image frames; interpolating frame rates; extrapolating frame rates; rendering prepared data; adjusting brightness, color, contrast, sharpness, and resolution; decoding data; and overlaying image data onto received data; or combinations thereof. In some embodiments, the client software engine performs operations on data prepared by a client-specific module.

[0016] In some embodiments, during the application development phase, developers can adjust the amount of processing performed by each layer (e.g., by percentage) via the client software engine. In some embodiments, this adjustment of processing volume is performed by defining the data types or categories (e.g., levels of high and low interaction frequency data) processed by the client-specific module and the virtual world processing module.

[0017] In some embodiments, the third layer includes a virtual world system. In one embodiment, the virtual world processing module is connected to a database including the virtual world system (e.g., hosted at the third layer), which includes multiple sub-universes, each including multiple virtual objects. In a further embodiment, at least some of the multiple virtual objects are virtual copies of corresponding real-world objects located in corresponding regions of the real world, wherein the virtual world processing module performs processing on the multiple virtual objects within regions of interest. In still other embodiments, the virtual objects also include applications and purely virtual objects that do not have corresponding real-world objects.

[0018] In some embodiments, the virtual world processing module is further configured to provide one or more application services, wherein the provision is at least partially based on spatial scenarios associated with one or more content scenarios through the virtual world system. The virtual world processing module is also configured to organize and supply one or more application services through the virtual world system, including, for example, mobility, security, energy management, recommendations, business transactions, etc. The organization and supply of the application services may be at least partially based on spatial scenarios associated with one or more content scenarios through the virtual world system. The application services may be allocated and sent to a client software engine on demand via a corresponding client-specific module. In a further embodiment, the spatial scenarios are classified into micro-scenarios or macro-scenarios. In one embodiment, each of the micro-scenarios or macro-scenarios may be further classified as a private or public scenario, which may be provided to a specific client software engine via a corresponding client-specific module. In some embodiments, application services and related content associated with private scenarios are provided based on an interest system, rights rules, or any suitable application service priority criteria.

[0019] In some embodiments, the virtual world system may include a virtual world layer, which may be divided into an augmented reality (AR) layer and a virtual reality (VR) layer. The separate layers enable access to the virtual world system in either augmented reality or virtual reality, and allow the combination of elements stored in each layer. The separate layers can be activated via a client device connected to at least one cloud server computer via a network whenever one or the other type of reality is accessed. Each layer may include augmented reality and virtual objects specific to that layer. In some embodiments, the state of at least one part of the world is updated by multiple connected devices via user input, server computation, or a combination thereof, including sensors that provide sensor data to the persistent virtual world system.

[0020] According to one embodiment, a method for optimizing per-user computation through an n-layer architecture comprising at least three layers includes the following steps: a client-dedicated module in the second layer receives user input data sent by a client software engine configured in the first layer and connected to the second layer via a network. The client-dedicated module sends a first portion of the input data to a virtual world processing module in the third layer of the n-layer architecture connected to the client-dedicated module via the network. For example, if the service of the virtual world processing module is required, the method continues, with the client-dedicated module sending the first portion of the input data to the virtual world processing module in the third layer connected to the client-dedicated module via the network. The virtual world processing module is configured to process the first portion of the input data to generate a world state update and dynamically instantiate the client-dedicated module to create a client-dedicated module instance, wherein the world state update is sent to the corresponding client-dedicated module instance. The method then continues, with the client-dedicated module receiving the world state update; and the client-dedicated module further processing the world state update, wherein the processing is performed according to a per-client approach, and the processed world state update is sent to the client software engine to perform the requested end-user processing. The method further includes: a client-dedicated module preparing a second portion of the input data received from the client software engine, and sending the prepared data to the client software engine for performing the requested end-user processing. For example, if the service of the virtual world processing module is not required, the method can continue with the client-dedicated module preparing the data received from the client software engine and sending the prepared data to the client software engine to perform the requested end-user processing.

[0021] In some embodiments, the method further includes identifying a second portion of the input data by the client software engine for processing by a client-specific module; and identifying a first portion of the input data for processing by a virtual world processing module. In one embodiment, the client engine layer identifies data as high-interaction-frequency data or low-interaction-frequency data, wherein the high-interaction-frequency data is processed by the client-specific module, while the low-interaction-frequency data is sent to the virtual world processing module for processing, thereby generating a world state update.

[0022] In some embodiments, the method further includes allocating at least one processor to one or more client devices during the instantiation of the client-specific module by the virtual world processing module, wherein the at least one processor of the at least one cloud server computer includes a general-purpose processing unit, a special-purpose processing unit, or a combination thereof.

[0023] In some embodiments, the input data received by the client software engine includes one or more of color data, geometric data, depth data, audio data, pose data, sensor data, and user input commands. In another embodiment, data prepared from a client-specific module includes color data, geometric data, depth data, and audio data that are streamed to the client software engine. In a further embodiment, the method further includes encoding the prepared data before streaming it to the client software engine.

[0024] In some embodiments, the method further includes the client software engine performing one or more of the following: generating intermediate frames that can increase the resolution of the final image; providing complementary image frames; interpolating frame rates; extrapolating frame rates; rendering prepared data; adjusting brightness, color, contrast, sharpness, and resolution; decoding data; and overlaying image data onto received data; or combinations thereof.

[0025] In some embodiments, the method further includes adjusting the percentage of processing performed by each layer via the client software engine during the application development phase.

[0026] The above summary does not include an exhaustive list of all aspects of this disclosure. It is conceivable that this disclosure encompasses all systems and methods that can be practiced from all suitable combinations of the aspects outlined above, as well as all systems and methods disclosed in the following detailed description, particularly those specifically pointed out in the claims filed with this application. Such combinations have particular advantages not specifically described in the above summary. Other features and advantages will become apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0027] The specific features, aspects, and advantages of this disclosure will be better understood with reference to the following description and accompanying drawings, in which:

[0028] Figure 1A-1C A schematic diagram of a system for optimizing virtual world computing through an n-layer architecture, according to one embodiment, is depicted.

[0029] Figure 2 A schematic diagram of a system for optimizing virtual world computing through an n-tier architecture, according to one embodiment, is depicted, the system including various software and hardware components of a client device and one or more cloud servers;

[0030] Figures 3A-3B A schematic diagram of data processing implemented in a system for optimizing virtual world computing through an n-layer architecture, according to one embodiment, is depicted.

[0031] Figures 4A-4CA flowchart is provided illustrating a system and method for implementing this disclosure in n layers, including three or more layers, according to one embodiment;

[0032] Figure 5 A schematic diagram of a virtual world system, according to one embodiment, is depicted, which can be stored in a database connected to a virtual world processing module.

[0033] Figure 6 A schematic diagram is depicted according to one embodiment of a method for optimizing virtual world computing through an n-layer architecture. Detailed Implementation

[0034] In the following description, reference is made to the accompanying drawings, which illustrate various embodiments by way of illustration. Furthermore, various embodiments will be described below with reference to several examples. It should be understood that embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.

[0035] In embodiments of this disclosure, the term "layer" is used to define the physical or virtual structured mechanism of the system infrastructure on which an application runs, wherein each layer is defined logically or functionally. Any system architecture containing more than one layer is generally referred to as a multi-tier architecture or an n-tier architecture. Therefore, applications running on such a system are considered distributed throughout the system, as each layer may implement one or more functions of the application in a different physical or virtual computer. Each layer communicates with each other by sending messages via different layers and using appropriate communication protocols to reach the corresponding layer on another layer.

[0036] This disclosure describes the functional division of each layer, without limiting the scope to different layers and protocols that can be used for data transmission purposes within each layer. Therefore, the benefits of implementing the systems and methods of this disclosure are not strictly limited to the number of layers used by the system architecture for distributed functions and processes, but rather to how different data processing functions are distributed throughout the system according to the data types received. Thus, for example, some types of data trigger personalized, client-specific data processing methods for 3D virtual worlds, while other types of data implement virtual world processing for tasks that may affect the virtual world itself and may tolerate higher latency. Furthermore, for simplicity, some embodiments of this disclosure are described using a three-layer architecture, although those skilled in the art will understand that similar embodiments can be implemented in more than three layers without affecting the scope of this disclosure.

[0037] Figure 1A-1C A schematic diagram of a system 100a-c for optimizing virtual world computing through an n-layer architecture, according to one embodiment, is depicted.

[0038] refer to Figure 1ASystem 100a includes a first layer 102, which includes a client software engine 104 implemented in a client device 106. The first layer 102 is connected to a second layer 110 via a network 108, which includes a client-specific module 112 implemented in at least one cloud server computer 114. The second layer 110 is also connected to a third layer 116 via the network 108 and includes a virtual world processing module 118.

[0039] In some embodiments, the client software engine 104 is configured to receive input data from one or more client devices 106, transmit the input data to a client-specific module 112 in the second layer 110 via network 108, and perform end-user processing of the data. In a further embodiment, the client-specific module 112 is dynamically instantiated for each client device by a virtual world processing module 118 in the third layer 116, and is configured to prepare data from the client software engine 104 in the first layer 102 for subsequent processing, or to transmit the data to the virtual world processing module 118 in the third layer 116, depending on the type of data. In a further embodiment, the virtual world processing module 118 in the third layer 116 is configured to receive and process data received from the second layer 110 to generate a world state update, and instantiate the client-specific module 112 to generate a client-specific instance. The virtual world processing module 118 can then transmit the world state update to the client-specific instance of the client-specific module 112 in the second layer 110 for further processing.

[0040] refer to Figure 1B System 100b includes four client devices 106, or client devices AD, connected to cloud server 114 via network 108. In this embodiment, at least some data received from at least some client devices AD is sent to virtual world processing module 118, thereby triggering virtual world processing module 118 to instantiate client-dedicated module 112, generating multiple client-dedicated instances 120, wherein each instance 120 is dedicated to at least one client device 106. Then, client-dedicated instance 120 prepares data and sends it as processed or semi-processed data to the corresponding client device 106 for further processing.

[0041] In some embodiments of this disclosure, systems 100a-b can be implemented in a cloud-to-edge infrastructure that can utilize computing devices and systems to demonstrate distributed computing capabilities, such as public or private clouds, fog servers, small clouds, peer-to-peer (P2P) cloud server systems, and edge devices and systems, such as enterprise systems, mobile platforms, and user devices, all of which can be connected via network 108. Using a cloud-to-edge computing network, access to computing power, computer infrastructure (e.g., via so-called Infrastructure as a Service or IaaS), applications, and business processes can be delivered as a service to users on demand via client devices 106. In this way, resources, including physical servers and network devices, can achieve dynamically allocated shared storage and computing. According to embodiments of this disclosure, different layers presented as part of a plurality of n-tier architectures can therefore be hosted in at least some computing devices and systems of the cloud-to-edge infrastructure. For example, resources from one or more client devices can be used as cloud resources in a distributed manner. Thus, for example, in some embodiments of this disclosure, client-specific modules hosted in a second layer of one or more cloud server computers can be hosted in a distributed manner by one or more client devices, for example, in a P2P cloud computer system.

[0042] Figure 1C An embodiment of system 100c implemented in a cloud-to-edge infrastructure including distributed ledger 122 is shown. The cloud-to-edge infrastructure including distributed ledger 122 can be a decentralized and immutable blockchain or distributed ledger network, which facilitates the transmission and storage of data required for the widespread distribution of the n-layer architecture of system 100c.

[0043] In some embodiments, to reduce hardware and network requirements, reduce network latency, and improve user experience, the system can be connected via a network 108 comprising a combination of millimeter wave (mmW) or mmW and sub-6 GHz communication systems, for example, via fifth-generation wireless system communication (5G). In other embodiments, the system can be connected via a wireless local area network (Wi-Fi). The provided communication system can allow low (e.g., about 1-about 5 milliseconds) end-to-end (E2E) latency and high (e.g., 1-10 Gbps) downlink speeds to the field endpoint. This results in high-quality, low-latency, real-time streaming of digital application content. In other embodiments, the system can communicate via fourth-generation wireless system communication (4G), which may be supported by a 4G communication system, or may include other wired or wireless communication systems.

[0044] According to one embodiment, client device 106 includes sensing mechanisms comprising a combination of an inertial tracking sensing mechanism and a transceiver, wherein data tracked by this combination can be used as embedded tracking data or tracking metadata of the sensed dataset. The inertial tracking sensing mechanism may utilize devices such as accelerometers, gyroscopes, magnetometers, etc., which may be integrated into an inertial measurement unit (IMU) or separate from the IMU. In one embodiment, the transceiver is an mmW transceiver. In embodiments employing an mmW antenna, the mmW transceiver is configured to receive mmW signals from the antenna and transmit data back to the antenna. The position tracking provided by the inertial sensor, the mmW transceiver, and the precise tracking, low latency, and high QoS capabilities provided by the mmW-based antenna enable sub-centimeter or sub-millimeter position and orientation tracking, which can improve accuracy when tracking the real-time position and orientation of connected elements. In some embodiments, tracking can be achieved by employing several techniques known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking techniques known in the art (e.g., visual imaging, radar technology, etc.). In an alternative embodiment, the sensing mechanism and transceiver can be coupled together in a single tracking module device.

[0045] In other embodiments, a Global Navigation Satellite System (GNSS) can be used as a satellite-based navigation system to achieve device positioning, such as, for example, GPS, BDS, GLONASS, QZSS, Galileo, and IRNSS. Utilizing signals from a sufficient number of satellites and techniques such as triangulation and trilateration, GNSS can calculate the device's position, velocity, altitude, and time. In one embodiment, the external positioning system is enhanced by assisted GNSS (AGNSS) via an existing cellular communication network architecture, where the existing architecture includes 5G. In other embodiments, the AGNSS tracking system is further supported by a 4G cellular communication network. In indoor embodiments, GNSS is further enhanced via a wireless local area network, such as Wi-Fi, preferably but not limited to providing 16 GHz data. In alternative embodiments, GNSS is enhanced via other techniques known in the art, such as differential GPS (DGPS), satellite-based augmentation systems (SBAS), and real-time kinematics (RTK) systems. In some embodiments, device tracking is achieved through a combination of AGNSS and inertial sensors within the device.

[0046] In some embodiments, client device 106 includes user equipment and other devices. User equipment may include, for example, mobile devices, personal computers, game consoles, media centers, head-mounted displays, and see-through devices (e.g., digital reality contact lenses). Other devices may be specifically configured to capture multi-source data from real-world objects via sensor mechanisms to update and enrich virtual copies of the corresponding real-world objects. According to one embodiment, other devices are one or more surveillance cameras, vehicles, traffic lights, buildings, streets, train tracks, home appliances, Internet of Things (IoT) devices, etc.

[0047] Figure 2 A schematic diagram of a system 200 for optimizing virtual world computing through an n-tier architecture, according to one embodiment, is depicted, including various software and hardware components such as client devices and one or more cloud servers 114. Figure 2 Some components can be with Figure 1A-1C Since the components are the same, similar or identical reference numerals can be used.

[0048] Figure 2 System 200 includes a client device 106 connected to at least one cloud server computer 114. The client device includes a memory 202, at least one processor 204, and at least one video decoder 206. The memory 202 stores a client software engine 104 that receives input data 208 from one or more user interfaces and / or sensor devices, wherein the client software engine is configured to identify portions of the input data for processing by modules and / or layers based on the input data type. Figure 2 In the illustrated embodiment, input data 208 includes a first category (e.g., low-interaction-frequency data 210) and a second category (e.g., high-interaction-frequency data 212). Categorizing input data 208 (e.g., low-interaction-frequency data 210 and high-interaction-frequency data 212) enables the automated and efficient offloading of data and tasks to a second or third layer. Memory 202 also stores client processing logic 214, which includes logical computer instructions for processing at least a portion of the input data 208 via at least one processor 204.

[0049] The cloud server computer 114 of system 200 includes a memory 216, at least one processor 218, and at least one video encoder 220. The memory 216 stores a client-specific module 112 including client-specific processing logic 222, which includes logical computer instructions for processing at least a portion of input data 208 via the at least one processor 218. The memory 216 also includes a virtual world processing module 118, which includes world processing logic 224, which includes logical computer instructions for processing the input data 208 transmitted by the client-specific module 112 via the at least one processor 218. The virtual world processing module 118 also includes a database 226 for storing and organizing data of a virtual world system 228 comprising multiple virtual objects, wherein at least some of the virtual objects are virtual copies of real-world objects.

[0050] In some embodiments, the virtual world processing module 118 is further configured to organize one or more application services through the virtual world system, including, for example, mobility, security, energy management, recommendation, business transactions, etc. The nature of the application services may depend on the specific applications for which the n-tier architecture of system 200 can be used, and their processing may be defined by the virtual world logic 224 of the virtual world processing module 118. Application services may be allocated and sent to the client software engine 104 upon request via the corresponding client-specific module 112, and may be provided in either virtual or augmented reality based on the user's current mode of accessing the virtual world system.

[0051] The organization and provision of application services can be based, at least in part, on spatial contexts within a virtual world system. A spatial context refers to data within a virtual world system that relates to the direct or indirect spatial environment (e.g., taking into account the user's 4D (three-dimensional plus time) coordinates). In some embodiments, a spatial context can be associated with one or more content contexts that determine the type of application to be provided to the user. A content context refers to a theme that may be associated with the spatial context, such as a shopping context, a music context (e.g., a concert), a learning context (e.g., a university or classroom), etc.

[0052] In some embodiments, spatial scenarios are classified as micro-scenarios or macro-scenarios. The term "micro-scenarios" refers to a defined spatial region within a given space and the elements included therein, such as a car, train, company office, building floor, airport, shopping mall, cinema, shop, school, library, restaurant, bar, etc. Micro-scenarios may include data of the defined spatial scenario region, such as 3D image data, 3D geometry, 3D entities, 3D sensor data, 3D moving objects, video data, audio data, text data, time data, metadata, priority data, security data, location data, lighting data, temperature data, Quality of Service (QOS), and scenario-specific applications and application configurations, etc.

[0053] The term "macro-scenario" refers to an indirect or more distant scenario surrounding a single element. A macro-scenario can include or originate from multiple micro-scenarios, generating more comprehensive information about the system; for example, the current efficiency of a manufacturing plant, air quality, climate change levels, company efficiency, city efficiency, national efficiency, etc. Depending on the specified machine learning capabilities and objectives, macro-scenarios can be considered and computed at different levels, including local (e.g., office or manufacturing plant), neighborhood, city, national, and even planetary levels. A macro-scenario including multiple micro-scenarios could be, for example, a car dealership, where each car is a separate micro-scenario. A car dealership macro-scenario can be nested within another macro-scenario; for example, in an industrial complex, the car dealership could represent a micro-scenario within the industrial complex macro-scenario.

[0054] In some embodiments, micro-scenes and macro-scenes can be further categorized into private or public space scenes. A public scene may display all elements publicly visible to all users connected to the virtual world system and receiving services in a given physical or virtual area, while a private scene may involve elements visible only to specific users in a given physical or virtual area. In some embodiments, services related to both public and private scenes are simultaneously delivered to the corresponding client software engine 104 via the client-specific module 112.

[0055] In some embodiments, the spatial context can further determine the level of computing resources required, allocated, and used during the processing of the application service, such as processing power for simulating one or more scenarios, bandwidth for data traffic, and memory usage. For example, the spatial context can define the level of detail (LOD) used when rendering images based on how close or far the user is from specific elements in the real or virtual world (e.g., via world processing logic 224, client-specific module 112, or client software engine 104), which may affect the level of computing resources used during said processing.

[0056] In a further embodiment, based on rights rules, the organization and provision of services in a specific scenario are made available to a particular client software engine through corresponding client-specific modules. Rights rules can be defined based on permissions assigned by a human or artificial system administrator, and can be based on, for example, service contracts or other types of agreements. For example, the right to receive energy management services in a public place may be limited to a specific town management official, while the right to receive energy management services in a specific home may be limited to the corresponding homeowner. In some embodiments, rights rules are at least partially based on priority factors. Priority factors may include assigning different priority scores to different types of users, for example, assigning higher scores to priority users (e.g., premium, premium, or VIP users) and lower scores to standard users. For example, priority-based rights rules could assign a higher Level of Decision (LOD) to priority users than to standard users.

[0057] In further embodiments, the organization and provision of services within a dedicated context are further based on an interest system (e.g., interest graphs, tags, and weights). The interest system can be derived or applied through a machine learning process, implemented by virtual world processing logic 118 via world processing logic 224, and user interaction data can be used to "learn" user behavior patterns used to infer user interests. For example, a restaurant recommendation service using an interest system can provide specific restaurant recommendations to a user based on the types of restaurants the user may have visited in the past. In other examples, user interests can also be explicitly added to a user profile by the corresponding user, or can be retrieved from other sources (e.g., social media) upon user authorization. In further embodiments, any other suitable application service prioritization criteria can be applied.

[0058] In one example, a micro-context (e.g., a car) can trigger the identification of user devices within the car via sensor devices or network detection, sending a signal to the virtual world processing module 118 of the cloud server 114 to provide specific car-related functions that the driver and / or passengers are entitled to within the car micro-context via the corresponding client-specific module 112. In this example, the car micro-context is associated with driving content contexts, meaning that the delivered application services and retrieved content may be driving-related. Furthermore, in this example, some content can be personalized based on interests and rights; for example, a child in the car can watch certain racing game applications while the car is in motion, while the driver (e.g., the father) can receive content related to driving assistance, such as digital viewing or being warned of speed limits, traffic jams, shortcuts, etc. In the example of a shopping mall micro-context, the system 200 can automatically retrieve application content related to brands available near the user from the virtual world processing module 118, which can be categorized by specific brand, clothing type, color, size, etc. In various examples within the entertainment field, given the relevant spatial and content contexts, such as live streaming content for sports events, esports, live music, movies, etc., application services can be displayed to users via client device 106 through system 200, providing content related to the corresponding entertainment venue. Based on the user's specific interests, rights, and specific content context, the provided content can be further modified through system 200. In the example of an airport macro-context, detecting the network of client device 106 can trigger the virtual world processing module 118 of cloud server 114 to provide one or more public airport services in the public context to client software engine 104 via client-specific module 112. However, in a dedicated context, system 200 can consider the data knowledge of users holding digital tickets from specific airlines, which can provide further airline-specific refinement of context application priorities to his or her users in the dedicated context. In the example above, once the application service and initial content are passed to the client software engine 104, the distribution of computational tasks can alternate between various processing modules (client software engine 104, client-specific module 112, or virtual world processing module 118) based on the type of data (e.g., low-interaction-frequency data and high-interaction-frequency data 210 and 212).

[0059] As can be seen from the above examples, system 200 can organize and provide application services related to spatial contexts, which are associated with the physical space of places in micro or macro contexts and the corresponding content contexts. Therefore, places accessed through virtual world system 228 can achieve spatial contextualization of the client experience to a first degree, and further refinement related to the user's personal interests and rights to a second degree, thereby enabling system 200's n-layer architecture to efficiently process highly targeted context-related experiences for users.

[0060] In this disclosure, low-interaction-frequency data 210 refers to data that does not require immediate processing but involves multiple or all users of the virtual world system 228, such as data related to world background, weather, or actions on objects considered to have low priority. High-interaction-frequency data 210 refers to data that requires immediate processing, such as when only minimal latency is desired or required. Therefore, in the example provided above, once the application services and related content are initially passed from the virtual world processing module 118 to the client software engine 104 via the client-specific module 112, the allocation of data processing tasks depends on the type of data to be processed, for example, whether the data is low-interaction-frequency data 210 or high-interaction-frequency data 212.

[0061] In some embodiments, memory 202 / 216 may be any suitable type capable of storing information accessible to processor 204 / 218, including computer-readable media or other media storing data (e.g., input data 208) that can be read by means of electronic devices (e.g., hard disk drives, memory cards, flash drives, ROM, RAM, DVDs or other optical discs) and other writable and read-only memories. In addition to permanent storage, memory 202 / 216 may include temporary storage. Instructions (e.g., client processing logic 214, client-specific processing logic 222, and world processing logic 224) may be executed directly (e.g., machine code) or indirectly (e.g., scripts) by processor 204 / 218. Instructions may be stored in object code format for direct processing by processor 204 / 218, or stored in any other computer language, including scripts or sets of independent source code modules that can be interpreted on demand or pre-compiled. Processor 204 / 218 may retrieve, store, or modify data according to instructions. For example, data may be stored in computer registers, relational databases, tables with multiple different fields and records, XML documents, or flat files, etc. Data can also be formatted in any computer-readable format.

[0062] In some embodiments, at least one processor 218 of at least one cloud server computer 114 is a general-purpose processing unit, a dedicated processing unit, or a combination thereof. At least one processor 218 can be allocated on demand during the instantiation of the virtual world processing module 118 for each client. Allocating at least one processor 218 to an instance of the client-specific module 112 provides more resources available for data processing, thereby delivering higher QoS and quality of experience (QOE) for end users. In one embodiment, in contrast to a typical system that may employ one or more standard central processing units (CPUs) and graphics processing units (GPUs), the dedicated processors 218 are specifically tailored for processing software engines that handle at least one of 3D graphics, artificial intelligence operations, physics operations, 3D structure processing operations, simulation operations, tracking and positioning, and communication. In some embodiments, each processor 218 can be coupled as a chip or system on an electronic chip including one or more specific hardware processing cores designated for processing each specific type of operation. The processing cores of one or more electronic chips can be viewed as functional units or modules of electronic circuitry of the electronic chip, designed to physically execute the algorithms required by each software engine. For example, at least one of the one or more processing cores can be implemented as an FPGA or any other form of integrated circuit that implements dedicated functions. Therefore, the electronic chip can include at least one integrated circuit that implements at least some of the one or more processing cores, wherein at least one processing core implements a different dedicated processor 218. These electronic chips can be mounted on, for example, any one or both of one or more external servers or a main server. In some embodiments, the electronic chip can additionally include general-purpose processing cores, such as a CPU and / or a GPU, or can be connected to an external CPU and / or GPU to provide additional support for each processing task.

[0063] In some embodiments, video encoder 220 may be configured as part of processor 218. Video encoder 220 may include electronic circuitry and / or software configured to convert data into a digital format compatible with client software engine 104 of client device 106. Since a portion of the media being processed and streamed to one or more client devices is a live media stream, there may be very little buffered video frame capacity at the client. This is because when a user issues a command to a client device, the command may need to traverse the network to reach cloud server 114, be processed and optionally rendered by client-specific processing logic 222 and / or world processing logic 224, and then encoded and compressed by video encoder 220 before being streamed back to client device 106. Therefore, the choice of video encoder can be very important for embodiments of this disclosure. In one example, a suitable encoder used in cloud server 114 could be an H.264 / MPEG-4 AVC encoder, which exhibits a high compression ratio and works well under stringent real-time requirements.

[0064] In some embodiments, video decoder 206 may be configured as part of processor 204. Video decoder 206 may include electronic circuitry and / or software configured to convert encoded data into a format useful to client device 106, for example, for any further processing that may be performed by client processing logic 214.

[0065] Figures 3A-3B Schematic diagrams depict data processing flows 300a-b implemented in a system for optimizing virtual world computing through an n-layer architecture, for example, regarding... Figure 1A-2 The system shown. Figures 3A-3B Some components can be with Figure 1A-2 Since the components are the same, similar or identical reference numerals can be used.

[0066] exist Figure 3A In the process, data processing flow 300a begins by inputting low-interaction-frequency data 210 from at least one client device 106 into the client software engine 104. Using client-specific processing logic 222 of the client-specific module 112, the low-interaction-frequency data 210 is transmitted to the virtual world processing module 118. The virtual world processing module 120 then performs any necessary world-related processing 302 on the data, generating data stored in the database 226 and applied to the virtual world system (e.g., ...). Figure 2The virtual world system 228) provides a world state update 304 available to all users. The virtual world processing module 118 then instantiates 306 a client-specific module 112, generating a client-specific instance of the client-specific module 112 dedicated to at least one client device 106. This instance then proceeds to perform data preparation 308 on the world state update 304 sent from the virtual world processing module 118. The prepared data is then encoded 310 and streamed 312 over the network to the client software engine 104. The client software engine 104 continues to decode the prepared data and then performs any necessary end-user processing 316.

[0067] In this disclosure, the term "instantiation" refers to the process of virtualizing server attributes (e.g., memory, computing power, network, operating system, etc.), such as one or more cloud server computers 114, to create an instance of one or more cloud server computers 114. The client-specific module instance is then dedicated to at least one client device 106, creating a dedicated data link between the client software engine 104 and the client-specific module 112 that does not require complex data transfer protocols. This results in lower latency while providing high QoS and QOE for the at least one client device 106 to which the client-specific module 112 is instantiated.

[0068] exist Figure 3B In this embodiment, data processing flow 300b begins by inputting high-interaction-frequency data 212 from at least one client device 106 into the client software engine 104. The high-interaction-frequency data 212 is then sent to the client-specific module 112 of the second layer 110, where it may undergo a data preparation process 308. The prepared data may then undergo encoding 310 and subsequently stream 312 to the client device 106, which can decode the prepared data 314 before performing any required end-user processing 316 via client processing logic 214. Therefore, in this embodiment, since the high-interaction-frequency data 212 is data that requires rapid processing due to the expected very low latency, immediate world processing 302 by the third layer 116 is not required, resulting in a smoother experience including high QoS and QOE.

[0069] In some embodiments, if at least some of the prepared high-interaction-frequency data 212 can affect the virtual world system, for example, affect other elements of the virtual world system and thus affect the experience of other users, then at least some of the prepared high-interaction-frequency data 212 is transmitted to the virtual world processing module 118 to undergo any further necessary world processing 302 to generate a world state update 304 stored in the database 226. The virtual world processing module 118 may continue to perform any instantiation 304 of the client-specific module 112 if necessary, before transmitting the world state update 304 back to the client-specific module 112 for any further data preparation 308 tasks. While the second layer 110 provides the typically faster processing required for the high-interaction-frequency data 210, the data transfer required for parallel or asynchronous processing can be performed in the third layer 116, specifically as in... Figure 3B The dashed data flow arrows are used to represent the data flow.

[0070] In some embodiments, the input data 208 received by the client software engine 104 of the first layer 102 includes one or more of color data, geometric data, depth data, audio data, gesture data, sensor data, and user input commands. In some embodiments, some of the input data 208 is metadata generated by the client device 106 during user interaction.

[0071] In other embodiments, the prepared data from the client-specific module 112 and / or the virtual world processing module 118 includes color data, geometric data, depth data, and audio data, which are streamed to the client software engine 104 of the first layer 102 for implementing any desired end-user processing 316. In some embodiments, some of the prepared data is metadata generated by the client-specific module 112 during data preparation 308 and / or by the virtual world processing module 118 during world processing 302.

[0072] In some other embodiments, the geometric data includes screen-space meshes, proxy geometry, or a combination thereof. Screen-space meshes are implemented using a technique called Screen-Space Mesh (SSM), which generates a mesh in screen space using depth data of a set of particles. Proxy geometry refers to a simplified 3D scene streamed 312 from client-specific module 112 to client software engine 104. A simplified 3D scene is one that, for example, might initially include millions or billions of polygons, but when simplified, may only include hundreds of thousands of polygons, which may be easier to stream to client device 106 for further end-user processing 316.

[0073] In some embodiments, the prepared data, after being encoded 310, streamed 312, and decoded 314, can then be processed by the end user 316. This processing may include performing one or more of the following: generating intermediate frames that increase the final image resolution; providing complementary image frames; interpolating frame rates; extrapolating frame rates; rendering the prepared data; adjusting brightness, color, contrast, sharpness, and resolution; and overlaying image data onto the received data; or combinations thereof. For example, by using a screen space grid and / or geometric proxies, along with prepared color data, depth data, and audio data, the client software engine 104 may have received sufficient prepared data to enable real-time re-creation of the scene by creating intermediate frames. In another example, the generated intermediate frames, through extrapolation and / or interpolation, can be used to increase the final image resolution or frame rate of the scene. All these actions can be performed directly on the prepared data on the client device 106 as post-processing tasks, which may result in efficient utilization of the client hardware while providing high flexibility in the type of real-time adjustments and real-time media content editing, which can occur after cloud processing from second and / or third layers. This may differ from typical systems that employ real-time cloud computing and thin or lightweight client processing, which limit the post-processing and tuning tasks that can be performed on client devices.

[0074] The example three-tier architecture disclosed as part of the broader n-tier architecture of this disclosure reduces the load on the presentation layer, which is typically handled by client devices, by centralizing business logic in the second tier when compared to a conventional three-tier architecture. However, current three-tier architectures typically allow only a limited level of data and task allocation to be offloaded to other tiers, resulting in a lack of high-level flexibility in terms of the processing level executed from one tier to another. Furthermore, during large-scale clustering, conventional three-tier architectures may experience reduced QoS and increased latency due to traffic congestion, as more computational tasks are offloaded to servers, thus limiting scalability for use in large-scale virtual worlds. However, by adding a client-specific module 112 dynamically instantiated 306 for each user by the virtual world processing module 120, and by classifying data into high-frequency and low-frequency data for determining tier partitioning and computational allocation, a more efficient load balancing is achieved between the cloud server 114 and the client device 106, while maintaining a highly centralized computation for each client device 106. This also leads to increased scalability and reusability of the different components in all tiers. Furthermore, it enables better or more efficient use of client hardware because applications can be programmed to perform certain post-processing or other computational tasks based on the physical capabilities of the client hardware. Therefore, the systems and methods disclosed herein enable a scalable, flexible, and efficient architecture that optimizes computation for each user in a virtual world system.

[0075] In some embodiments, reference Figures 3A-3B During application development, developers can adjust the relative amount (e.g., by percentage) of processing performed at each layer via the client software engine 104 of layer 102. In some embodiments, this adjustment of the relative amount of processing is performed by defining the data types (e.g., low-interaction-frequency data 210 and high-interaction-frequency data 212) processed by the client software engine 104 of layer 102, the client-specific module 112 of layer 110, and / or the virtual world processing module 118 of layer 116. In yet other embodiments, most of the processing is performed by the virtual world processing module 118 of layer 116.

[0076] Figures 4A-4C A diagram illustrating the system and methods of this disclosure implemented in an n-tier architecture 400a-c is provided, wherein the n-tier comprises three or more tiers. In the n-tier architecture 400a-c, an xy-axis diagram is shown illustrating the different functions performed by each tier, where the y-axis represents the distance to the client device 402, and the x-axis represents the n-tier 404.

[0077] refer to Figure 4A This illustrates a typical n-tier architecture 400a, showing the different functions performed by each tier. Figure 4A In the example, the first layer is configured to perform post-processing operations 406 on data input on the client device, while the last layer is configured at the far end of the x-axis to perform database management 408. The n-layer architecture 400a also includes other functions 410 located in between, which will refer to Figure 4C To further explain, n layers include any number of necessary layers hosted on any number of corresponding physical or virtual machines connected via a network.

[0078] refer to Figure 4B The n-layer architecture 400b implemented on three layers includes a client software engine 104 in the first layer 102, a client-specific module 112 in the second layer 110, and a virtual world processing module 118 in the third layer 116, which can be as follows: Figures 3A-3B This is implemented using an exemplary embodiment.

[0079] refer to Figure 4C The 400c architecture, implemented on a six-layer basis, uses an n-tier structure where each layer is configured to perform one or more functions on data input from client devices. Figure 4CIn the example, the first layer is configured to perform post-processing operations 406, while the sixth layer is configured at the far end of the x-axis to perform database management 408. The n-layer architecture 400c also includes other functions 410 located in between. For example, the second layer is configured to perform rendering operations 412, the third layer is configured to perform data preparation 308, the fourth layer is configured to perform Internet of Things (IoT) data processing 414, and the fifth layer is configured to perform virtual world processing 302.

[0080] Figure 5 A schematic diagram of a virtual world system 228 according to one embodiment is depicted, which can be stored in a database connected to a virtual world processing module 118 of a third layer 116 (e.g., Figure 2-3B Database 226).

[0081] Virtual world system 228 includes virtual objects 502, wherein at least some of the virtual objects 502 are virtual copies 504 of corresponding real-world elements. Virtual objects 502 may further include applications 506 and purely virtual objects 508, such as objects that do not have real-world counterparts. In some embodiments, virtual world system 228 refers to a persistent virtual world system or a dedicated (e.g., created or processed on demand) virtual world system.

[0082] In this disclosure, the term "persistent" is used to characterize the state of a system that can continue to exist without a continuously executing process or network connection. For example, the term "persistent" can be used to characterize a virtual world system, in which the virtual world system and all the objects contained therein continue to exist independently of the user connected to the virtual world system after the process for creating the virtual objects stops. Therefore, the virtual world system is stored (e.g., on a server) in a non-volatile storage location. In this way, virtual copies, pure virtual objects, and applications can interact and collaborate when configured to achieve a specific goal, even when the user is not connected to the server.

[0083] In some embodiments, the persistent virtual world system 228 may include a virtual world layer 510, which may be divided into an augmented reality (AR) layer 512 and a virtual reality (VR) layer 514. The separate layers enable access to the virtual world system in either augmented reality or virtual reality, and allow the combination of elements stored in each layer, which can be activated via a client device connected to at least one cloud server computer via a network whenever one or the other type of reality is accessed. Each layer may include enhancements to the real and virtual objects 502 for that layer. For example, the AR layer 512 may consist only of virtual copies 504, applications 506, and purely virtual objects 508, which are only visible when accessing the persistent virtual world system 228 in augmented reality. Similarly, the VR layer 514 may consist only of virtual copies 504 and applications 506, which are only visible when accessing the persistent virtual world system 228 in virtual reality. However, the user may decide to activate both layers, or adjust the level of each layer, such that at least some objects in one layer are visible in the other. For example, when accessing a scene in virtual reality, AR layer 512 can be activated, enabling the viewing of enhanced real objects virtualized in VR layer 514, such as viewing further descriptions of buildings or tourist attractions.

[0084] In some embodiments, the virtual world system 228 is further subdivided into multiple sub-universes, each sub-universe including multiple virtual objects 502 specific to that sub-universe. For example, a city sub-universe may be referred to as a city space, which may represent all the buildings, trees, people, streets, traffic lights, vehicles, etc., of the city, as well as their relationships, interactions, and contexts. There may also be a sub-universe within another sub-universe, such as a house space within a city space. A house space may include 3D designs of the house's interior and exterior, all the house's appliances, people, walls, energy consumption data, etc. A network of all sub-universes can form the universe of a persistent virtual world system, which can be viewed in virtual or augmented reality. Each of these sub-universes can be dynamically divided into multiple units, within which resources are dynamically allocated.

[0085] In this disclosure, the term "virtual copy" refers to an accurate and persistent virtual representation of a real-world element. In one embodiment, the virtual copy 504 of this disclosure refers to a virtual version or virtual twin of a real-world element or real twin, which can be drawn or modeled using computer-aided drafting (CAD) or computer-aided engineering (CAE) methods, input through a computer model of a real object (e.g., Building Information Modeling, BIM) or other suitable methods, and which can reflect not only the appearance of the real twin but also its behavior. The real-world object may additionally include sensors that can provide multi-source input data to the virtual copy 504 to enrich the virtual copy 504 and synchronize the virtual copy 504 with its corresponding real-world counterpart. In this disclosure, the term "enrich" is used to describe, for example, the act of providing more attributes to the virtual copy based on multi-source data. For example, enriching a virtual copy may refer to providing real-world data captured from sensing mechanisms on multiple devices, wherein further real-world data includes video data, temperature data, real-time energy consumption data, real-time water consumption data, velocity or acceleration data, etc.

[0086] In some embodiments, virtual copy 504 includes one or more of 3D world and building data, such as SLAM-based or derived mapping data; 3D geometric data; 3D point cloud data; or geographic information system data representing real-world structural properties, which can be used to model 3D structures.

[0087] In some embodiments, application 506 can be one or more traditional applications, distributed applications, or decentralized applications. Traditional applications are typically based on a traditional client-server model and run on dedicated servers in a static infrastructure. Distributed applications are applications primarily stored on cloud computing platforms, such as the cloud servers disclosed herein, and can run simultaneously on multiple systems and devices on the same network, or can run on a blockchain or a distributed database based on a distributed ledger. Decentralized applications primarily run on decentralized infrastructure, such as a blockchain or a distributed database based on a distributed ledger.

[0088] In some embodiments, interaction mechanisms with and between virtual objects 502 are defined using computer code included in computer scripts and computer programs, and can be implemented through applications, smart contracts, or combinations thereof provided in the virtual world system 228 where the interaction and interactive applications occur. Therefore, applications, smart contracts, or combinations thereof can digitally encode and represent possible and desired interactions and interfaces between all elements of the real world through the virtual world system 228. Users can experience the interaction through interactive, game-like applications or game-like interaction mechanisms.

[0089] Figure 6 A schematic diagram depicts a method 600 for optimizing virtual world computing through an n-layer architecture, according to one embodiment. Method 600 can be provided by a system (e.g., Figure 1A-2 The system shown (100a-200) is implemented and can use data processing procedures, for example, Figures 3A-3B Data processing flow 300a-b.

[0090] According to one embodiment, method 600 begins at step 602, where a client-dedicated module in the second layer receives user input data sent by a client software engine configured in the first layer and connected to the second layer via a network. Then, at step 604, the method continues by checking if virtual world processing is required. If virtual world processing module services are required, method 600 continues in step 606, where the client-dedicated module sends the input data to a virtual world processing module in the third layer connected to the client-dedicated module via a network. In some embodiments, the virtual world processing module is configured to process the input data to generate a world state update and dynamically instantiate the client-dedicated module to create a client-dedicated module instance. Then, method 600 continues at step 608, where at least one instance of the client-dedicated module receives the world state update. Method 600 may then end at step 610, where the client-dedicated module further processes the world state update.

[0091] If the virtual world processing module service is not immediately needed, method 600 can proceed directly from step 604 to step 610, where the client-specific module prepares the data received from the client software engine and sends the prepared data to the client software engine to perform the requested end-user processing.

[0092] In some embodiments, method 600 further includes identifying data as high-interaction-frequency data or low-interaction-frequency data through a first-layer client engine, wherein high-interaction-frequency data is processed by a second-layer client-specific module, while low-interaction-frequency data is sent to a virtual world processing module for processing, thereby generating a world state update.

[0093] In some embodiments, method 600 further includes assigning at least one processor to one or more client devices during the instantiation of a client-specific module by the virtual world processing module, wherein at least one processor of at least one cloud server computer includes a general-purpose processing unit, a special-purpose processing unit, or a combination thereof.

[0094] In some embodiments, the input data received by the client software engine includes one or more of color data, geometric data, depth data, audio data, pose data, sensor data, and user input commands. In another embodiment, prepared data from a client-specific module includes color data, geometric data, depth data, and audio data that are streamed to the first-layer client software engine. In a further embodiment, the method further includes encoding the prepared data before streaming it to the client software engine.

[0095] In some embodiments, method 600 further includes the client software engine performing one or more of the following: generating intermediate frames that can increase the resolution of the final image; providing complementary image frames; interpolating frame rates; extrapolating frame rates; rendering prepared data; adjusting brightness, color, contrast, sharpness, and resolution; decoding data; and overlaying image data onto received data; or combinations thereof.

[0096] In some embodiments, method 600 further includes adjusting the percentage of processing performed by each layer via a client software engine during the application development phase, wherein most processing is performed at the third layer.

[0097] While certain embodiments have been described and illustrated in the accompanying drawings, it should be understood that these embodiments are merely illustrative and not limiting of the present disclosure, and that the present disclosure is not limited to the specific constructions and arrangements shown and described, as various other modifications will be apparent to those skilled in the art. Therefore, this description is to be considered illustrative rather than restrictive.

Claims

1. A computer system implementing an n-layer 3D engine architecture, characterized in that, include: The first layer includes a client software engine configured to receive input data, identify the input data as high-frequency data including high-priority data or low-frequency data including low-priority data, send the high-frequency data and the low-frequency data to the second layer, and perform end-user processing of the input data. The second layer is connected to the first and third layers via a network. The second layer includes a client-specific module, which is dynamically instantiated by the third layer for each client device and configured to receive the high-frequency interaction data and the low-frequency interaction data from the first layer, process the high-frequency interaction data, and send the low-frequency interaction data to the third layer. as well as The third layer includes a virtual world processing module configured to receive and process the low-interaction-frequency data from the second layer to generate world state updates and dynamically instantiate the client-specific module to generate client-specific module instances, wherein the world state updates are sent to the corresponding client-specific module instances for further processing.

2. The computer system according to claim 1, characterized in that, The client software engine is stored in the memory of at least one client device and executed by at least one processor of the at least one client device, wherein the client-specific module and the virtual world processing module are stored in the memory of one or more cloud server computers and executed by at least one processor of the one or more cloud server computers.

3. The computer system according to claim 1, characterized in that, The n-layer 3D engine architecture is implemented in a cloud-to-edge infrastructure or distributed ledger employing computing devices and systems, including public or private clouds, fog servers, small clouds, edge devices, or combinations thereof.

4. The computer system according to claim 1, characterized in that, During each client instantiation process implemented by the virtual world processing module, at least one processor is allocated to one or more client devices as needed.

5. The computer system according to claim 1, characterized in that, The input data received by the client software engine includes one or more of color data, geometric data, depth data, audio data, posture data, sensor data, and user input commands.

6. The computer system according to claim 1, characterized in that, The n-layer 3D engine architecture allows for adjustments to the processing volume of each layer via the client software engine.

7. The computer system according to claim 1, characterized in that, The third layer includes a virtual world system, wherein the virtual world system includes multiple sub-universes, each of the sub-universes includes multiple virtual objects, wherein at least some of the multiple virtual objects are virtual copies of corresponding real-world objects located in corresponding regions of the real world, and wherein the virtual world processing module performs processing on multiple virtual objects within regions of interest.

8. The computer system according to claim 7, characterized in that, The virtual world processing module is also configured to provide one or more application services, wherein the provision is at least in part based on spatial scenarios associated with one or more content scenarios through the virtual world system.

9. The computer system according to claim 8, characterized in that, The spatial scenarios are classified as micro-scenarios or macro-scenarios.

10. A method for optimizing virtual world computation through an n-layer architecture, characterized in that, include: The client-dedicated module of the second layer of the n-layer architecture receives high-frequency interaction data, including high-priority data, and low-frequency interaction data, including low-priority data, from the client software engine configured in the first layer of the n-layer architecture, wherein the first layer is connected to the second layer via a network. The client-dedicated module transmits the low-interaction-frequency data to a third-layer virtual world processing module of the n-layer architecture connected to the client-dedicated module via the network. The virtual world processing module is configured to process the low-interaction-frequency data to generate a world state update and dynamically instantiate the client-dedicated module to generate a client-dedicated module instance, wherein the world state update is sent to the corresponding client-dedicated module instance. The world state update is received by the client-specific module; and The world state update is processed by the client-specific module, wherein the processing is performed on a per-client basis; The processed world state update is sent to the client software engine to perform the requested end-user processing; and The client-specific module prepares the high-frequency interaction data received from the client software engine and sends the prepared data to the client software engine to perform the requested end-user processing.

11. The method according to claim 10, characterized in that, The client software engine is stored in the memory of at least one client device and executed by at least one processor of the at least one client device, wherein the client-specific module and the virtual world processing module are stored in the memory of one or more cloud server computers and executed by at least one processor of the one or more cloud server computers.

12. The method according to claim 10, characterized in that, It also includes allocating at least one processor to one or more client devices during the instantiation of the client-specific module executed by the virtual world processing module.

13. The method according to claim 10, characterized in that, The input data received by the client software engine includes one or more of the following: color data, geometric data, depth data, audio data, posture data, sensor data, and user input commands.

14. The method according to claim 10, characterized in that, This also includes adjusting the amount of processing performed at each layer via the client software engine during the application development phase.

15. The method according to claim 11, characterized in that, The third layer includes a virtual world system, wherein the virtual world system includes multiple sub-universes, each of the sub-universes includes multiple virtual objects, wherein at least some of the multiple virtual objects are virtual copies of corresponding real-world objects located in corresponding regions of the real world, and wherein the virtual world processing module performs processing on multiple virtual objects within regions of interest.

16. The method according to claim 15, characterized in that, It also includes one or more application services provided by the virtual world processing module, wherein the provision is at least in part based on spatial scenarios associated with one or more content scenarios through the virtual world system, and wherein the application services are allocated and sent to the corresponding client software engine as required by the corresponding client-specific module.

17. The method according to claim 16, characterized in that, The spatial scenarios are classified as micro-scenarios or macro-scenarios.

18. A non-transitory computer-readable medium, characterized in that, It has instructions stored thereon, the instructions being configured to cause one or more computers to perform the following steps, including: The client-dedicated module of the second layer of the n-layer architecture receives high-frequency interaction data, including high-priority data, and low-frequency interaction data, including low-priority data, from the client software engine configured in the first layer of the n-layer architecture, wherein the first layer is connected to the second layer via a network. The client-dedicated module sends the low-interaction-frequency data to the third layer of the n-layer architecture virtual world processing module connected to the client-dedicated module via the network. The virtual world processing module is configured to process the low-interaction-frequency data to generate world state updates and dynamically instantiate the client-dedicated module to generate client-dedicated module instances, wherein the world state updates are sent to the corresponding client-dedicated module instances. The world state update is received by the client-specific module; and The world state update is processed by the client-specific module, wherein the processing is performed on a per-client basis; The processed world state update is sent to the client software engine to perform the requested end-user processing; and The client-specific module prepares the high-frequency interaction data received from the client software engine and sends the prepared data to the client software engine to perform the requested end-user processing.

Images