Location-based platform for multiple 3D engines for delivery of location-based 3D content to user

Abstract

A location-based platform for hosting a plurality of 3D engines that deliver location-based 3D content. The engine platform server system is configured to host one or more engine servers hosting a 3D engine, and to provide an engine platform including a digital reality application developed via the 3D engine. The digital reality application is configured in a predetermined location within the permanent virtual world system or in a mobile virtual copy of a respective real world object. A client device in proximity to the digital reality application sends the viewing location and orientation along with view requests to an engine platform server, which requests virtual frames from respective 3D engines, synthesizes each virtual frame into one view, and sends the resulting media stream to the user via the client device.

Description

This application is a divisional application of the patent application filed on June 18, 2020, with application number 202010559611.5 and invention name “Location-based platform with multiple 3D engines for delivering location-based 3D content to users”. CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Provisional Application No. 62 / 863,090, filed June 18, 2019, the entire disclosure of which is incorporated herein for all purposes. Technical Field

[0002] The present disclosure relates generally to computer systems and, more particularly, to systems and computer-implemented methods that enable delivery of location-based 3D content. Background Art

[0003] Current technological trends include the development of digital reality, which collectively encompasses augmented reality, virtual reality, and mixed reality technologies. The impact of these technologies can be considered a breakthrough that will revolutionize entertainment, learning, finance, healthcare, machinery, and other industries. Digital reality applications are developed using 3D engines that provide a purpose-built environment to implement these application-specific functions.

[0004] However, developers may currently be limited to selecting only one 3D engine at a time, with no current alternative to choosing from among many 3D engines within a single market. Similarly, end users of digital reality applications do not have access to a platform that includes multiple digital reality applications that are visible and accessible to them regardless of which 3D engine was used during their development. For example, a user may need to first download a digital reality application before being able to receive the media stream corresponding to the application. In this sense, a user may not be able to view a digital representation of the application in the real world before downloading and installing a particular digital reality application.

[0005] Therefore, there is a need for improvements in the way 3D engines and digital reality applications are provided to users. Summary of the Invention

[0006] This Summary is provided to introduce selected concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] One or more of the shortcomings described in the background or other technical problems are addressed by the systems and methods of the present disclosure, which include a location-based platform for hosting multiple 3D engines that deliver location-based 3D content. The system and method provide an engine platform hosted on an engine platform server that enables developers to select from a variety of engines (e.g., game engines) that can be used to create digital reality applications and locate them in predetermined real-world locations available in a persistent virtual world system, which can be stored in a database or data structure. The system and method also enables end users to view and interact with digital reality applications at predetermined locations, where the end user receives a synthetic view of a virtual frame with digital content from the digital reality application that has been calculated by one or more 3D engines. The resulting synthetic view is a reflection of the digital reality application developed via one or more game engines, which, in some embodiments, are provided to the user in real time and at an appropriate frame rate (e.g., within about 60 to about 120 frames per second (FPS)).

[0008] In one embodiment, a system for enabling a location-based platform for hosting multiple 3D engines for delivering location-based 3D content includes an engine platform server system comprising one or more server computers configured to host one or more engine servers (e.g., engine servers provided by a third party) and provide an engine platform comprising one or more digital reality applications developed via one or more 3D engines hosted on the engine servers or engine platform servers. The digital reality applications are associated with (e.g., virtually attached to) predetermined locations within a persistent virtual world system or a mobile virtual copy of a corresponding real-world object. In one embodiment, one or more client devices are connected to the engine platform server system via a network and configured to provide the digital reality applications to a user, thereby enabling the user to interact with the digital reality applications.

[0009] In one embodiment, the engine platform system is configured to receive a viewing position and orientation and a corresponding view request from a client device as the client device approaches and renders an initial view of a digital reality application. The engine platform server system is configured to, upon receiving the view request, request virtual frames from different 3D engines, combine the virtual frames into a media stream view, and send the media stream view to the client device.

[0010] In some embodiments, the system may use a cloud and edge infrastructure that can implement distributed computing capabilities, including employing public or private clouds, cloudlets, and edge systems such as enterprise systems, mobile platforms, and user devices.

[0011] In some embodiments, one or more 3D engines are local 3D engines hosted in an engine platform server. In other embodiments, one or more 3D engines are third-party 3D engines hosted in one or more third-party engine servers. After a predetermined time or after reaching a specified level of interactivity, the client device may also need to download a runtime instance of one or more 3D engines and applications, thereby switching from server rendering to local rendering. One or more digital reality applications can be created using one or more corresponding 3D engines. Therefore, when accessing the engine platform, application developers can seamlessly view and select from a variety of 3D engines.

[0012] The 3D engine available at the engine platform can provide a specially created environment to implement functions specific to 3D digital reality applications. Thus, the 3D engine performs tasks that enable aspects such as the management of animated models, conflicts between objects, the behavior and interaction of light with individual objects, and the interaction between objects of the scene, as well as the interaction between users and applications. The 3D engine may include, for example, a physics engine that simulates the laws of physics within a virtual environment, an audio engine that adds music and complex acoustic effects, or an artificial intelligence (AI) engine that programs the intelligent behavior of a computer. The 3D engine can be used in any type of application that requires rendering 3D graphics with real-time performance, including applications in virtual reality (VR), augmented reality (AR), mixed reality (MR), or a combination thereof.

[0013] In some embodiments, digital reality applications are virtually attached to one or more predetermined 3D locations selected within a persistent virtual world system available from an engine platform server, or to mobile virtual replicas whose positions can change based on the movement of corresponding real-world elements. The predetermined 3D locations or mobile virtual replicas are selected during the development of the digital reality applications. When a user approaches one or more digital reality applications, the digital reality applications broadcast a proximity-based signal to the client device, alerting the client device to the digital reality application's presence within proximity. Upon receiving this signal, the client device proceeds to send a viewing position and orientation, along with a corresponding view request, to the engine platform server. Upon receiving the view request, the engine platform server requests virtual frames from one or more 3D engines used in the development of the one or more digital reality applications, and then proceeds to composite each virtual frame into a single view. Thus, the virtual frames are composed of data segments collected from one or more 3D engines, as the user may be viewing an area that includes more than one digital reality application developed by several different 3D engines. The engine platform server then transmits the composite view to the client device, enabling smooth, natural, low-latency, and real-time viewing and interaction with the one or more digital reality applications.

[0014] According to one embodiment, a digital reality application may utilize one or more of a variety of techniques to broadcast a signal to a client device, alerting the client device that the digital reality application is available near the client device's location. In one embodiment, if a user has previously subscribed to the engine platform service, the client device's location may be available in the persistent virtual world system stored on the engine platform server. Therefore, when the user approaches the digital reality application, the digital reality application may already be ready to broadcast a signal to the client device. In another embodiment, if the user has previously subscribed to the engine platform server or has subscribed to one or more specific digital reality applications from the engine platform server, the digital reality application may continuously search for registered devices before broadcasting a signal. In another embodiment, the digital reality application may broadcast a signal when it detects a device within a predefined proximity threshold in a predefined geographic location. For example, the predefined geographic location may be a building or a room within a building. In another example, the predefined geographic location may be within a city block. In another example, the predefined geographic location may be within a city. In one embodiment, when a user with a client device enters a location where the digital reality application is active, the digital reality application may detect a signal from the device indicating that the device is available to receive the digital reality application media stream before broadcasting the signal.

[0015] In this disclosure, the term "virtual frame" refers to one of many elements (e.g., still images) that make up a virtual animation, which can be included in a media stream that transmits digital content of a digital reality application. For example, a virtual frame can be a virtual sprite or a two-dimensional bitmap that is integrated into a larger scene. When displaying a virtual animation, each virtual frame is flashed to a user at a predetermined position for a short period of time. The position and orientation of the virtual frame displayed to the user depends on the user's viewing position and orientation, which can be sensed by a sensor installed in the client device.

[0016] The plurality of virtual frames may be composited to generate a unified media stream to a client device that can be viewed and interacted with by a user. The media stream may include digital reality content including 3D image data, 3D geometry, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, textual data, time data, position data, orientation data, haptic data, and lighting data.

[0017] In some embodiments, synthesizing virtual frames to generate a single media stream view for a user may include sorting virtual frames or virtual frame segments in 3D space; culling virtual frames in 3D space (e.g., viewport culling); depth masking the virtual frames to achieve partial occlusion of the virtual frames based on real-world elements; requesting components of the virtual frames; and combining the virtual frames into a single view. Alternatively, synthesizing the virtual frames may include any combination of fewer or additional steps. In some embodiments, synthesizing the virtual frames is performed independently of the frame rate at which the virtual frames are received. In one embodiment, the synthesized virtual frames are sent to the user via the client device at a native frame rate of 60 FPS, 90 FPS, 120 FPS, another frame rate between 60 and 120 FPS, or some other frame rate. Within this frame rate range, the user can experience the perception of receiving and consuming the media stream, and thereby the experience from the digital reality application, in real time. In some embodiments, the third-party engine server may render the corresponding portions of the virtual frames as they are sent by the third-party engine server, such that the virtual frames received by the engine platform server are already rendered.

[0018] In some embodiments, classification of the virtual frame in 3D space involves arranging different elements of the virtual frame to specific positions and orientations in 3D space so that the media stream produced by the animated virtual frame can be adjusted to the user's viewing position and orientation.

[0019] In some embodiments, viewport culling of a virtual frame in 3D space can remove unnecessary elements from the viewing frustum of the virtual frame. These elements can be considered unnecessary because, for example, they may be completely outside the viewing frustum of the virtual frame or they may be blocked by real-world elements.

[0020] In some embodiments, depth masking refers to the process of partially blocking portions of a virtual frame, particularly portions located in the background that may affect the depth of an image displayed on the virtual frame. Depth masking can be performed using several techniques known in the art, such as layer masking, clipping masking, and alpha channel masking. In some embodiments, depth masking is performed on geometry already available in a database or data structure stored in the engine platform server system and / or from virtual copies of real-world elements stored in the persistent virtual world system. In other embodiments, depth masking is performed on geometry generated in real time by the engine platform server.

[0021] In some embodiments, requesting components of a virtual frame refers to retrieving actual content of the virtual frame corresponding to the digital reality application that will be part of the unified media stream.

[0022] In some embodiments, combining virtual frames into a media stream view is performed by one or more of warping, stitching, and interpolating multiple virtual frames.

[0023] According to one embodiment, the digital content sent to the virtual frame by one or more engines includes a 3D icon. In one embodiment, the 3D icon includes one or more voxels, static 3D representations, dynamic 3D representations, or a combination thereof, as determined during the development phase of the digital reality application, which are used to graphically represent the 3D application. A 3D icon can be distinguished from a 3D object in that a 3D icon is a basic graphical representation of an application, while a 3D object represents a more immersive experience as the user reaches a deeper level of engagement with the application. In one embodiment, the 3D object includes a mathematical model of the graphical representation.

[0024] In some embodiments, the virtual frame may include elements rendered by the engine platform server. In other embodiments, the virtual frame may include elements rendered by a third-party engine server. In other embodiments, the virtual frame may include elements rendered by a client device. In other embodiments, the virtual frame may include a mixture of elements rendered by the engine platform server, the third-party engine server, and the client device.

[0025] As an example, the virtual frames are initially rendered by the engine platform server or a third-party engine server, synthesized by the engine platform server, and output by the client device. In this example, the client device only needs to perform lightweight operations on the media stream. In a further example, the digital reality application is configured to be downloaded and executed locally by the client device. In an illustrative scenario, as the user's engagement with the digital reality application increases, the client device can proceed to fully download and install the digital reality application and its 3D objects, as well as a runtime instance of the corresponding 3D engine, and perform local rendering of the received media stream, so that all or most of the virtual frames are rendered locally while executing the application. In other embodiments, after a predetermined amount of time has passed since the user viewed or engaged in the digital reality application, the client device can continue to fully download the digital reality application and perform rendering locally while executing the application.

[0026] Digital reality applications are visual and can be interacted with via client devices, which may include one or more mobile devices, personal computers, game consoles, smart contact lenses, media centers, and head-mounted displays. The client device may be equipped with sensors to determine the relative position and orientation of the client device (three coordinates) and the relative orientation of the headset relative to the viewer (three angles). In some embodiments, this tracking information provides the client device with six degrees of freedom of position and orientation, which can determine how to generate output streams from multiple virtual frames.

[0027] In an embodiment, in order to reduce hardware and network requirements, help reduce network latency, and improve the general digital reality experience, the system can be connected through a network including millimeter wave (mmW) or a combination of mmW and sub 6GHz communication systems, such as through fifth-generation wireless system communication (5G). In other embodiments, the system can be connected through a wireless local area network (Wi-Fi) that provides data at 60GHz. The communication system provided can allow low (e.g., about 1 to about 5 milliseconds) end-to-end (E2E) latency and high (e.g., 1-10Gbps) downlink speeds to on-site endpoints, meeting the parameters required to execute typical highly interactive digital reality applications. This results in high-quality, low-latency, real-time digital application content streaming. In other embodiments, the system can be connected through a fourth-generation wireless system communication (4G) communication, can be supported by a 4G communication system, or can include other wired or wireless communication systems.

[0028] According to one embodiment, a sensing mechanism installed on a client device includes a combination of an inertial tracking sensing mechanism and a transceiver. The inertial tracking sensing mechanism may utilize devices such as accelerometers and gyroscopes, which may be integrated into an inertial measurement unit (IMU). The transceiver may be implemented to transmit and receive wireless communication signals to and from an antenna. In an embodiment, the transceiver is a mmW transceiver. In embodiments employing mmW antennas, the mmW transceiver is configured to receive mmW signals from the antenna and transmit data back to the antenna. The inertial sensors, along with the position tracking provided by the mmW transceiver and the precise tracking, low latency, and high quality of service (QoS) capabilities provided by the mmW-based antennas, can achieve sub-centimeter or sub-millimeter position and orientation tracking, which can increase accuracy when tracking the real-time position and orientation of connected elements. In some embodiments, tracking can be implemented using a variety of techniques known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking techniques known in the art (e.g., visual imaging, radar technology, etc.). In alternative embodiments, the sensing mechanism and transceiver may be coupled together in a single tracking module device. The sensing mechanism of the client device may also include one or more cameras. For example, the camera may be a depth camera installed in the client device. The camera may be configured to capture and provide the user's viewing position and orientation, which determines the viewing position and orientation of the virtual frame sent via the engine platform server.

[0029] Providing accurate tracking of connected elements can help display the reliable state of client devices within the persistent virtual world system, particularly their position and orientation, which can be relevant to various applications. In addition, achieving accurate, real-time tracking of client devices can reduce the need to physically sense other client devices.

[0030] According to an embodiment, a method for delivering location-based 3D content includes the following steps: an engine platform server system receives a client device position and orientation and a view request associated with a digital reality application; the engine platform server system requests virtual frames from one or more 3D engines; the engine platform server system receives the virtual frames sent by the one or more 3D engines; the engine platform server system combines the virtual frames into a media stream view of the digital reality application; and sends the synthesized view to the client device for viewing and interaction with the user.

[0031] According to an embodiment, synthesis of virtual frames includes classifying virtual frames in 3D space; culling virtual frames in 3D space; depth masking the virtual frames to achieve partial occlusion of the virtual frames based on real-world elements; requesting components of the virtual frames; and combining the virtual frames into a view.

[0032] According to one embodiment, the method further includes switching from server rendering to local rendering (e.g., switching from server-based rendering of a 3D icon to local rendering of a 3D object). The switching can be based on the engagement time or engagement level achieved by the user with the digital reality application. The engine platform server receives a 3D icon from the 3D engine; renders the 3D icon and sends the 3D icon to the virtual frame; the engine platform server synthesizes the virtual frame to obtain a media stream with the 3D icon; sends the media stream with the 3D icon to the client device for local processing, such as performing lightweight tasks on the media stream, and displays it to the user; checks whether sufficient time has passed or whether a deeper engagement level has been reached, wherein, for a negative case where a threshold time has not passed or a threshold level of engagement has not been reached, the method can loop back to receiving the 3D icon from the 3D engine; and in a positive case where a threshold time has passed or a threshold level of engagement has been achieved, or a combination thereof, authorizes or instructs the client device to download and install the digital reality application including the 3D object and the corresponding runtime instance of one or more 3D engines.

[0033] From the perspective of a client device, the disclosed method includes downloading and installing, by the client device, a digital reality application including a 3D object, and a corresponding runtime instance of one or more 3D engines; the client device locally rendering the 3D object; and the client device executing the digital reality application including a media stream with the 3D object. According to one embodiment, depending on the duration or level of interaction between the user and the digital reality application, rendering may be performed in part by an engine server (e.g., an engine platform server or a third-party engine server) or by the client device. When rendering is performed by a mix of engine servers and client devices, the unified media stream view may include 3D icons and 3D objects. In other embodiments, rendering may be switched back and forth between the server and the client depending on the quality of service (QoS) available in the network.

[0034] According to an optional embodiment of the transition from server rendering of 3D icons to local rendering of 3D objects, the client device can also be configured to synthesize a virtual frame after receiving the rendered 3D icon from the engine platform server. Comparing the virtual frames synthesized by the server with the virtual frames synthesized by the client, the former case can achieve a standard but less flexible frame rate for the media stream displayed to the user and lower bandwidth and client device computing requirements, while the latter case can provide higher frame rate flexibility for the media stream displayed to the user and higher bandwidth and client device computing requirements.

[0035] Also disclosed is a computer-readable medium having stored thereon instructions configured to cause one or more computing devices (eg, a server system or a client device) to perform the methods disclosed herein.

[0036] The above summary of the invention does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects outlined above, as well as those disclosed in the following detailed description and those specifically identified in the claims filed with this application. Such combinations have specific advantages not specifically stated in the above summary of the invention. Other features and advantages will be apparent from the accompanying drawings and from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The specific features, aspects, and advantages of the present disclosure will be better understood with reference to the following description and accompanying drawings, in which:

[0038] Figure 1 Depicted is a schematic representation of a system for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content, according to an embodiment.

[0039] Figure 2Depicted is an isometric schematic representation of a system for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content, showing a real-world view of the location-based platform with multiple applications, in accordance with an embodiment.

[0040] Figures 3A-3B Depicted is a schematic representation of a virtual frame according to an embodiment.

[0041] Figure 4 A schematic representation of a system for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content is shown, showing a transition from server-rendered 3D icons to client-rendered 3D objects, according to an embodiment.

[0042] Figures 5A-5B Depicted are schematic representations of server-rendered 3D icons and client-rendered 3D objects, according to an embodiment.

[0043] Figure 6 Shown is a schematic representation of a client device according to an embodiment.

[0044] Figure 7 A method for enabling a location-based platform for hosting multiple 3D engines for delivering location-based 3D content is shown in accordance with an embodiment.

[0045] Figure 8 A method for synthesizing a virtual frame according to an embodiment is described.

[0046] Figure 9 A method for transitioning from a server-rendered 3D icon to a client-rendered 3D object according to an embodiment is described.

[0047] Figure 10 Another method for transitioning from a server-rendered 3D icon to a client-rendered 3D object according to an embodiment is shown. DETAILED DESCRIPTION

[0048] In the following description, reference is made to the accompanying drawings which illustrate various embodiments. In addition, various embodiments will be described below with reference to a number of examples. It should be understood that the embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.

[0049] Figure 1 Depicted is a schematic representation of a system 100 for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content, according to an embodiment.

[0050] System 100 includes an engine platform server 102 configured to host one or more third-party engine servers 104 and provide an engine platform 106. Engine platform 106 includes one or more digital reality applications 108 developed via one or more 3D engines, which may be third-party 3D engines 110 or native 3D engines 112.

[0051] As an example, in Figure 1 In the illustration of , digital reality application AD is developed via a third-party 3D engine 110, and digital reality application E is developed via a local 3D engine 112. More specifically, digital reality application AB is developed via a third-party 3D engine hosted by engine server A, digital reality application C is developed via a 3D engine B hosted by engine server B, digital reality application D is developed via a 3D engine C hosted by engine server C, and digital reality application E is developed by a local 3D engine 112 hosted by the engine platform server 102.

[0052] The digital reality applications 108 are anchored to predetermined locations within the persistent virtual world system or to mobile virtual replicas whose locations can change based on the movement of corresponding real-world elements. Thus, a user 114 near one of the predetermined locations can use a client device 116 to view one or more digital reality applications 108 independent of the 3D engine used during development. The client device 116 is connected to the engine platform server 102 via a network 118 and is configured to provide the digital reality applications 108 to the user 114, thereby enabling user interaction. In further embodiments, when accessing the engine platform 106, an application developer 120 is able to view and select from a variety of third-party 3D engines 110 or native 3D engines 112 to develop digital reality applications 108 using the client device 116.

[0053] In the present disclosure, the term "persistent" is used to describe a state of a system that can continue to exist in the absence of a continuously executing process or a network connection. For example, the term "persistent" can be used to describe a virtual world system, wherein the virtual world system and all virtual copies, purely virtual objects, and digital reality applications included therein continue to exist after the process used to create the virtual copies, purely virtual objects, and digital reality applications is stopped, and does not rely on a user being connected to the virtual world system. Therefore, the virtual world system is saved in a non-volatile storage location, such as in server 102. In this way, even if a user is not connected to the server, the virtual copies, purely virtual objects, and digital reality applications can interact and collaborate with each other when configured to achieve a specific goal.

[0054] The presently disclosed system 100 can be implemented in a cloud-to-edge infrastructure that can employ public or private clouds, fog servers, distributed ledger infrastructure, and edge devices and systems such as enterprise systems, mobile platforms, and user devices to expose distributed computing capabilities, all of which can be connected via a network. Using a cloud-to-edge computing network, access to computing power, computer infrastructure (e.g., through so-called Infrastructure as a Service or IaaS), applications, and business processes can be delivered to users as a service on demand via client devices. In this way, resources including physical servers and network devices enable shared storage and compute that can be dynamically allocated based on factors such as the user's distance to the resources and network, and the computing demands from the user.

[0055] The third-party 3D engine 110 available at the engine platform 106 can provide a specially created environment to implement functions specific to the digital reality application 108. Thus, the 3D engine is configured to perform tasks that enable aspects such as the management of animated models, collisions between objects, the behavior and interaction of light with individual objects, and the interaction between objects of the scene and the interaction between the user and the application. The 3D engine can include, for example, a physics engine that simulates the physical laws within the virtual environment, an audio engine that adds music and complex acoustic effects, and an artificial intelligence (AI) engine that programs the intelligent behavior of the computer. The 3D engine can be used in any type of application that requires rendering 3D graphics with real-time performance, including applications in virtual reality (VR), augmented reality (AR), mixed reality (MR), or a combination thereof.

[0056] In some embodiments, one or more digital reality applications 108 are developed via one or more 3D engines using the systems and methods described in U.S. Patent Application No. 16 / 421,155 filed by Jewat Yearley on May 23, 2019, entitled “Method and System for Developing, Testing, and Deploying Digital Reality Applications into the Real World via a Virtual World,” which is incorporated herein by reference in its entirety. In such embodiments, a system and method for developing, testing, and deploying digital reality applications into the real or virtual world is provided by integrating a digital reality application development system and an application management system within a digital reality ecosystem. In the digital reality application development system, an application developer creates a digital reality application; configures the application, including location and space (i.e., real or virtual location, 3D positioning and scaling), time, and target user parameter settings; and tests the application in a developer space, a final real location for augmented reality, or a virtual / hybrid space for virtual / mixed reality. In the application management system, the application administrator tests the application; approves / rejects the application; sends an adjustment request to the application developer when necessary; and after the adjustment, if compliant, deploys the application in the online application store for user access, whereby the application can function based on the previously defined and approved content and settings.

[0057] In some embodiments, a persistent virtual world system, stored in the engine platform server 102 and capable of selecting real-world locations for digital reality applications, is created via systems and methods for developing and deploying virtual replicas of real-world elements into the persistent virtual world system. Virtual replica creation is performed within a virtual environment using a replica editor that enables the development and configuration of virtual replicas to reflect the behavior and appearance of corresponding real-world elements. The virtual replicas are enriched with data captured by sensing mechanisms that synchronize the virtual replicas with the real-world elements. In one embodiment, virtual replicas are shared within an ongoing virtual world system quality assessment, where they can be approved or rejected for subsequent adjustments, if necessary. Following approval and deployment, the replicas are shared within the deployed persistent virtual world system, which is visible to end users for managing and interacting with the virtual replicas.

[0058] In an embodiment, to reduce hardware and network requirements, help reduce network latency, and improve the general digital reality experience, the system 100 can be connected via a network 118 including a millimeter wave (mmW) or a combination of mmW and sub 6 GHz communication systems, such as via fifth-generation wireless system communications (5G). In other embodiments, the system can be connected via a wireless local area network (Wi-Fi) that provides data at 60 GHz. The provided communication system can allow low (e.g., about 1 to about 5 milliseconds) end-to-end (E2E) latency and high (e.g., 1-10 Gbps) downlink speeds to on-site endpoints, meeting the parameters required to execute typical highly interactive digital reality applications 108. This results in high-quality, low-latency, real-time digital application content streaming. In other embodiments, the system can be connected via a fourth-generation wireless system communication (4G) communication, can be supported by a 4G communication system, or can include other wired or wireless communication systems.

[0059] The digital reality application 108 can be viewed by and interacted with a client device 116 via the client device 116, which may include one or more mobile devices, personal computers, game consoles, media centers, smart contact lenses, and head-mounted displays. The client device 116 can be equipped with sensors to determine the relative position and orientation of the client device 116 (three coordinates) and the relative orientation of the headset relative to the viewer (three angles). This tracking information is equivalent to 6 degrees of freedom of the client device 116, which can determine how to generate an output stream from multiple virtual frames.

[0060] In some embodiments, system 100 may utilize cloud and edge infrastructure that can implement distributed computing capabilities, including employing public or private clouds, micro clouds, and edge systems such as enterprise systems, mobile platforms, and user devices (e.g., client device 116).

[0061] Figure 2 Depicted is an isometric schematic representation of a system 200 for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content, showing a real-world view of the location-based platform with multiple digital reality applications, according to an embodiment. Figure 2 Some components can be used with Figure 1 Elements of the present invention are similar, and thus similar or identical reference numerals may be used to describe those elements.

[0062] Reference Figure 2, a digital reality application 108 (e.g., digital reality applications AE developed by third-party 3D engines 110A-C and a local 3D engine 112, respectively) is associated with (e.g., virtually attached to) one or more predetermined 3D locations 202 in a real-world location area 204 stored within a persistent virtual world system at an engine platform server. The predetermined 3D locations 202 are selected during development of the digital reality application.

[0063] When a digital reality application 208 is virtually attached to a real-world element, such as a specific 3D location, the digital reality application 208 can be positioned in space, for example, in tandem with the element to which it is virtually attached. If the digital reality application 208 is not attached, the application can be positioned in space independent of the location of any object. If the digital reality application 208 is detached from the element to which it is virtually attached, the application can be removed from the space where the element still exists, remain in a fixed position independent of the location of the object, or move independently of the object.

[0064] In one embodiment, when a user 114 employing a client device 116 approaches one or more digital reality applications 108 in one of the predetermined 3D locations 202, the digital reality applications 108 broadcast a proximity-based signal to the client device 116, thereby notifying the client device 116 of the presence of the nearby digital reality applications 108. Upon receiving this signal, the client device 116 proceeds to send a viewing position and orientation, along with a corresponding view request, to the engine platform server. Upon receiving the view request, the engine platform server requests virtual frames 206 from one or more third-party 3D engines 110 or native 3D engines 112 used in the development of the one or more digital reality applications 108, and then proceeds to composite each virtual frame 206 into a media stream view that is sent to the client device 116. Thus, the virtual frames 206 are composed of data bits collected from one or more 3D engines, as the user 114 may be viewing an area encompassing multiple applications developed using several different 3D engines. The engine platform server then transmits the composite view to the client device 116, enabling smooth, natural, low-latency, and real-time viewing and interaction with the one or more digital reality applications.

[0065] like Figure 2As shown, each of the digital reality applications 108 can be visually represented by a 3D icon, such as a rectangular prism representing digital reality applications 108A and E, a sphere representing digital reality applications B and D, and a pyramid representing digital reality application C. Because all digital reality applications 108 are available at the 3D engine platform, and because all virtual frames 206 corresponding to the visual representations and media content of the 3D engine are synthesized by the engine platform server, the user 114 can seamlessly and continuously view these visual representations of the digital reality applications through the client device 116 regardless of the 3D engine used during their development. Figure 2 , application AC can be included within the viewing cone 208 of user 114 and its virtual frame 206 can be composited to user 114 and output through client device 116, even though application AB was developed via 3D engine A and application C was developed via 3D engine B.

[0066] According to one embodiment, digital reality application 108 may utilize one or more of a variety of techniques to broadcast a signal to client device 116, thereby alerting client device 116 that digital reality application 108 is available nearby. In one embodiment, if user 114 has previously subscribed to the engine platform service, the location of client device 116 is always available in the persistent virtual world system available on the engine platform server. Therefore, when user 114 approaches digital reality application 108, digital reality application 108 may already be ready to broadcast a signal to client device 116. In another embodiment, if user 114 has previously subscribed to the engine platform service or has subscribed to one or more specific digital reality applications 108 from the engine platform service, digital reality application 108 may continuously search for registered devices before broadcasting a signal. In another embodiment, digital reality application 108 may broadcast a signal when it detects client device 116 within a predetermined proximity threshold of a predetermined 3D location 202. For example, predetermined 3D location 202 may be a building or a room within a building. In another example, predetermined 3D location 202 may be within a city block. In another example, predetermined 3D location 202 may be within a city. In one embodiment, when user 114 with client device 116 enters a location where a digital reality application is activated, the digital reality application may detect a signal from the device indicating that client device 116 may receive the digital reality application media stream before broadcasting the signal.

[0067] In this disclosure, the term "virtual frame" refers to one of many image elements (e.g., still images) that make up a virtual animation, which can be included in a media stream that conveys the digital content of a digital reality application 108. For example, a virtual frame 206 can be a virtual sprite integrated into a larger scene, or a two-dimensional bitmap. When displaying a virtual animation, each virtual frame 206 is briefly flashed to the user at a predetermined 3D position 202. The position and orientation at which the virtual frame 206 is displayed to the user depends on the user's viewing position and orientation, as sensed by the client device 116.

[0068] Multiple virtual frames 206 may be synthesized to generate a single media stream to client device 116 that can be viewed and interacted with by user 114. The media stream may include digital reality content, including 3D image data, 3D geometry, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, text data, time data, position data, orientation data, haptic data, and lighting data. The calculations used to synthesize virtual frames 206 may be optimized to only consider applications within the viewing cone 208 of user 114. For example, in Figure 2 , only applications AC may be included within the viewing frustum 208 of the user 114 , and applications DE are culled because they are outside the viewing frustum 208 . Figure 3A -B depicts a schematic representation of a virtual frame 206 according to an embodiment. Figure 3A - Some components of B can be Figure 1-2 Elements of the present invention are similar, and thus similar or identical reference numerals may be used to describe those elements.

[0069] like Figure 3A As shown, a virtual frame 206 is located at the real-world location area 204 and is generated and populated with content from each of the third-party 3D engine 110 and the native 3D engine 112. Depending on the user's viewing frustum, the virtual frame 206 may be independently generated by a separate 3D engine, which may be the case if the user directs his or her view towards a separate digital reality application. However, as Figure 3B As shown, the virtual frame 206 may also be generated by more than one 3D engine, which may be the case if a user directs his or her view toward more than one digital reality application.

[0070] Figure 4 Depicted is a schematic representation of a system 400 for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content, showing a transition from a server-rendered 3D icon 402 to a client-rendered 3D object 404, according to an embodiment. Figure 4 Some components can be used with Figure 1-3BElements of the present invention are similar, and thus similar or identical reference numerals may be used to describe those elements.

[0071] According to one embodiment, the digital content sent by the engine server 406 to the virtual frame 206 includes a 3D icon 402, which may be one or more voxels, a static 3D representation, a dynamic 3D representation, or a combination thereof determined via the 3D engine 408 during the development phase of the digital reality application.

[0072] In some embodiments, the virtual frame 206 of the media stream may include elements rendered by the engine platform server. In other embodiments, the virtual frame 206 of the media stream may include elements rendered by a third-party engine server. In other embodiments, the virtual frame 206 of the media stream may include elements rendered by the client device 116. In other embodiments, the virtual frame 206 of the media stream may include a mix of elements rendered by the engine platform server, the third-party engine server, and the client device 116.

[0073] In one embodiment, reference Figure 4 In the first column from left to right, the virtual frame 206 of the media stream 410 with the 3D icon is initially rendered by the engine platform server or by a third-party engine server. Figure 4 The engine platform server 406 is grouped into an engine server; the engine platform server synthesizes; and is streamed to the client device 116, which outputs a media stream 410 with a 3D icon. In this embodiment, the client device 116 only needs to perform lightweight operations on the media stream 410 with a 3D icon.

[0074] In a further embodiment, and with reference to Figure 4 In the second column, as time and / or user engagement 412 with the digital reality application increases, the client device 116 can proceed to fully download and install the digital reality application, after which the client device 116 can proceed to execute the digital reality application.

[0075] Furthermore, in these examples, and with reference to Figure 4 In the third column, after the digital reality application has been completely downloaded and installed in the client device 116, the 3D icon 402 can be converted into a 3D object 404, which is rendered locally and sent to the client device as a media stream with the 3D object 414. The client device can then finally execute the digital reality application locally.

[0076] Figure 5A -B depicts a schematic representation of a server-rendered 3D icon and a client-rendered 3D object according to an embodiment. Figure 5A - Some components of B can be Figure 1-4Elements of the present invention are similar, and thus similar or identical reference numerals may be used to describe those elements.

[0077] Reference Figure 5A , a user 114 using a client device 116 located at a real location area 204 can view one or more visual representations of a digital reality application at a predetermined 3D location 202, which digital reality application can include one or more server-rendered 3D icons 502, client-rendered 3D objects 504, or a combination thereof contained within the viewing frustum 208 of the user 114.

[0078] Reference Figure 5B , the virtual frames synthesized into the media stream view and output via the client device may include virtual frames 502a-c. Virtual frame 502a may include a server-rendered 3D icon 502; virtual frame 502b may include a mixture of server-rendered 3D icon 502 and client-rendered 3D object 504; and virtual frame 502c may include a purely client-rendered 3D object 504.

[0079] Figure 6 A schematic representation of a client device 116 is shown, according to an embodiment.

[0080] The client device 116 may include operational components such as an input / output (I / O) module 602 ; a power supply 604 ; a memory 606 ; a sensor 608 and a transceiver 610 forming a tracking module 612 ; and a network interface 614 , all operably connected to a processor 616 .

[0081] The I / O module 602 is implemented as computing hardware and software that is configured to interact with a user and provide user input data to one or more other system components. For example, the I / O module 602 can be configured to interact with a user, generate user input data based on the interaction, and provide user input data to the processor 616 before the user input data is transmitted to other processing systems (such as to a server) via a network. In another example, the I / O module 602 is implemented as an external computing pointing device (e.g., a touch screen, a mouse, a 3D control, a joystick, a game controller, etc.) and / or a text input device (e.g., a keyboard, a dictation tool, etc.) configured to interact with the client device 116. In other embodiments, the I / O module 602 can provide additional, less, or different functions than those described above.

[0082] The power supply 604 is implemented as computing hardware and software configured to provide power to the client device 116. In one embodiment, the power supply 604 can be a battery. The power supply 604 can be built into the device or can be removed from the device and can be rechargeable or non-rechargeable. In one embodiment, the device can be repowered by replacing one power supply 604 with another power supply 604. In another embodiment, the power supply 604 can be recharged by a cable attached to a charging source, such as a Universal Serial Bus ("USB") FireWire, Ethernet, Thunderbolt interface, or headphone cable attached to a personal computer. In yet another embodiment, the power supply 604 can be recharged by inductive charging, wherein an electromagnetic field is used to transfer energy from the inductive charger to the power supply 606 when the inductive charger and the power supply 604 are in close proximity but do not need to be plugged into each other via a cable. In another embodiment, a docking station can be used to facilitate charging.

[0083] The memory 606 may be implemented as computing hardware and software suitable for storing application instructions. The memory 606 may be of any suitable type capable of storing information accessible by the processor 616, including computer-readable media, or other media that stores data that can be read by an electronic device, such as a hard drive, memory card, flash drive, ROM, RAM, DVD or other optical disk, and other writable and read-only memory. In addition to persistent storage, the memory 606 may also include temporary storage.

[0084] The sensing mechanism can be implemented as computing hardware and software that is suitable for obtaining data from the real world and determining / tracking the position and orientation of the client device 116, and sending this information to the engine platform server to determine the position and orientation of the virtual frame and synthetic media stream view sent to the client device. For example, the sensor 608 may include one or more cameras, such as one or more depth cameras. The sensor may also include one or more inertial measurement units (IMUs), accelerometers, and gyroscopes. The IMU is configured to measure and report the velocity, acceleration, angular momentum, translational velocity, rotational velocity, and other telemetry metadata of the client device 116 by using a combination of accelerometers and gyroscopes. The accelerometer within the IMU and / or configured separately from the IMU can be configured to measure the acceleration of the interactive device, including the acceleration caused by the earth's gravitational field. In one embodiment, the accelerometer includes a three-axis accelerometer capable of measuring acceleration in three orthogonal directions.

[0085] The transceiver 610 may be implemented as computing hardware and software configured to enable the device to receive radio waves from an antenna and transmit data back to the antenna. In some embodiments, a mmW transceiver may be employed that is configured to receive mmW wave signals from the antenna and transmit data back to the antenna when interacting with immersive content. The transceiver 610 may be a two-way communication transceiver 610.

[0086] In an embodiment, the tracking module 612 may be implemented by combining the capabilities of an IMU, accelerometer, and gyroscope with the position tracking provided by the transceiver 610, and the precise tracking, low latency, and high QOS capabilities provided by the mmW-based antenna may enable sub-centimeter or sub-millimeter position and orientation tracking, which may increase accuracy when tracking the real-time position and orientation of the client device 116. In an alternative embodiment, the sensing mechanism and transceiver 610 may be coupled together in a single tracking module device.

[0087] Network interface 614 may be implemented as computing software and hardware that communicatively connects to a network, receives computer-readable program instructions from the network sent by a server or by client device 116 , and forwards the computer-readable program instructions for storage in memory 606 for execution by processor 616 .

[0088] The processor 616 can be implemented as computing hardware and software configured to receive and process sensor and digital reality application data and instructions. For example, the processor 616 can be configured to provide imaging requests, receive imaging data, process the imaging data into environmental or other data, process user input data and / or imaging data to generate user interaction data, perform machine learning training and reasoning based on the edge (device), provide server requests, receive server responses, and / or provide user interaction data, environmental data, and content object data to one or more other system components. For example, the processor 616 can receive user input data from the I / O module 602 and can respectively implement the application stored in the memory 606. In other examples, the processor 616 can receive data captured from the real world from a sensing mechanism, or can receive the precise location and orientation of the client device 116 through the tracking module 612, and can prepare some data before sending the data to the server for further processing. In other examples, the processor 616 can perform edge-based rendering of the media stream received from the engine platform server while executing the digital reality application. In other examples, the processor 616 may receive a media stream rendered by an engine platform server and may perform lightweight operations on the media stream in order to output the media stream.

[0089] Figure 7A block diagram of a method 700 for enabling a location-based platform for hosting multiple 3D engines for delivering location-based 3D content is shown in accordance with an embodiment. The method 700 may be implemented by a system such as a system described in reference Figure 1-6 Depicted system.

[0090] Method 700 may begin at blocks 702 and 704 by providing an engine platform server configured to host one or more third-party engine servers and provide an engine platform, the engine platform further configured to provide a digital reality application developed via one or more 3D engines residing in the third-party engine server or the engine platform server. Method 700 continues at block 706 by providing one or more client devices connected to the engine platform server via a network and configured to output the digital reality application and provide a user interface capable of interacting with the user. Subsequently, method 700 proceeds at block 708 by receiving, by the engine platform server, a client device position and orientation and a view request associated with the digital reality application sent by the client device. This may occur when a user employing the client device approaches a predetermined location where the digital reality application has been located during development.

[0091] The method 700 continues in step 710 by the engine platform server requesting virtual frames from one or more 3D engines, and then in step 712 by receiving the virtual frames sent by the one or more 3D engines. The method 700 then continues in step 714 by compositing the virtual frames into a media stream view, and finally by sending the composited media stream view to a client device for viewing and user interaction, as shown in block 716, before concluding the process in terminator 718.

[0092] Figure 8 Describes a method for synthesizing a reference according to an embodiment Figure 7 A block diagram of a method 800 for implementing a virtual frame. The method 800 may be implemented by a system such as a reference system. Figure 1-6 Depicted system.

[0093] In some embodiments, the virtual frame is synthesized independently of the frame rate of the received frame and sent to the user via the client device at a natural frame rate of 60 FPS, 90 FPS, 120 FPS, another frame rate between 60 and 120 FPS, or some other frame rate. Within this frame rate range, the user can perceive the media stream as being received, and therefore receive the experience from the digital reality application, and can participate in real time. In some embodiments, when the third-party engine server sends corresponding portions of the virtual frame, the third-party engine server can render these virtual frame portions so that the virtual frame received by the engine platform server has been rendered.

[0094] Method 800 begins in blocks 802 and 804 by classifying a virtual frame in 3D space. In some embodiments, the classification of the virtual frame in 3D space involves arranging different elements of the virtual frame into specific positions and orientations in the 3D space so that the media stream generated by the animated virtual frame can be adjusted to the user's viewing position and orientation.

[0095] The method 800 continues at block 806 by performing viewport culling of the virtual frame in 3D space to remove unnecessary elements from the viewing frustum of the virtual frame. These elements may be considered unnecessary because, for example, they may be completely outside the viewing frustum of the virtual frame or they may be blocked by real-world elements.

[0096] Subsequently, method 800 proceeds in block 808 by performing depth masking of the virtual frame to achieve partial blocking of the virtual frame based on real-world elements. In some embodiments, depth masking refers to the process of performing partial blocking of some portions of the virtual frame, particularly portions located in the background that may affect the depth of the image displayed on the virtual frame. Depth masking can be performed using several techniques known in the art, such as layer masking, clipping masking, and alpha channel masking. In some embodiments, depth masking is performed on geometry that is already available from virtual copies of real-world elements stored in the persistent virtual world system. In other embodiments, depth masking is performed on geometry generated in real time by the engine platform server.

[0097] The method 800 proceeds in block 810 by requesting components of a virtual frame, which may involve retrieving actual content of the virtual frame corresponding to the digital reality application that will be part of the unified media stream.

[0098] Method 800 then continues at block 812 by combining the virtual frames into a single view. In some embodiments, combining the virtual frames into a single view is performed by warping, stitching, interpolating, and / or performing other operations on one or more of the multiple virtual frames. Some of these techniques are described in detail in U.S. Patent Application No. 15 / 764,696, filed by Jewat Yearley on March 29, 2018, which is incorporated herein by reference in its entirety. Combining the virtual frames can be a more complex reconstruction process based on the input virtual frames. For example, the process can rely on a combination of standard image reconstruction techniques, such as stitching, warping, interpolation, and extrapolation. For example, extrapolation may be required to fill gaps or holes in the media data in areas where no or limited (visual) information is available based on the available virtual frames. However, it should be understood that the reconstruction process is not limited to computer vision techniques and can further consider spatial data about the scene, which can include any combination of reconstructed 3D geometry, parameters about materials, and a light field, which can correspond to optical flow in the captured scene. In some embodiments, the spatial data can be used to re-render the combined virtual frames using 3D rendering techniques. In one or more embodiments, the generation of the output stream may include the use of deep learning techniques and / or neural networks, which may be used to recreate virtual frames of the output stream from a sequence of virtual frames of a media stream of the same scene taken from different viewpoints. This may enable complex reconstruction and generation of the output stream even if at least a portion of the scene is not captured completely or in full detail.

[0099] Method 800 may end in terminator 814 .

[0100] Figure 9 A block diagram of a method 900 according to an embodiment is shown, which details the conversion between a server-rendered 3D icon and a client-rendered 3D object. The method 900 may be implemented by a system such as a reference system. Figure 1-6 Depicted system.

[0101] Method 900 begins at blocks 902 and 904 by receiving a 3D icon from a 3D engine via an engine platform server. The 3D engine may be a local 3D engine hosted on the engine platform server or a third-party 3D engine hosted on one or more third-party engine servers. Method 900 continues at block 906 by rendering the 3D icon by the engine platform server or by the third-party engine server and sending the 3D icon to a virtual frame.

[0102] Subsequently, method 900 continues at block 908 by compositing a virtual frame and obtaining a media stream with a 3D icon by the engine platform server, which is performed by the engine platform server. At block 910, method 900 sends the media stream with the 3D icon to the client device, which performs lightweight tasks on the media stream to display it to the user. Method 900 then checks at decision block 912 whether sufficient time has elapsed, whether a deeper level of engagement has been reached, or a combination thereof. If not, method 900 loops back to block 904 by receiving the 3D icon from the 3D engine. If yes, method 900 continues at block 914 by downloading and installing a digital reality application including the 3D object and corresponding runtime instances of one or more engines hosted on the engine platform server by the client device, and then continues at block 916 by locally rendering the 3D object. At block 918, method 900 continues by executing the digital reality application including the media stream with the 3D object by the client device, before concluding the process at terminator 920.

[0103] According to one embodiment, depending on the duration or level of interaction between the user and the digital reality application, or a combination thereof, rendering can be performed in part by an engine server (e.g., an engine platform server or a third-party engine server) or by a client device. When rendering is performed by a mix of engine servers and client devices, the unified media stream view can include 3D icons and 3D objects. In other embodiments, the transition can switch back and forth between the server and the client based on the quality of service (QoS) available in the network.

[0104] Figure 10 A block diagram of method 1000 is described, which details an optional embodiment of a transition between a server-rendered 3D icon and a client-rendered 3D object. The transition can be based on the engagement time or engagement level achieved by the user with the digital reality application. In other embodiments, the transition can switch back and forth between the server and the client based on the quality of service (QoS) available in the network. Method 1000 can be implemented by a system, such as reference Figure 1-6 Depicted system.

[0105] Method 1000 begins at blocks 1002 and 1004 by receiving a 3D icon from a 3D engine by the engine platform server. The 3D engine may be a local 3D engine hosted on the engine platform server or a third-party 3D engine hosted on one or more third-party engine servers. Method 1000 continues at block 1006 by rendering the 3D icon by the engine platform server or by the third-party engine server and sending the 3D icon to a virtual frame.

[0106] Method 1000 then continues at block 1008 by sending the rendered virtual frame to the client device. Thereafter, at block 1010, method 1000 continues by the client device synthesizing the virtual frame and obtaining a unified media stream with the 3D icon. Method 1000 then continues by the client device performing lightweight tasks on the unified media stream to display the media stream to the user, as shown in block 1012. At decision block 1014, method 1000 checks whether sufficient time has elapsed, whether a deeper level of engagement has been reached, or a combination thereof. In the negative case, method 1000 loops back to block 1004 by receiving the 3D icon from the 3D engine. In the positive case, method 1000 continues at block 1016 by the client device downloading and installing the digital reality application including the 3D object and the corresponding runtime instance of one or more engines hosted on the engine platform server, and then by locally rendering the 3D object at block 1018. In block 1020 , method 900 continues by executing, by the client device, a digital reality application including a media stream with a 3D object, and finally ends the process in terminator 1022 .

[0107] Will refer to Figure 9 The server of the method 900 synthesizes the virtual frame and the reference Figure 10 Compared with the client-synthesized virtual frames of method 1000, the former case can achieve a standard but less flexible frame rate of the media stream displayed to the user as well as lower bandwidth and client device computing requirements, while the latter case can provide higher frame rate flexibility of the media stream displayed to the user as well as higher bandwidth and client device computing requirements.

[0108] Although certain embodiments have been described and shown in the accompanying drawings, it should be understood that such embodiments are merely illustrative and not restrictive of the broad invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art. Accordingly, this description is to be regarded as illustrative and not restrictive.

Claims

1. A system for delivering 3D content, characterized in that: The system comprises: an engine platform comprising one or more digital reality applications hosted in an engine platform server system; The engine platform server system is configured to receive a viewing position, orientation, and corresponding view request from a client device, wherein the engine platform server system is configured to, after receiving the view request, request virtual frames corresponding to the digital reality application from one or more 3D engines, and synthesize the virtual frames into a media stream view to be sent to the client device; and Wherein, one or more of the 3D engines include an artificial intelligence (AI) engine to program computer intelligent behavior.

2. The system according to claim 1, wherein: Combining the virtual frames into the one media stream view includes: Classifying virtual frames in 3D space; Perform viewport culling of virtual frames in 3D space; performing depth masking of a virtual frame to achieve partial occlusion of the virtual frame based on real-world elements; requesting the contents of the virtual frame; and The virtual frames are combined into one view.

3. The system according to claim 2, characterized in that The depth masking is performed on geometry that is already available in a database or data structure stored in the engine platform server system, or is performed on geometry that is generated in real time by the engine platform server system.

4. The system according to claim 1, wherein: The engine platform system hosts one or more engine servers, and one or more of the engine servers hosts one or more 3D engines, and the one or more 3D engines are configured to develop one or more of the digital reality applications included in the engine platform.

5. The system according to claim 4, characterized in that The one or more 3D engines include a local 3D engine hosted in the engine platform server system, or a third-party 3D engine hosted in one or more third-party engine servers.

6. The system according to claim 1, wherein: The digital reality application is virtually attached to one or more predetermined 3D locations in a real-world area, or is virtually attached to a moving virtual copy of a real-world object.

7. The system according to claim 1, wherein: At least one of the digital reality applications is configured to be downloaded and executed locally by the client device after one or more of a predetermined engagement time or level is reached.

8. A method for delivering location-based 3D content, characterized in that The method comprises: receiving, by an engine platform server system from a client device, a viewing position and orientation and a corresponding view request, wherein the engine platform includes one or more digital reality applications hosted in the engine platform server system; After receiving the view request, the engine platform server system requests a virtual frame corresponding to the digital reality application from one or more 3D engines; The engine platform server system synthesizes the virtual frames into a media stream view; The engine platform server system sends the media stream view to the client device; and Wherein, one or more of the 3D engines include an artificial intelligence (AI) engine, and the AI ​​engine is configured to program computer intelligent behavior.

9. The method according to claim 8, characterized in that The synthetic virtual frame includes: Classifying virtual frames in 3D space; Perform viewport culling of virtual frames in 3D space; performing depth masking of a virtual frame to achieve partial occlusion of the virtual frame based on real-world elements; requesting the contents of the virtual frame; and The virtual frames are combined into one view.

10. The method according to claim 9, characterized in that The depth masking is performed on geometry that is already available in a database or data structure stored in the engine platform server, or is performed on geometry that is generated in real time by the engine platform server system.

11. The method according to claim 8, characterized in that The engine platform system hosts one or more engine servers, wherein one or more engine servers host one or more 3D engines, and the one or more 3D engines are configured to develop one or more digital reality applications included in the engine platform.

12. The method according to claim 11, characterized in that The one or more 3D engines include a local 3D engine hosted in the engine platform server, or a third-party 3D engine hosted in one or more of the engine servers.

13. The method according to claim 8, characterized in that The virtual frames are synthesized by the engine platform server system regardless of the frame rate at which the frames are received.

14. The method according to claim 8, characterized in that Compositing the virtual frame further includes transitioning from server rendering of the 3D icon to local rendering of the 3D object by: Receiving, by the engine platform server system, the 3D icon from one or more of the 3D engines; Rendering the 3D icon by the engine platform server system or by the engine server, and sending the 3D icon to the virtual frame; The engine platform server system synthesizes the virtual frame and obtains a media stream having the 3D icon; as well as The media stream with the 3D icon is sent to the client device for local processing.

15. The method according to claim 14, characterized in that Further including: authorizing the client device to download and install at least one of the digital reality applications including the 3D object and corresponding runtime instances of one or more of the 3D engines if the client device has engaged in at least one of the one or more digital reality applications for at least a threshold amount of time or a threshold level of engagement has been reached by the client device with at least one of the digital reality applications, or a combination thereof; locally rendering the 3D object by the client device; The client device synthesizes the virtual frame and obtains a media stream having a 3D object corresponding to at least one digital reality application; as well as At least one of the digital reality applications including a media stream having a 3D object is executed by the client device.

16. The method according to claim 15, characterized in that Switching from server rendering of the 3D icon to local rendering of the 3D object is based on an engagement time, engagement level, or a combination thereof achieved by a user of the client device with at least one of the digital reality applications.

17. The method according to claim 15, characterized in that Depending on the duration of or the level of interaction between a user of the client device and at least one of the digital reality applications, the rendering may be performed in part by the engine server or by the client device, wherein the synthesized virtual frame and unified media stream view include 3D icons and 3D objects.

18. A computer-readable medium having instructions stored thereon, characterized in that The instructions are configured to cause the server system to perform the following steps: receiving, by an engine platform server system from a client device, a viewing position and orientation and a corresponding view request, wherein the engine platform includes one or more digital reality applications hosted in the engine platform server system; After receiving the view request, the engine platform server system requests a virtual frame corresponding to the digital reality application from one or more 3D engines; The engine platform server system synthesizes the virtual frames into a media stream view; The engine platform server system sends the media stream view to the client device; and Wherein, one or more of the 3D engines include an artificial intelligence (AI) engine, and the AI ​​engine is configured to program computer intelligent behavior.

19. The computer-readable medium of claim 18, wherein: The one or more 3D engines include a local 3D engine hosted in the engine platform server, or a third-party 3D engine hosted in one or more of the engine servers.

20. The computer-readable medium of claim 18, wherein The engine platform system hosts one or more engine servers, wherein one or more engine servers host one or more 3D engines, and the one or more 3D engines are configured to develop one or more digital reality applications included in the engine platform.

21. The computer-readable medium of claim 18, wherein: Compositing virtual frames involves switching from server rendering to local rendering.

Images