Location-based platform of multiple 3D engines for delivering location-based 3D content to users

Abstract

A location-based platform for hosting multiple 3D engines that deliver location-based 3D content. The engine platform server system is configured to host one or more engine servers that host 3D engines, and is configured to provide an engine platform that includes digital reality applications developed via 3D engines. The digital reality applications are configured in a predetermined location within a permanent virtual world system or in a mobile virtual copy of a corresponding real-world object. A client device approaching the digital reality application sends a viewing position and orientation along with a view request to the engine platform server, which requests a virtual frame from the corresponding 3D engine, synthesizes each virtual frame into a view, and sends the resulting media stream to the user via the client device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Provisional Application No. 62 / 863090, filed June 18, 2019, the entire disclosure of which is incorporated herein for all purposes. Technical Field

[0003] The present disclosure relates generally to computer systems, and more particularly to systems and computer-implemented methods that enable delivery of location-based 3D content. Background Art

[0004] Current technology trends include the development of digital reality, which collectively includes augmented reality, virtual reality, and mixed reality technologies. The impact of these technologies can be considered a breakthrough that will revolutionize entertainment, learning, finance, medicine, machinery, and other industries. Digital reality applications are developed via 3D engines that provide a specially created environment to implement the functions specific to these applications.

[0005] However, developers may currently be limited to selecting only one 3D engine at a time, with no current alternative to selecting among many 3D engines within a single market. Likewise, end users of digital reality applications do not have access to a platform that includes multiple digital reality applications that are visible and accessible to them regardless of which 3D engine was used during their development. For example, a user may need to first download a digital reality application before being able to receive a media stream corresponding to the application. In this sense, a user may not be able to view a digital representation of an application in the real world before downloading and installing a particular digital reality application.

[0006] Therefore, there is a need for improvements in the way 3D engines and digital reality applications are provided to users. Summary of the invention

[0007] This Summary is provided to introduce selected concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0008] One or more disadvantages described in the background technology or other technical problems are solved by the system and method of the present disclosure, which includes a location-based platform for hosting multiple 3D engines that deliver location-based 3D content. The system and method provide an engine platform hosted in an engine platform server, which enables developers to select from a variety of engines (e.g., game engines) that can be used to create digital reality applications and locate them in a predetermined real-world location available in a persistent virtual world system, which can be stored in a database or data structure. The system and method also enable end users to view and interact with digital reality applications at a predetermined location, where the end user receives a synthetic view of a virtual frame with digital content from a digital reality application that has been calculated by one or more 3D engines. The resulting synthetic view is a reflection of a digital reality application developed via one or more game engines, and in some embodiments, the one or more game engines are provided to the user in real time and at an appropriate frame rate (e.g., within about 60 to about 120 frames per second (FPS)).

[0009] In one embodiment, a system for enabling a location-based platform for hosting multiple 3D engines for delivering location-based 3D content includes an engine platform server system, the engine platform server system including one or more server computers, the engine platform server system configured to host one or more engine servers (e.g., engine servers provided by a third party) and provide an engine platform, the engine platform including one or more digital reality applications developed via one or more 3D engines hosted in the engine servers or engine platform servers. The digital reality applications are associated with (e.g., virtually attached to) predetermined locations within a persistent virtual world system or a mobile virtual copy of a corresponding real world object. In one embodiment, one or more client devices are connected to the engine platform server system via a network and are configured to provide the digital reality application to a user, thereby enabling the user to interact with the digital reality application.

[0010] In one embodiment, the engine platform system is configured to receive a viewing position and orientation and a corresponding view request from a client device as the client device approaches and renders an initial view of a digital reality application. The engine platform server system is configured to request virtual frames from different 3D engines upon receiving the view request, combine the virtual frames into a media stream view, and send the media stream view to the client device.

[0011] In some embodiments, the system may use a cloud and edge infrastructure that can implement distributed computing capabilities, including employing public or private clouds, cloudlets, and edge systems such as enterprise systems, mobile platforms, and user devices.

[0012] In some embodiments, one or more 3D engines are local 3D engines hosted in an engine platform server. In other embodiments, one or more 3D engines are third-party 3D engines hosted in one or more third-party engine servers. After a predetermined time or reaching a specified level of interaction, the client device may also need to download a runtime instance of one or more 3D engines and applications, thereby switching from server rendering to local rendering. One or more digital reality applications can be created through one or more corresponding 3D engines. Therefore, when accessing the engine platform, application developers can seamlessly view and select from a variety of 3D engines.

[0013] The 3D engine available at the engine platform can provide a specially created environment to implement functions dedicated to 3D digital reality applications. Therefore, the 3D engine performs tasks that enable aspects such as the management of animation models, conflicts between objects, the behavior and interaction of light with individual objects, and the interaction between objects of the scene and the interaction between users and applications. The 3D engine can include, for example, a physics engine that simulates the physical laws within a virtual environment, an audio engine that adds music and complex acoustic effects, or an artificial intelligence (AI) engine that programs the intelligent behavior of computers. The 3D engine can be used in any type of application that requires rendering 3D graphics with real-time performance, including applications in virtual reality (VR), augmented reality (AR), mixed reality (MR), or a combination thereof.

[0014] In some embodiments, the digital reality application is virtually attached to one or more predetermined 3D locations selected within a persistent virtual world system available at an engine platform server, or to a mobile virtual copy whose position can change according to the movement of the corresponding reality element. The predetermined 3D location or mobile virtual copy is selected during the development of the digital reality application. When the user approaches one or more digital reality applications, the digital reality application broadcasts a proximity-based signal to the client device so that the client device knows the presence of the digital reality application in proximity. Upon receiving the signal, the client device proceeds to send the viewing position and orientation and the corresponding view request to the engine platform server. Upon receiving the view request, the engine platform server requests a virtual frame from one or more 3D engines used in the development of one or more digital reality applications, and then proceeds to synthesize each virtual frame into a view. Therefore, the virtual frame is composed of data segments collected from one or more 3D engines, because the user may be viewing an area that includes more than one digital reality application developed by several different 3D engines. The engine platform server then sends the synthesized view to the client device, which enables smooth, natural, low latency and real-time views and interactions with one or more digital reality applications.

[0015] According to one embodiment, the digital reality application can utilize one or more of a variety of technologies to broadcast a signal to the client device, thereby alerting the client device that the digital reality application is available near the location of the client device. In one embodiment, if the user has previously subscribed to the engine platform service, the location of the client device may be available at the persistent virtual world system stored in the engine platform server. Therefore, when the user approaches the digital reality application, the digital reality application may be ready to broadcast a signal to the client device. In another embodiment, if the user has previously subscribed to the engine platform server or subscribed to one or more specific digital reality applications from the engine platform server, the digital reality application can continuously search for registered devices before broadcasting the signal. In another embodiment, when the digital reality application detects a device within a predefined proximity threshold in a predefined geographic location, the digital reality application can broadcast a signal. For example, the predefined geographic location can be a building or a room in a building. In another example, the predefined geographic location can be within a city block. In another example, the predefined geographic location can be within a city. In one embodiment, when a user with a client device enters a location where the digital reality application is activated, the digital reality application can detect a signal from the device, indicating that the device can receive the digital reality application media stream before broadcasting the signal.

[0016] In the present disclosure, the term "virtual frame" refers to one of many elements (e.g., still images) that make up a virtual animation, which can be included in a media stream that transmits digital content of a digital reality application. For example, a virtual frame can be a virtual sprite integrated into a larger scene, or a two-dimensional bitmap. When displaying a virtual animation, each virtual frame is flashed to a user at a predetermined position for a short period of time. The position and orientation of the virtual frame displayed to the user depends on the user's viewing position and orientation, which can be sensed by a sensor installed in a client device.

[0017] The plurality of virtual frames may be synthesized to generate a unified media stream to a client device that may be viewed and interacted with by a user. The media stream may include digital reality content including 3D image data, 3D geometry, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, text data, time data, position data, orientation data, tactile data, and lighting data.

[0018] In some embodiments, synthesizing virtual frames to generate a single media stream view for a user may include classifying virtual frames or virtual frame fragments in 3D space; culling (e.g., viewport culling) virtual frames in 3D space; depth masking virtual frames to achieve partial blocking of virtual frames based on real-world elements; requesting components of virtual frames; and combining virtual frames into a view. Alternatively, the synthesis of virtual frames may include fewer or additional steps in any combination. In some embodiments, the synthesis of virtual frames is performed independently of the frame rate at which the virtual frames are received. In one embodiment, the synthesized virtual frames are sent to the user via a client device at a natural frame rate of 60 FPS, 90 FPS, 120 FPS, another frame rate between 60 and 120 FPS, or some other frame rate. Within this frame rate range, the user can get the feeling that the media stream and the experience from the digital reality application are received and used in real time. In some embodiments, when the third-party engine server sends the corresponding parts of the virtual frame, the third-party engine server can render these virtual frame parts so that the virtual frame received by the engine platform server has been rendered.

[0019] In some embodiments, classification of the virtual frame in 3D space involves arranging different elements of the virtual frame to specific positions and orientations in 3D space so that the media stream to be generated by the animated virtual frame can be adjusted to the user's viewing position and orientation.

[0020] In some embodiments, viewport culling of a virtual frame in 3D space can remove unnecessary elements from the viewing frustum of the virtual frame. These elements can be considered unnecessary because, for example, they can be completely outside the viewing frustum of the virtual frame, or they can be blocked by elements of the real world.

[0021] In some embodiments, depth masking refers to the process of partially blocking some parts of the virtual frame, especially parts located in the background that may affect the depth of the image displayed on the virtual frame. Several techniques known in the art can be used to perform depth masking, such as layer masking, clipping masking, and alpha channel masking. In some embodiments, depth masking is performed on geometric structures that are already available in a database or data structure stored in the engine platform server system and / or from virtual copies of real-world elements stored in the persistent virtual world system. In other embodiments, depth masking is performed on geometric structures generated in real time by the engine platform server.

[0022] In some embodiments, requesting components of a virtual frame refers to retrieving actual content of the virtual frame corresponding to a digital reality application that will be part of a unified media stream.

[0023] In some embodiments, combining virtual frames into a media stream view is performed by one or more of warping, stitching, and interpolating multiple virtual frames.

[0024] According to one embodiment, the digital content sent to the virtual frame by one or more engines includes a 3D icon. In one embodiment, the 3D icon includes one or more voxels, static 3D representations, dynamic 3D representations, or a combination thereof, as determined during the development phase of the digital reality application, which is used to graphically represent the 3D application. The 3D icon can be different from a 3D object because the 3D icon is a basic graphical representation of the application, while the 3D object represents a more immersive experience when the user reaches a deeper level of engagement with the application. In one embodiment, the 3D object includes a mathematical model of the graphical representation.

[0025] In some embodiments, the virtual frame may include elements rendered by the engine platform server. In other embodiments, the virtual frame may include elements rendered by a third-party engine server. In other embodiments, the virtual frame may include elements rendered by a client device. In other embodiments, the virtual frame may include a mixture of elements rendered by an engine platform server, a third-party engine server, and a client device.

[0026] As an example, the virtual frame is initially rendered by the engine platform server or a third-party engine server, synthesized by the engine platform server, and output by the client device. In this example, the client device only needs to perform lightweight operations on the media stream. In a further example, the digital reality application is configured to be downloaded and executed locally by the client device. In an illustrative scenario, as the user's participation in the digital reality application increases, the client device can proceed to fully download and install the digital reality application and its 3D objects, as well as the runtime instance of the corresponding 3D engine, and perform local rendering of the received media stream, so that all or most of the virtual frames are rendered locally while executing the application. In other embodiments, after a predetermined amount of time has passed since the user viewed or participated in the digital reality application, the client device can continue to fully download the digital reality application and perform rendering locally while executing the application.

[0027] Digital reality applications are visual and can be interacted with via client devices, which may include one or more mobile devices, personal computers, game consoles, smart contact lenses, media centers, head-mounted displays, etc. The client device can be equipped with sensors to determine the relative position and orientation of the client device (three coordinates) and the relative direction of the headset relative to the observer (three angles). In some embodiments, the tracking information provides the client device with 6 degrees of freedom position and orientation, which can determine how to generate output streams from multiple virtual frames.

[0028] In an embodiment, in order to reduce hardware and network requirements, help reduce network latency, and improve the general digital reality experience, the system can be connected through a network including a combination of millimeter wave (mmW) or mmW and sub 6GHz communication systems, such as through a fifth generation wireless system communication (5G). In other embodiments, the system can be connected through a wireless local area network (Wi-Fi), which provides data at 60GHz. The communication system provided can allow low (e.g., about 1 to about 5 milliseconds) end-to-end (E2E) latency and high (e.g., 1-10Gbps) downlink speeds to field endpoints, in line with the parameters required to perform typical highly interactive digital reality applications. This results in high-quality, low-latency, real-time digital application content streaming. In other embodiments, the system can be connected through a fourth generation wireless system communication (4G) communication, can be supported by a 4G communication system, or can include other wired or wireless communication systems.

[0029] According to one embodiment, the sensing mechanism installed on the client device includes a combination of an inertial tracking sensing mechanism and a transceiver. The inertial tracking sensing mechanism may use devices such as accelerometers and gyroscopes, which may be integrated in an inertial measurement unit (IMU). The transceiver may be implemented to send wireless communication signals to and receive wireless communication signals from an antenna. In an embodiment, the transceiver is a mmW transceiver. In an embodiment using a mmW antenna, the mmW transceiver is configured to receive mmW signals from the antenna and send data back to the antenna. Inertial sensors, as well as position tracking provided by the mmW transceiver, and accurate tracking, low latency, and high QOS capabilities provided by mmW-based antennas, can achieve sub-centimeter or sub-millimeter position and orientation tracking, which can increase accuracy when tracking the real-time position and orientation of connected elements. In some embodiments, tracking can be implemented by adopting multiple technologies known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking technologies known in the art (e.g., visual imaging, radar technology, etc.). In an alternative embodiment, the sensing mechanism and the transceiver may be coupled together in a single tracking module device. The sensing mechanism of the client device may also include one or more cameras. For example, the camera may be a depth camera installed in the client device. The camera may be configured to capture and provide the user's viewing position and orientation, which determines the viewing position and orientation of the virtual frame sent via the engine platform server.

[0030] Providing accurate tracking of connected elements can help to display the reliable status of client devices within the persistent virtual world system, particularly their position and orientation, which can be relevant to various applications. In addition, achieving accurate, real-time tracking of client devices can reduce the need to physically sense other client devices.

[0031] According to an embodiment, a method for delivering location-based 3D content includes the following steps: receiving, by an engine platform server system, a client device location and orientation and a view request associated with a digital reality application; the engine platform server system requests virtual frames from one or more 3D engines; the engine platform server system receives virtual frames sent by one or more 3D engines; the engine platform server system combines the virtual frames into a media stream view of the digital reality application; and sending the synthesized view to the client device for viewing and interaction with the user.

[0032] According to an embodiment, synthesis of virtual frames includes classifying virtual frames in 3D space; culling virtual frames in 3D space; depth masking virtual frames to achieve partial blocking of virtual frames based on real-world elements; requesting components of virtual frames; and combining virtual frames into a view.

[0033] According to one embodiment, the method also includes switching from server rendering to local rendering (e.g., switching from server-based rendering of 3D icons to local rendering of 3D objects). The switching can be based on the engagement time or engagement level reached by the user with the digital reality application. The engine platform server receives the 3D icon from the 3D engine; renders the 3D icon and sends the 3D icon to the virtual frame; the engine platform server synthesizes the virtual frame to obtain a media stream with the 3D icon; sends the media stream with the 3D icon to the client device for local processing, such as performing lightweight tasks on the media stream and displaying it to the user; checks whether enough time has passed or whether a deeper engagement level has been reached, wherein, for a negative case where a threshold time has not passed or a threshold level of engagement has not been reached, the method can loop back to receiving the 3D icon from the 3D engine; and in the affirmative case, where a threshold time has passed or a threshold level of engagement has been reached, or a combination thereof, authorizes or instructs the client device to download and install a digital reality application including a 3D object and a corresponding runtime instance of one or more 3D engines.

[0034] From the perspective of a client device, the disclosed method includes downloading and installing, by the client device, a digital reality application including a 3D object, and corresponding runtime instances of one or more 3D engines; the client device locally renders the 3D object; and the client device executes the digital reality application including a media stream with the 3D object. According to one embodiment, depending on the duration of the interaction or the level of interaction between the user and the digital reality application, the rendering may be performed in part by an engine server (e.g., an engine platform server or a third-party engine server) or by the client device. When the rendering is performed by a mix of the engine server and the client device, the unified media stream view may include 3D icons and 3D objects. In other embodiments, the rendering may be switched back and forth between the server and the client depending on the quality of service (QoS) available in the network.

[0035] According to an optional embodiment of the conversion from server rendering of 3D icons to local rendering of 3D objects, the client device can also be configured to synthesize a virtual frame after receiving the rendered 3D icon from the engine platform server. Comparing the virtual frame synthesized by the server with the virtual frame synthesized by the client, the former case can achieve a standard but less flexible frame rate of the media stream displayed to the user and lower bandwidth and client device computing requirements, while the latter case can provide higher frame rate flexibility of the media stream displayed to the user and higher bandwidth and client device computing requirements.

[0036] Also disclosed is a computer-readable medium having instructions stored thereon, the instructions being configured to cause one or more computing devices (eg, a server system or a client device) to perform the methods disclosed herein.

[0037] The above summary of the invention does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects outlined above, as well as those disclosed in the following detailed description and those specifically pointed out in the claims submitted with the present application. Such combinations have specific advantages that are not specifically stated in the above summary of the invention. Other features and advantages will be apparent from the accompanying drawings and from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The specific features, aspects and advantages of the present disclosure will be better understood with respect to the following description and accompanying drawings, in which:

[0039] Figure 1 Depicted is a schematic representation of a system for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content in accordance with an embodiment.

[0040] Figure 2Depicted is an isometric schematic representation of a system for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content, showing a real-world view of the location-based platform with multiple applications, in accordance with an embodiment.

[0041] Figure 3A-3B Depicted is a schematic representation of a virtual frame according to an embodiment.

[0042] Figure 4 A schematic representation of a system for enabling a location-based platform for hosting multiple 3D engines for delivering location-based 3D content is shown, showing a transition from server-rendered 3D icons to client-rendered 3D objects, according to an embodiment.

[0043] Figure 5A-5B Depicted are schematic representations of server-rendered 3D icons and client-rendered 3D objects in accordance with an embodiment.

[0044] Figure 6 A schematic representation of a client device according to an embodiment is shown.

[0045] Figure 7 A method for enabling a location-based platform for hosting multiple 3D engines for delivering location-based 3D content according to an embodiment is shown.

[0046] Figure 8 A method for synthesizing a virtual frame according to an embodiment is depicted.

[0047] Fig. 9 A method for transitioning from a server-rendered 3D icon to a client-rendered 3D object according to an embodiment is described.

[0048] Fig.10 Another method for transitioning from a server-rendered 3D icon to a client-rendered 3D object according to an embodiment is shown. DETAILED DESCRIPTION

[0049] In the following description, reference is made to the accompanying drawings which show various embodiments by way of illustration. In addition, various embodiments will be described below by reference to a number of examples. It should be understood that the embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.

[0050] Figure 1 Depicted is a schematic representation of a system 100 for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content, according to an embodiment.

[0051] The system 100 includes an engine platform server 102 configured to host one or more third-party engine servers 104 and provide an engine platform 106. The engine platform 106 includes one or more digital reality applications 108 developed via one or more 3D engines, which may be third-party 3D engines 110 or native 3D engines 112.

[0052] As an example, in Figure 1 In the illustration of , digital reality application AD is developed via a third-party 3D engine 110, and digital reality application E is developed via a local 3D engine 112. More specifically, digital reality application AB is developed via a third-party 3D engine hosted by engine server A, digital reality application C is developed via a 3D engine B hosted by engine server B, digital reality application D is developed via a 3D engine C hosted by engine server C, and digital reality application E is developed by a local 3D engine 112 hosted by engine platform server 102.

[0053] The digital reality applications 108 are anchored at predetermined locations within the persistent virtual world system, or are anchored to mobile virtual copies, the locations of which may change in accordance with the movement of the corresponding reality elements. Thus, a user 114 proximate to one of the predetermined locations may use a client device 116 to view one or more digital reality applications 108 independent of the 3D engine used during development. The client device 116 is connected to the engine platform server 102 via a network 118 and is configured to provide the digital reality applications 108 to the user 114, thereby enabling user interaction. In a further embodiment, when accessing the engine platform 106, the application developer 120 is able to view and select from a variety of third-party 3D engines 110 or native 3D engines 112 to develop a digital reality application 108 using a client device 116.

[0054] In the present disclosure, the term "persistence" is used to characterize the state of a system that can continue to exist without the continued execution of a process or a network connection. For example, the term "persistence" can be used to characterize a virtual world system, wherein the virtual world system and all virtual copies, purely virtual objects, and digital reality applications included therein continue to exist after the process used to create the virtual copies, purely virtual objects, and digital reality applications stops, and does not rely on the user being connected to the virtual world system. Therefore, the virtual world system is saved in a non-volatile storage location, for example, in the server 102. In this way, even if the user is not connected to the server, the virtual copies, purely virtual objects, and digital reality applications can interact and collaborate with each other when configured to achieve a specific goal.

[0055] The currently disclosed system 100 can be implemented in a cloud-to-edge infrastructure that can employ public or private clouds, fog servers, distributed ledger infrastructure, and edge devices and systems such as enterprise systems, mobile platforms, and user devices to display distributed computing capabilities, all of which can be connected via a network. Using a cloud-to-edge computing network, access to computing power, computer infrastructure (e.g., through so-called infrastructure as a service or IaaS), applications, and business processes can be delivered to users as a service via client devices on demand. In this way, resources including physical servers and network devices enable shared storage and computing that can be dynamically allocated based on factors such as the distance of the user to the resources and the network and the computing needs from the user.

[0056] A third-party 3D engine 110 available at the engine platform 106 can provide a specially created environment to implement functions dedicated to the digital reality application 108. Therefore, the 3D engine is configured to perform tasks that enable aspects such as the management of animated models, conflicts between objects, the behavior and interaction of light with individual objects and the interaction between objects of the scene and the interaction between users and applications. The 3D engine can include, for example, a physics engine that simulates the laws of physics within a virtual environment, an audio engine that adds music and complex acoustic effects, and an artificial intelligence (AI) engine that programs the intelligent behavior of a computer. The 3D engine can be used in any type of application that requires rendering 3D graphics with real-time performance, including applications in virtual reality (VR), augmented reality (AR), mixed reality (MR), or a combination thereof.

[0057] In some embodiments, one or more digital reality applications 108 are developed via one or more 3D engines via the systems and methods described in U.S. Patent Application No. 16 / 421,155, filed by Jawat Yearley on May 23, 2019, entitled "Methods and Systems for Developing, Testing Digital Reality Applications via Virtual Worlds and Deploying Digital Reality Applications into the Real World," which is incorporated herein by reference in its entirety. In such embodiments, a system and method for developing, testing, and deploying digital reality applications into the real or virtual world by integrating a digital reality application development system and an application management system in a digital reality ecosystem is provided. In the digital reality application development system, an application developer creates a digital reality application; configures the application, including location and space (i.e., real or virtual location, 3D positioning and scaling), time, and target user parameter settings; and tests the application in the developer space, the final real location of augmented reality, and the virtual / hybrid space of virtual / hybrid reality. In the application management system, the application administrator tests the application; approves / rejects the application; sends an adjustment request to the application developer when necessary; and after the adjustment, if compliant, deploys the application in the online application store for user access, whereby the application can function based on the previously defined and approved content and settings.

[0058] In some embodiments, a persistent virtual world system stored in the engine platform server 102 and capable of selecting real-world locations of digital reality applications is created via a system and method for developing and deploying virtual copies of real-world elements into a persistent virtual world system. The creation of the virtual copy is performed in a virtual environment through a copy editor that enables the development and configuration of the virtual copy to reflect the behavior and appearance of the corresponding real-world elements. The virtual copy is enriched with data captured by a sensing mechanism that synchronizes the virtual copy with the real-world elements. In one embodiment, the virtual copies are shared in an ongoing virtual world system quality assessment, in which they can be approved or rejected for subsequent adjustments if necessary. After approval and deployment, the copy is shared in a deployed persistent virtual world system that is visible to end users for managing and interacting with the virtual copy.

[0059] In an embodiment, in order to reduce hardware and network requirements, help reduce network latency, and improve the general digital reality experience, the system 100 can be connected via a network 118 including a combination of millimeter wave (mmW) or mmW and sub 6GHz communication systems, such as via a fifth generation wireless system communication (5G). In other embodiments, the system can be connected via a wireless local area network (Wi-Fi) that provides data at 60GHz. The communication system provided can allow low (e.g., about 1 to about 5 milliseconds) end-to-end (E2E) latency and high (e.g., 1-10Gbps) downlink speeds to field endpoints, in line with the parameters required to execute typical highly interactive digital reality applications 108. This results in high-quality, low-latency, real-time digital application content streaming. In other embodiments, the system can be connected via a fourth generation wireless system communication (4G) communication, can be supported by a 4G communication system, or can include other wired or wireless communication systems.

[0060] The digital reality application 108 may be viewed by and interacted with via the client device 116, which may include one or more mobile devices, personal computers, game consoles, media centers, smart contact lenses, head-mounted displays, etc. The client device 116 may be equipped with sensors to determine the relative position and orientation (three coordinates) of the client device 116 and the relative orientation (three angles) of the headset relative to the viewer. This tracking information is equivalent to 6 degrees of freedom of the client device 116, which may determine how to generate an output stream from multiple virtual frames.

[0061] In some embodiments, system 100 may use cloud and edge infrastructure that can implement distributed computing capabilities, including employing public or private clouds, micro clouds, and edge systems such as enterprise systems, mobile platforms, and user devices (e.g., client device 116).

[0062] Figure 2 Depicted is an isometric schematic representation of a system 200 for enabling a location-based platform for hosting multiple 3D engines that deliver location-based 3D content, showing a real-world view of the location-based platform with multiple digital reality applications, according to an embodiment. Figure 2 Some components can be used with Figure 1 Elements of the invention are similar, and thus similar or identical reference numerals may be used to describe those elements.

[0063] Reference Figure 2, a digital reality application 108 (e.g., a digital reality application AE developed by a third-party 3D engine 110A-C and a local 3D engine 112, respectively) is associated with (e.g., virtually attached to) one or more predetermined 3D locations 202 in a real-world location area 204 stored within a persistent virtual world system at an engine platform server. The predetermined 3D locations 202 are selected during development of the digital reality application.

[0064] When the digital reality application 208 is virtually attached to an element of the real world, such as a specific 3D location, the digital reality application 208 can be located in space, for example, in tandem with the element to which it is virtually attached. If the digital reality application 208 is not attached, the application can be located in space independent of the location of any object. If the digital reality application 208 is detached from the element to which it is virtually attached, the application can be removed from space where the element still exists, or remain in a fixed position independent of the location of the object, or move independently of the object, for example.

[0065] In one embodiment, when a user 114 employing a client device 116 approaches one or more digital reality applications 108 in one of the predetermined 3D locations 202, the digital reality applications 108 broadcast a proximity-based signal to the client device 116 so that the client device 116 is aware of the presence of the nearby digital reality applications 108. The client device 116, upon receiving the signal, proceeds to send the viewing position and orientation and a corresponding view request to the engine platform server. Upon receiving the view request, the engine platform server requests a virtual frame 206 from one or more third-party 3D engines 110 or a local 3D engine 112 used in the development of the one or more digital reality applications 108, and then proceeds to synthesize each virtual frame 206 into a media stream view that is sent to the client device 116. Thus, the virtual frame 206 is composed of bits of data collected from one or more 3D engines, as the user 114 may be viewing an area that includes multiple applications developed by several different 3D engines. The engine platform server then sends the synthesized view to the client device 116, which enables smooth, natural, low latency, and real-time views and interactions with the one or more digital reality applications.

[0066] like Figure 2As shown, each of the digital reality applications 108 can be visually represented by a 3D icon, such as a rectangular prism representing digital reality applications 108A and E, a sphere representing digital reality applications B and D, and a pyramid representing digital reality application C. Because all digital reality applications 108 are available at the 3D engine platform, and because all virtual frames 206 corresponding to the visual representations and media content of the 3D engine are synthesized by the engine platform server, the user 114 can seamlessly and continuously view these visual representations of the digital reality applications through the client device 116, regardless of the 3D engine used during their development. Figure 2 , application AC may be included within the viewing cone 208 of user 114 and its virtual frame 206 may be composited to user 114 and output via client device 116 , even though application AB was developed via 3D engine A and application C was developed via 3D engine B.

[0067] According to one embodiment, the digital reality application 108 may utilize one or more of a variety of techniques to broadcast a signal to the client device 116, thereby alerting the client device 116 that the digital reality application 108 is available nearby. In one embodiment, if the user 114 has previously subscribed to the engine platform service, the location of the client device 116 is always available at the persistent virtual world system available at the engine platform server. Therefore, when the user 114 approaches the digital reality application 108, the digital reality application 108 may be ready to broadcast a signal to the client device 116. In another embodiment, if the user 114 has previously subscribed to the engine platform service or has subscribed to one or more specific digital reality applications 108 from the engine platform service, the digital reality application 108 may continuously search for registered devices before broadcasting a signal. In another embodiment, when the digital reality application 108 detects the client device 116 within a predetermined proximity threshold in the predetermined 3D location 202, the digital reality application 108 may broadcast a signal. For example, the predetermined 3D location 202 may be a building or a room within a building. In another example, the predetermined 3D location 202 may be within a city block. In another example, predetermined 3D location 202 may be within a city. In one embodiment, when user 114 with client device 116 enters a location where a digital reality application is activated, the digital reality application may detect a signal from the device indicating that client device 116 may receive the digital reality application media stream before broadcasting the signal.

[0068] In the present disclosure, the term "virtual frame" refers to one of many image elements (e.g., still images) that make up a virtual animation, which can be included in a media stream that transmits the digital content of the digital reality application 108. For example, the virtual frame 206 can be a virtual sprite that is integrated into a larger scene, or a two-dimensional bitmap. When the virtual animation is displayed, each virtual frame 206 is flashed to the user for a short time at a predetermined 3D position 202. The position and orientation of the virtual frame 206 displayed to the user depends on the user's viewing position and orientation, as sensed by the client device 116.

[0069] Multiple virtual frames 206 may be synthesized to generate a single media stream to client device 116 that can be viewed and interacted with by user 114. The media stream may include digital reality content, including 3D image data, 3D geometry, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, text data, time data, position data, orientation data, tactile data, and lighting data. The calculations used to synthesize virtual frames 206 may be optimized by considering only applications within the viewing cone 208 of user 114. For example, in Figure 2 , only applications AC may be included within viewing frustum 208 of user 114, and applications DE are eliminated because they are located outside viewing frustum 208. Figure 3A -B depicts a schematic representation of a virtual frame 206 according to an embodiment. Figure 3A -Some components of B can be Figure 1-2 Elements of the invention are similar, and thus similar or identical reference numerals may be used to describe those elements.

[0070] like Figure 3A As shown, virtual frame 206 is located at real world location area 204 and is generated and populated with content from each of third party 3D engine 110 and native 3D engine 112, respectively. Depending on the user's viewing frustum, virtual frame 206 may be independently generated by a separate 3D engine, which may be the case if the user directs his or her view toward a separate digital reality application. However, as Figure 3B As shown, virtual frame 206 may also be generated by more than one 3D engine, which may be the case if a user directs his or her view toward more than one digital reality application.

[0071] Figure 4 Depicted is a schematic representation of a system 400 for enabling a location-based platform for hosting multiple 3D engines for delivering location-based 3D content, showing a transition from a server-rendered 3D icon 402 to a client-rendered 3D object 404, according to an embodiment. Figure 4 Some components can be used with Figure 1-3BElements of the invention are similar, and thus similar or identical reference numerals may be used to describe those elements.

[0072] According to one embodiment, the digital content sent by the engine server 406 to the virtual frame 206 includes a 3D icon 402, which may be one or more voxels, a static 3D representation, a dynamic 3D representation, or a combination thereof determined via the 3D engine 408 during the development phase of the digital reality application.

[0073] In some embodiments, the virtual frame 206 of the media stream may include elements rendered by the engine platform server. In other embodiments, the virtual frame 206 of the media stream may include elements rendered by a third-party engine server. In other embodiments, the virtual frame 206 of the media stream may include elements rendered by the client device 116. In other embodiments, the virtual frame 206 of the media stream may include a mix of elements rendered by the engine platform server, the third-party engine server, and the client device 116.

[0074] In one embodiment, reference Figure 4 In the first column from left to right in FIG. 1 , the virtual frame 206 of the media stream 410 with the 3D icon is initially rendered by the engine platform server or by a third-party engine server. Figure 4 The engine platform server 406 is grouped into the media stream 410 with the 3D icon; the engine platform server synthesizes; and streams to the client device 116, which outputs the media stream 410 with the 3D icon. In this embodiment, the client device 116 only needs to perform lightweight operations on the media stream 410 with the 3D icon.

[0075] In a further embodiment, and with reference to Figure 4 In the second column, as time and / or user engagement 412 with the digital reality application increases, the client device 116 can proceed to fully download and install the digital reality application, after which the client device 116 can proceed to execute the digital reality application.

[0076] Furthermore, in these embodiments, and with reference to Figure 4 In the third column of FIG. 1 , after the digital reality application has been completely downloaded and installed in the client device 116, the 3D icon 402 can be converted into a 3D object 404, which is rendered locally and sent to the client device as a media stream with a 3D object 414. The client device can then finally execute the digital reality application locally.

[0077] Figure 5A -B depicts a schematic representation of a server-rendered 3D icon and a client-rendered 3D object according to an embodiment. Figure 5A -Some components of B can be Figure 1-4Elements of the invention are similar, and thus similar or identical reference numerals may be used to describe those elements.

[0078] Reference Figure 5A , a user 114 using a client device 116 located at a real location area 204 can view one or more visual representations of a digital reality application at a predetermined 3D location 202, which digital reality application can include one or more server-rendered 3D icons 502 contained within a viewing frustum 208 of the user 114, client-rendered 3D objects 504, or a combination thereof.

[0079] Reference Figure 5B , the virtual frames synthesized into the media stream view and output via the client device may include virtual frames 502a-c. Virtual frame 502a may include a server-rendered 3D icon 502; virtual frame 502b may include a mixture of server-rendered 3D icon 502 and client-rendered 3D object 504; and virtual frame 502c may include a purely client-rendered 3D object 504.

[0080] Figure 6 A schematic representation of a client device 116 is shown in accordance with an embodiment.

[0081] The client device 116 may include operational components such as an input / output (I / O) module 602 ; a power supply 604 ; a memory 606 ; a sensor 608 and a transceiver 610 forming a tracking module 612 ; and a network interface 614 , all operably connected to a processor 616 .

[0082] The I / O module 602 is implemented as computing hardware and software that is configured to interact with a user and provide user input data to one or more other system components. For example, the I / O module 602 can be configured to interact with a user, generate user input data based on the interaction, and provide user input data to the processor 616 before the user input data is transmitted to other processing systems (such as to a server) via a network. In another example, the I / O module 602 is implemented as an external computing indicating device (e.g., a touch screen, a mouse, a 3D control, a joystick, a game controller, etc.) and / or a text input device (e.g., a keyboard, a dictation tool, etc.) configured to interact with the client device 116. In other embodiments, the I / O module 602 can provide additional, less or different functions than those described above.

[0083] The power supply 604 is implemented as computing hardware and software configured to provide power to the client device 116. In one embodiment, the power supply 604 can be a battery. The power supply 604 can be built into the device or can be removed from the device, and can be rechargeable or non-rechargeable. In one embodiment, the device can be repowered by replacing one power supply 604 with another power supply 604. In another embodiment, the power supply 604 can be recharged by a cable attached to a charging source, such as a universal serial bus ("USB") FireWire, Ethernet, Thunderbolt interface, or headphone cable attached to a personal computer. In yet another embodiment, the power supply 604 can be recharged by inductive charging, wherein an electromagnetic field is used to transfer energy from the inductive charger to the power supply 606 when the inductive charger and the power supply 604 are very close but do not need to be inserted into each other via a cable. In another embodiment, a docking station can be used to facilitate charging.

[0084] The memory 606 may be implemented as computing hardware and software suitable for storing application instructions. The memory 606 may be any suitable type capable of storing information accessible by the processor 616, including computer-readable media, or other media storing data that can be read by an electronic device, such as a hard drive, a memory card, a flash drive, a ROM, a RAM, a DVD or other optical disk, and other writable and read-only memories. In addition to persistent storage, the memory 606 may also include temporary storage.

[0085] The sensing mechanism can be implemented as computing hardware and software that is suitable for obtaining data from the real world and determining / tracking the position and orientation of the client device 116, and sending this information to the engine platform server to determine the position and orientation of the virtual frame and the synthetic media stream view sent to the client device. For example, the sensor 608 may include one or more cameras, such as one or more depth cameras. The sensor may also include one or more inertial measurement units (IMUs), accelerometers, and gyroscopes. The IMU is configured to measure and report the speed, acceleration, angular momentum, translation speed, rotation speed, and other telemetry metadata of the client device 116 by using a combination of accelerometers and gyroscopes. The accelerometer in the IMU and / or configured separately from the IMU can be configured to measure the acceleration of the interactive device, including the acceleration caused by the earth's gravitational field. In one embodiment, the accelerometer includes a three-axis accelerometer capable of measuring accelerations in three orthogonal directions.

[0086] The transceiver 610 may be implemented as computing hardware and software configured to enable the device to receive radio waves from an antenna and send data back to the antenna. In some embodiments, a mmW transceiver may be employed that may be configured to receive mmW wave signals from an antenna and send data back to the antenna when interacting with immersive content. The transceiver 610 may be a two-way communication transceiver 610.

[0087] In an embodiment, the tracking module 612 may be implemented by combining the capabilities of an IMU, accelerometer, and gyroscope with the position tracking provided by the transceiver 610, and the precise tracking, low latency, and high QOS capabilities provided by the mmW-based antenna may enable sub-centimeter or sub-millimeter position and orientation tracking, which may increase accuracy when tracking the real-time position and orientation of the client device 116. In an alternative embodiment, the sensing mechanism and transceiver 610 may be coupled together in a single tracking module device.

[0088] The network interface 614 may be implemented as computing software and hardware that communicatively connects to a network, receives computer readable program instructions from the network sent by a server or by the client device 116 , and forwards the computer readable program instructions for storage in the memory 606 for execution by the processor 616 .

[0089] The processor 616 can be implemented as computing hardware and software configured to receive and process sensor and digital reality application data and instructions. For example, the processor 616 can be configured to provide imaging requests, receive imaging data, process imaging data into environment or other data, process user input data and / or imaging data to generate user interaction data, perform machine learning training and reasoning based on edge (on the device), provide server requests, receive server responses, and / or provide user interaction data, environment data and content object data to one or more other system components. For example, the processor 616 can receive user input data from the I / O module 602, and can implement the application stored in the memory 606 respectively. In other examples, the processor 616 can receive data captured from the real world from the sensing mechanism, or can receive the precise position and orientation of the client device 116 through the tracking module 612, and can prepare some data before sending data to the server for further processing. In other examples, the processor 616 can perform edge-based rendering of the media stream received from the engine platform server while executing the digital reality application. In other examples, processor 616 may receive a media stream rendered by an engine platform server and may perform lightweight operations on the media stream in order to output the media stream.

[0090] Figure 7A block diagram of a method 700 for enabling a location-based platform for hosting multiple 3D engines for delivering location-based 3D content according to an embodiment is shown. The method 700 may be implemented by a system such as a system described in reference Figure 1-6 Depicted system.

[0091] Method 700 may begin in blocks 702 and 704 by providing an engine platform server configured to host one or more third-party engine servers and provide an engine platform, the engine platform also being configured to provide a digital reality application developed via one or more 3D engines residing in the third-party engine server or the engine platform server. Method 700 continues in block 706 by providing one or more client devices connected to the engine platform server via a network and configured to output the digital reality application and provide a user interface capable of interacting with a user. Subsequently, method 700 proceeds in block 708 by receiving, by the engine platform server, a client device position and orientation and a view request associated with the digital reality application sent by the client device, which may occur when a user employing the client device approaches a predetermined location where the digital reality application has been located during development.

[0092] The method 700 continues in step 710 by requesting a virtual frame from one or more 3D engines by the engine platform server, and then by receiving the virtual frame sent by the one or more 3D engines in step 712. Subsequently, the method 700 continues in step 714 by compositing the virtual frames into a media stream view, and finally by sending the composited media stream view to a client device for viewing and user interaction, as shown in block 716, and ends the process in terminator 718.

[0093] Figure 8 Describes a method for synthesizing a reference according to an embodiment Figure 7 A block diagram of a method 800 for implementing a virtual frame. The method 800 may be implemented by a system such as a system Figure 1-6 Depicted system.

[0094] In some embodiments, the synthesis of virtual frames is performed independently of the frame rate of the received frames, and sent to the user via the client device at a natural frame rate of 60 FPS, 90 FPS, 120 FPS, another frame rate between 60 and 120 FPS, or some other frame rate. Within this frame rate range, the user can feel that the media stream is received, and therefore receives the experience from the digital reality application, and can participate in real time. In some embodiments, when the third-party engine server sends the corresponding parts of the virtual frame, the third-party engine server can render these virtual frame parts so that the virtual frame received by the engine platform server has been rendered.

[0095] Method 800 begins in blocks 802 and 804 by classifying a virtual frame in 3D space. In some embodiments, the classification of the virtual frame in 3D space involves arranging different elements of the virtual frame to specific positions and orientations in the 3D space so that the media stream to be generated by the animated virtual frame can be adjusted to the viewing position and orientation of the user.

[0096] The method 800 continues in block 806 by performing viewport culling of the virtual frame in 3D space to remove unnecessary elements from the viewing frustum of the virtual frame. These elements may be considered unnecessary because, for example, they may be completely outside the viewing frustum of the virtual frame or they may be blocked by real-world elements.

[0097] Subsequently, method 800 proceeds in box 808 by performing depth masking of the virtual frame to achieve partial blocking of the virtual frame based on the real-world elements. In some embodiments, depth masking refers to the process of performing partial blocking of some parts of the virtual frame, especially parts located in the background that may affect the depth of the image displayed on the virtual frame. Depth masking can be performed using several techniques known in the art, such as layer masking, clipping masking, and alpha channel masking. In some embodiments, depth masking is performed on geometry that is already available from virtual copies of real-world elements stored in a persistent virtual world system. In other embodiments, depth masking is performed on geometry generated in real time by an engine platform server.

[0098] The method 800 proceeds in block 810 by requesting components of a virtual frame, which may involve retrieving actual content of the virtual frame corresponding to a digital reality application that will be part of a unified media stream.

[0099] Subsequently, the method 800 continues in block 812 by combining the virtual frames into one view. In some embodiments, combining the virtual frames into one view is performed by warping, stitching, and interpolating one or more of the multiple virtual frames, etc. Some of these techniques are described in detail in U.S. Patent Application No. 15 / 764,696 filed by Jewat Yearley on March 29, 2018, which is incorporated herein by reference in its entirety. Combining virtual frames can be a more complex reconstruction process based on the input virtual frames. For example, the process can rely on a combination of standard image reconstruction techniques, such as stitching, warping, interpolation, and extrapolation. For example, extrapolation may be required in areas where no or limited (visual) information is available based on the available virtual frames in order to fill in gaps or holes in the media data. However, it should be understood that the reconstruction process is not limited to computer vision techniques and may further consider spatial data about the scene, which may include any combination of reconstructed 3D geometric information, parameters about materials, and light fields, which may correspond to optical flows in the captured scene, etc. In some embodiments, the spatial data can be used to re-render the combined virtual frame using 3D rendering techniques. In one or more embodiments, the generation of the output stream may include the use of deep learning techniques and / or neural networks, which may be used to recreate virtual frames of the output stream from a sequence of virtual frames of a media stream of the same scene taken from different viewpoints. This may enable complex reconstruction and generation of the output stream even if at least a portion of the scene is not captured completely or in full detail.

[0100] Method 800 may end in terminator 814 .

[0101] Fig. 9 A block diagram of a method 900 according to an embodiment is shown, which describes in detail the conversion between a server-rendered 3D icon and a client-rendered 3D object. The method 900 may be implemented by a system, such as a reference Figure 1-6 Depicted system.

[0102] Method 900 begins at blocks 902 and 904, where a 3D icon is received from a 3D engine by an engine platform server. The 3D engine may be a local 3D engine hosted on the engine platform server, or a third-party 3D engine hosted on one or more third-party engine servers. Method 900 continues in block 906 by rendering the 3D icon by the engine platform server or by the third-party engine server and sending the 3D icon to a virtual frame.

[0103] Subsequently, the method 900 continues in box 908 by synthesizing a virtual frame by the engine platform server and obtaining a media stream with a 3D icon, which is performed by the engine platform server. In box 910, the method 900 sends the media stream with the 3D icon to the client device, and the client device performs lightweight tasks on the media stream to display the media stream to the user. Then, the method 900 checks in decision box 912 whether enough time has passed, whether a deeper level of engagement has been reached, or a combination thereof. In the negative case, the method 900 can loop back to box 904 by receiving the 3D icon from the 3D engine. In the affirmative case, the method 900 continues in box 914 by downloading and installing a digital reality application including a 3D object and a corresponding runtime instance of one or more engines hosted on the engine platform server by the client device, and then continues in box 916 by locally rendering the 3D object. In box 918, the method 900 continues by executing the digital reality application including the media stream with the 3D object by the client device, and finally ends the process in terminator 920.

[0104] According to one embodiment, depending on the duration of the interaction between the user and the digital reality application or the level of interaction or a combination thereof, the rendering may be performed in part by an engine server (e.g., an engine platform server or a third-party engine server) or by a client device. When the rendering is performed by a mix of engine servers and client devices, the unified media stream view may include 3D icons and 3D objects. In other embodiments, the conversion may switch back and forth between the server and the client based on the quality of service (QoS) available in the network.

[0105] Fig.10 A block diagram of method 1000 is described, which details an alternative embodiment of a conversion between a server-rendered 3D icon and a client-rendered 3D object. The conversion can be based on the engagement time or engagement level achieved by the user with the digital reality application. In other embodiments, the conversion can switch back and forth between the server and the client based on the quality of service (QoS) available in the network. Method 1000 can be implemented by a system, such as reference Figure 1-6 Depicted system.

[0106] Method 1000 begins by receiving a 3D icon from a 3D engine by an engine platform server in blocks 1002 and 1004. The 3D engine may be a local 3D engine hosted on the engine platform server or a third-party 3D engine hosted on one or more third-party engine servers. Method 1000 continues in block 1006 by rendering the 3D icon by the engine platform server or by a third-party engine server and sending the 3D icon to a virtual frame.

[0107] Subsequently, the method 1000 continues in block 1008 by sending the rendered virtual frame to the client device. Thereafter, in block 1010, the method 1000 continues by the client device synthesizing the virtual frame and obtaining a unified media stream with a 3D icon. The method 1000 then continues by the client device performing a lightweight task on the unified media stream to display the media stream to the user, as shown in block 1012. In decision block 1014, the method 1000 checks whether enough time has passed, whether a deeper level of engagement has been reached, or a combination thereof. In the negative case, the method 1000 can loop back to block 1004 by receiving the 3D icon from the 3D engine. In the affirmative case, the method 1000 continues in block 1016 by downloading and installing, by the client device, a digital reality application including a 3D object and a corresponding runtime instance of one or more engines hosted on an engine platform server, and then in block 1018 by locally rendering the 3D object. In block 1020 , method 900 continues by executing, by the client device, a digital reality application including a media stream having a 3D object, and finally ends the process in terminator 1022 .

[0108] Will refer to Fig. 9 The server of method 900 synthesizes the virtual frame and the reference Fig.10 Compared with the client-synthesized virtual frames of method 1000, the former case can achieve a standard but less flexible frame rate for the media stream displayed to the user as well as lower bandwidth and client device computing requirements, while the latter case can provide higher frame rate flexibility for the media stream displayed to the user as well as higher bandwidth and client device computing requirements.

[0109] Although certain embodiments have been described and shown in the accompanying drawings, it should be understood that such embodiments are merely illustrative and not restrictive of the broad invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to one of ordinary skill in the art. Therefore, this description is to be regarded as illustrative and not restrictive.

Claims

1. A system for delivering location-based 3D content, characterized in that: The system comprises: An engine platform server system, the engine platform server system comprising one or more server computers, the engine platform server system being configured to host one or more engine servers and provide an engine platform, the engine platform comprising one or more digital reality applications developed via one or more 3D engines, respectively, one or more of the 3D engines being hosted in the engine server or the engine platform server system; wherein the digital reality application is virtually attached to one or more predetermined 3D locations in a real-world area, or is virtually attached to a moving virtual copy of a real-world object, wherein the engine platform server system is configured to receive a viewing position and orientation and a corresponding view request from a client device as the client device approaches and renders an initial view of at least one of the digital reality applications virtually attached to one of the predetermined 3D locations or one of the real-world objects, wherein the engine platform server system is configured to request virtual frames from different 3D engines after receiving the view request, and synthesize the virtual frames into a media stream view, and The engine platform server system is configured to send the media stream view to the client device.

2. The system according to claim 1, characterized in that The synthesizing the virtual frames into a media stream view comprises: Classifying virtual frames in 3D space; Perform viewport culling of virtual frames in 3D space; performing depth masking of a virtual frame so as to achieve partial occlusion of the virtual frame based on the real world element; requesting the contents of the virtual frame; and The virtual frames are combined into one view.

3. The system according to claim 2, characterized in that The depth masking is performed on geometry that is already available in a database or data structure stored in the engine platform server system, or is generated in real time by the engine platform server system.

4. The system according to claim 1, characterized in that The one or more 3D engines include a local 3D engine hosted in the engine platform server system, or a third-party 3D engine hosted in one or more third-party engine servers.

5. The system according to claim 1, characterized in that The virtual frames are synthesized by the engine platform server system regardless of the frame rate at which the frames are received.

6. The system according to claim 1, characterized in that The virtual frame is represented as a 3D icon, a 3D object, or a combination thereof.

7. The system according to claim 1, characterized in that At least one of the digital reality applications is configured to be downloaded and executed locally by the client device after one or more of a predetermined engagement time or level is reached.

8. A method for delivering location-based 3D content, characterized in that: The method comprises: Receiving, by an engine platform server system, a position and orientation of a client device and a view request, wherein the engine platform server system is configured to host one or more engine servers and provide an engine platform, the engine platform including one or more digital reality applications developed via one or more 3D engines, respectively, one or more of the 3D engines being hosted in one or more of the engine servers or the engine platform server system, and wherein the view request is associated with at least one of the one or more digital reality applications; requesting, by the engine platform server system, a virtual frame from at least one of the one or more 3D engines corresponding to at least one of the digital reality applications; The engine platform server system receives a virtual frame sent by one or more of the 3D engines; and The engine platform server system synthesizes the virtual frames into a unified media stream view of at least one of the digital reality applications.

9. The method according to claim 8, characterized in that The synthetic virtual frame includes: Classifying virtual frames in 3D space; Perform viewport culling of virtual frames in 3D space; performing depth masking of a virtual frame to achieve partial occlusion of the virtual frame based on real world elements; requesting the contents of the virtual frame; and The virtual frames are combined into one view.

10. The method according to claim 9, characterized in that The depth masking is performed on geometry that is already available in a database or data structure stored in the engine platform server, or is generated in real time by the engine platform server system.

11. The method according to claim 8, characterized in that The one or more 3D engines include a local 3D engine hosted in the engine platform server, or a third-party 3D engine hosted in one or more of the engine servers.

12. The method according to claim 8, characterized in that The virtual frames are synthesized by the engine platform server system regardless of the frame rate at which the frames are received.

13. The method according to claim 8, characterized in that The synthesizing virtual frame further includes transitioning from server rendering of the 3D icon to local rendering of the 3D object by: Receiving, by the engine platform server system, the 3D icon from one or more of the 3D engines; The engine platform server system or the engine server renders the 3D icon, and sends the 3D icon to the virtual frame; The engine platform server system synthesizes the virtual frame and obtains a media stream having the 3D icon; as well as The media stream with the 3D icon is sent to the client device for local processing.

14. The method according to claim 13, characterized in that Further including: authorizing the client device to download and install at least one of the digital reality applications including the 3D object and corresponding runtime instances of one or more of the 3D engines if the client device has engaged in at least one of the one or more digital reality applications for at least a threshold amount of time or a threshold level of engagement has been reached by the client device with at least one of the digital reality applications, or a combination thereof; locally rendering the 3D object by the client device; The client device synthesizes the virtual frame and obtains a media stream having a 3D object corresponding to at least one of the digital reality applications; as well as At least one of the digital reality applications including a media stream having a 3D object is executed by the client device.

15. The method according to claim 14, characterized in that The transitioning from server rendering of the 3D icon to local rendering of the 3D object is based on an engagement time, engagement level, or a combination thereof achieved by a user of the client device with at least one of the digital reality applications.

16. The method according to claim 14, characterized in that The rendering is performed in part by the engine server or by the client device depending on the duration of interaction or the level of interaction between the user of the client device and at least one of the digital reality applications, wherein the synthesized virtual frame and unified media stream view include 3D icons and 3D objects.

17. A computer readable medium having instructions stored thereon, characterized in that: The instructions are configured to cause the server system to perform the following steps: Receiving, by an engine platform server system, a position and orientation of a client device and a view request, wherein the engine platform server system is configured to host one or more engine servers and provide an engine platform, the engine platform including one or more digital reality applications developed via one or more 3D engines, respectively, one or more of the 3D engines being hosted in one or more of the engine servers or the engine platform server system, and wherein the view request is associated with at least one of the one or more digital reality applications; Requesting, by the engine platform server system, a virtual frame from one or more of the 3D engines; The engine platform server system receives a virtual frame sent by one or more of the 3D engines; and The engine platform server system synthesizes the virtual frames into a unified media stream view of at least one of the digital reality applications.

18. The computer-readable medium of claim 17, wherein: The one or more 3D engines include a local 3D engine hosted in the engine platform server, or a third-party 3D engine hosted in one or more of the engine servers.

19. The computer-readable medium of claim 17, wherein: The virtual frames are synthesized by the engine platform server system regardless of the frame rate at which the frames are received.

20. The computer-readable medium of claim 17, wherein: Compositing virtual frames includes switching from server rendering to local rendering.

Images