Cloud Native-Based Real-Time XR Cloud Computing System in a Multi-Cloud Environment
A microservice-based cloud-native system in a multi-cloud environment addresses XR content processing limitations by offloading tasks to the cloud, ensuring efficient resource utilization and ultra-low latency interactions for enhanced XR services.
Patent Information
- Application Number
- KR1020240187440
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2044-12-16
AI Technical Summary
Existing XR devices face challenges in processing large-capacity XR content due to limited computing resources and high network latency, leading to increased Motion-to-Photon latency and VR Sickness, which degrades the Quality of Experience (QoE).
A microservice-based cloud-native computing system in a multi-cloud environment is utilized to offload XR content processing to the cloud, incorporating a computing virtualization unit, content loading, rendering, encoding, interaction, and caching units to provide ultra-low latency XR content services.
The system enables efficient utilization of computing resources through Auto Scaling, allows high-quality XR content rendering in the cloud, optimizes content for transmission, and provides seamless, ultra-low latency interactions, thereby enhancing the XR experience.
Smart Images

Figure 112024139440090-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to an XR cloud computing system for providing real-time XR content services by utilizing cloud-native based on a microservices architecture in a multi-cloud environment. Specifically, it relates to a computing system for providing real-time XR content services by offloading the processing of large-capacity XR content, which is difficult to process on XR devices, to a cloud-native environment and transmitting the processing results with ultra-low latency. Background Technology
[0003] XR Cloud is a cloud computing-based service and infrastructure technology that supports and provides extended reality technologies such as Augmented Reality (AR) and Virtual Reality (VR). Through this, it enables users to have rich interactions between the real and virtual environments and provides a technical environment that delivers high-quality XR content in real time via various XR devices.
[0004] Gartner’s 2023 Top 10 Strategic Technology Trends predicted that a combination of XR cloud and digital twins would be used in metaverse-based projects worldwide by 2027. Gartner predicts that XR cloud will have a very high impact as it will change the way people interact with the world around them, and reports that XR cloud will provide a digital abstraction layer for people, places, and things, securing a foothold across business and consumer applications and impacting all industries regardless of region.
[0005] As such, the application of cloud computing technology is necessary to overcome the technical limitations (processing capabilities of XR devices, network latency) of Extended Reality (XR), which is emerging as a new future market. If the Motion-to-Photon (MTP) response time to user input or actions exceeds 20ms, a phenomenon known as VR Sickness occurs, severely degrading the Quality of Experience (QoE). Since XR content is generally much larger than standard content (typically over 12K), failure to utilize cloud computing technology that takes these characteristics into account inevitably leads to increased MTP latency due to transmission issues with large-capacity content.
[0006] For example, NVIDIA offers a service through its game cloud platform, GeForce Now, that allows users to enjoy high-spec games regardless of the performance of their devices. However, the service has not achieved great success as it struggles to provide high-quality content at the level users desire due to high transmission rates during the process of streaming large-capacity media or 3D game content in real time.
[0007] delete Prior art literature
[65535] Republic of Korea Registered Patent No. 10-2669261 Cloud XR-based Program Virtualization Method (Registration Date: May 21, 2024) The problem to be solved
[0008] Therefore, the objective of the present invention is to construct a computing system capable of providing high-quality XR content services by utilizing microservice-based cloud-native in a multi-cloud environment to offload XR content processing, which requires significant computing resources, to the cloud environment for processing, and to transmit it to user XR devices in a lightweight manner for ultra-low latency. means of solving the problem
[0010] An XR cloud computing system for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment according to the present invention for achieving the above objective comprises: a computing virtualization unit that supports a multi-cloud environment, virtualizes computing resources, and controls microservices; a content loading unit that loads large-capacity XR content; a content rendering unit that renders XR content in a cloud environment; a content encoding unit that performs optimization work to transmit rendered XR content with ultra-low latency; a content interaction unit for transmitting XR content and interacting with a user's XR device; a cache unit for accelerating the processing speed of the above modules; and an XR device SDK unit for decoding transmitted XR content and rendering it to a user device.
[0011] Here, the computing virtualization unit can virtualize computing for processing XR content and control XR content processing microservices in a multi-cloud environment using cloud environments of various cloud providers such as AWS, Azure, and Google.
[0012] And the content loading department can load content to service broadcasts, 3D games, etc. as XR content.
[0013] Here, the content rendering unit may render the XR content loaded by the content loading unit on the cloud first to undergo an encoding process, rather than transmitting it directly to the user's XR device.
[0014] And the content encoding unit can encode XR content rendered by the content rendering unit in real time for ultra-low latency transmission.
[0015] Here, the content interaction unit can transmit encoded XR content to the user's XR device and also receive the user's interaction from the user's XR device and transmit it to the XR cloud system.
[0016] And the cache unit can provide in-memory caching for various data to improve the processing speed of all modules running on the computing virtualization unit, such as content loading, rendering, encoding, and interaction.
[0017] Here, the XR device SDK part can decode the encoded and transmitted XR content, render it on the user's XR device, and transmit various interactions of the user to the XR cloud computing system. Effects of the invention
[0019] According to the present invention, computing virtualization of CPUs, GPUs, etc., capable of processing XR content by integrating a multi-cloud environment through a computer virtualization unit is provided, and the effect of efficient utilization of computing resources is achieved through Auto Scaling for microservices as well as computing environments.
[0020] In addition, by loading large-capacity multi-view XR content through the content loading section and pre-warming the loaded content in the cache, the content rendering section, which is a subsequent processing module, can quickly access the content.
[0021] In addition, through the content rendering unit, large-capacity XR content can be rendered in cloud computing rather than directly on XR devices with low computing power, thereby enabling high-quality XR content services.
[0022] In addition, the content encoding section has the effect of optimizing large-capacity XR content rendered in cloud computing for transmission over a network.
[0023] In addition, by transmitting encoded XR content to the user's XR device through the content interaction unit and also receiving the user's interaction in real time and transmitting it to cloud computing, it is possible to provide an ultra-low latency interaction service.
[0024] In addition, through the cache section, multi-view XR content, rendering results, encoding results, etc. are cached in an In-Memory storage, which has the effect of enabling ultra-low latency processing.
[0025] In addition, XR content encoded and transmitted through the XR device SDK is decoded and rendered on the user's XR device, and the user's interaction is transmitted to cloud computing in real time, which has the effect of enabling seamless, high-quality XR content services. Brief explanation of the drawing
[0027] FIG. 1 is an overall block diagram of an XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud native in a multi-cloud environment according to the present invention. Figure 2 is a block diagram of the computing virtualization unit. Fig. 3 is a block diagram of the content loading section. Fig. 4 is a block diagram of the content rendering section. Figure 5 is a block diagram of the content encoding section. Fig. 6 is a block diagram of the content interaction section. Fig. 7 is a block diagram of the cache section. Fig. 8 is a block diagram of the XR device SDK section. Specific details for implementing the invention
[0028] Hereinafter, an XR cloud computing system (1) for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment according to a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.
[0029] FIG. 1 is an overall block diagram of an XR cloud computing system (1) for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment according to the present invention, FIG. 2 is a block diagram of a computing virtualization unit (10), FIG. 3 is a block diagram of a content loading unit (20), FIG. 4 is a block diagram of a content rendering unit (30), FIG. 5 is a block diagram of a content encoding unit (40), FIG. 6 is a block diagram of a content interaction unit (50), FIG. 7 is a block diagram of a cache unit (60), and FIG. 8 is a block diagram of an XR device SDK unit (70).
[0030] Referring to FIGS. 1 to 7, the configuration of an XR cloud computing system (1) for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment is described.
[0031] An XR cloud computing system (1) for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment includes a computing virtualization unit (10), a content loading unit (20), a content rendering unit (30), a content encoding unit (40), a content interaction unit (50), a cache unit (60), and an XR device SDK unit (70).
[0032] The virtualization unit (10) supports a multi-cloud environment, virtualizes computing resources, and manages microservices.
[0033] The content loading section (20) can load large-capacity XR content.
[0034] The content rendering unit (30) can render XR content in a cloud environment.
[0035] The content encoding unit (40) can perform optimization work to transmit rendered XR content with ultra-low latency.
[0036] The content interaction unit (50) can transmit XR content and interact with the user's XR device.
[0037] The cache section (60) can accelerate the processing speed of the modules.
[0038] The XR device SDK part (70) can decode the transmitted XR content and render it to the user device.
[0039] As shown in Fig. 1, computing is virtualized on a multi-cloud, and on the virtualized computing, a content loading unit, a content rendering unit, a content encoding unit, a content interaction unit, and a cache unit each operate as microservices. Separately, the XR device SDK unit operates on the user's XR device.
[0040] As shown in Fig. 2, the computing virtualization unit first supports a multi-cloud environment to integrate computing resources such as CPUs and GPUs. On top of these integrated computing resources, the content loading unit, content rendering unit, content encoding unit, content interaction unit, and cache unit for XR content processing are each deployed as microservices. The computing virtualization unit is responsible for managing these microservices; when a specific microservice becomes overloaded, it instantaneously deploys more of that microservice within the available range of current computing resources to ensure that the bottleneck processing is processed quickly. If the surged processing ends and there are many idle microservices among the additionally deployed microservices, the deployed microservices are reclaimed to secure computing resources. Furthermore, if there are no more computing resources available to deploy microservices, additional computing resources are secured through an Auto Scaling policy.
[0041] As shown in Fig. 3, the content loading unit loads XR content such as broadcasts and 3D games. Generally, since broadcast content with a fixed viewpoint is insufficient for service on XR devices, multi-view video content captured simultaneously by multiple cameras is loaded in the case of broadcast content. Additionally, in the case of 3D games, since various viewpoint changes are possible within the game, multi-view game content is loaded in the same way as broadcast content. Because the loaded multi-view content is large in size, it is stored in a cache so that it can be quickly utilized in subsequent processing.
[0042] As shown in Fig. 4, the content rendering unit renders multi-view content in a cloud environment. Transmitting large-capacity multi-view content as is to render it on an XR device would inevitably result in a decline in content quality due to the limited computing power of the device. Therefore, multi-view content is rendered on cloud computing, and optimized content is transmitted to the user's XR device. The content rendering unit starts operating when a cache save completion signal is received from the content loading unit, and loads and renders the multi-view content from the cache. At this time, each multi-view content is rendered simultaneously in each microservice.
[0043] As shown in Fig. 5, the content encoding unit consists of a multi-view separator, a depth information extractor / estimator, a deduplication unit, a restoration information generator, and a data encoder. The content encoding unit operates simultaneously when multi-view content is rendered in the content rendering unit. The multi-view separator selects one of the rendered multi-view contents as the Main View and groups the remaining viewpoints into Additional Views. The depth information extractor / estimator generates depth map data by extracting depth information from each viewpoint image or estimating it using an artificial intelligence model. The deduplication unit utilizes the depth map data generated by the depth information extractor / estimator to remove parts from the Additional View that overlap with the Main View—that is, parts that can be safely removed—based on the Main View. It then cuts the non-overlapping parts from the Additional View into a Patch and appends them to the Main View. The restoration information generator constructs information on how to restore the original viewpoint view using the Main View and the Patch. The data encoder encodes the Main View, depth map data, Patch data, and restoration information, converts them into a form optimized for transmission over a network, and stores them in a cache.
[0044] As shown in FIG. 6, the content interaction unit consists of a bitstream generator / transmitter that transmits XR content and a user interaction receiver that receives user interactions from an XR device. The bitstream generator retrieves the encoding result stored in the cache from the content encoding unit and converts it into a bitstream for network transmission. The bitstream transmitter transmits the generated bitstream to the user terminal using a protocol designed for ultra-low latency transmission. The user interaction receiver receives the interaction between the user and the content sent from the user's XR device and transmits it to the content rendering unit. Based on the user interaction, the content rendering unit re-renders the corresponding multi-view content, and this content is re-encoded and transmitted to the user.
[0045] As shown in FIG. 7, the cache section consists of a data input / output API, a data reader, and an in-memory storage. The data input / output API provides APIs for storing data in the cache and loading the stored data. At this time, the API response speed must be O(1) or at least faster than O(logn) when the data size is n. The data reader determines whether the data being stored is binary data or string data, etc. The in-memory storage stores the data in a structure that can have the fastest response speed depending on the form of the read data.
[0046] As shown in FIG. 8, the XR device SDK section consists of a data decoder, a data renderer, and a user interaction transmitter. The data decoder decodes and restores the bitstream transmitted over the network to construct the original multi-view content. The constructed multi-view content is passed to the data renderer and rendered on the user's XR device. At this time, interactions between the user and the content are transmitted to the XR cloud computing system through the user interaction transmitter, and the XR cloud computing system processes the corresponding XR content with ultra-low latency processing to enable a seamless and immersive XR content service.
[0047] Due to the XR cloud computing system (1) for providing real-time XR content services by utilizing microservice-based cloud native in the above multi-cloud environment, computing virtualization such as CPUs and GPUs that can process XR content by integrating the multi-cloud environment through a computer virtualization unit is provided, and efficient utilization of computing resources is possible through Auto Scaling of microservices as well as computing environments.
[0048] In addition, a large volume of multi-view XR content is loaded through the content loading section, and the loaded content is pre-warmed in the cache, allowing the content to be accessed quickly in the content rendering section, which is a subsequent processing module.
[0049] In addition, through the content rendering unit, high-quality XR content services can be provided by rendering large-capacity XR content in cloud computing instead of rendering it directly on XR devices with low computing power.
[0050] In addition, the content encoding unit encodes large-capacity XR content rendered in cloud computing and optimizes it for transmission over a network.
[0051] In addition, an ultra-low latency interaction service is possible by transmitting encoded XR content to the user's XR device through the content interaction unit, and also by receiving the user's interaction in real time and transmitting it to cloud computing.
[0052] In addition, through the cache section, multi-view XR content, rendering results, encoding results, etc. are cached in an In-Memory storage, enabling ultra-low latency processing.
[0053] In addition, XR content encoded and transmitted through the XR device SDK is decoded and rendered on the user's XR device, and the user's interaction is also transmitted to cloud computing in real time, enabling seamless high-quality XR content services. Explanation of the symbols
[0055] 1: XR cloud computing system for providing real-time XR content services utilizing microservices-based cloud-native in a multi-cloud environment 10: Computing Virtualization Section 20: Content Loading Section 30: Content Rendering Section 40: Content Encoding Section 50: Content Interaction Section 60: Cache section 70: XR Device SDK Section
Claims
Claim 1 In an XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud-native in a multi-cloud environment, the system comprises: a computing virtualization unit that integrates and virtualizes CPU and GPU computing resources distributed across cloud environments of different cloud providers, additionally deploys or retrieves multiple XR content processing microservices according to content processing load, and expands computing resources according to an auto-scaling policy when available computing resources are insufficient; a content loading unit that loads multi-view XR content including multiple viewpoint images and pre-warms the loaded multi-view XR content in an in-memory storage; a content rendering unit that operates in response to a cache storage completion signal of the multi-view XR content and renders each of the multiple viewpoint images in parallel in different microservices; a system that selects one viewpoint image among the rendered multi-view XR content as the primary viewpoint image and groups the remaining viewpoint images as secondary viewpoint images, uses depth information of each viewpoint image to remove parts in the secondary viewpoint images that overlap with the primary viewpoint image and creates non-overlapping parts as patches, generates restoration information to restore the original viewpoint image from the primary viewpoint image and the patches, and encodes the primary viewpoint image, depth information, patches, and restoration information to the in-memory A content encoding unit that stores in a storage unit; a content interaction unit that converts the encoding result stored in the in-memory storage unit into a bitstream and transmits it to a user XR device, and transmits user interaction received from the user XR device to the content rendering unit so that multi-view XR content corresponding to the user interaction is re-rendered and re-encoded; and a cache unit that stores the processing target data and processing result of the content loading unit, the content rendering unit, the content encoding unit, and the content interaction unit in the in-memory storage unit as a structure corresponding to the data type.An XR cloud computing system for providing real-time XR content services utilizing microservices-based cloud-native in a multi-cloud environment, characterized by including an XR device SDK part for decoding transmitted XR content and rendering it on a user device. Claim 2 An XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud-native in a multi-cloud environment, wherein the computing virtualization unit virtualizes computing for processing XR content in a multi-cloud environment using cloud environments of various cloud operators and controls XR content processing microservices. Claim 3 An XR cloud computing system for providing real-time XR content services utilizing microservice-based cloud-native in a multi-cloud environment, wherein, in claim 1, the content loading unit loads the multi-view XR content corresponding to a plurality of cameras or a plurality of virtual viewpoints to provide XR content including broadcasting and 3D games. Claim 4 An XR cloud computing system for providing real-time XR content services utilizing microservice-based cloud-native in a multi-cloud environment, wherein, in claim 1, the content rendering unit loads the multi-view XR content from the in-memory storage and simultaneously renders each viewpoint image using a microservice corresponding to each viewpoint image. Claim 5 An XR cloud computing system for providing real-time XR content services utilizing microservice-based cloud-native in a multi-cloud environment, characterized in that, in claim 1, the content encoding unit comprises a multi-viewpoint separator, a depth information extractor or estimator, a deduplicater, a restoration information generator, and a data encoder. Claim 6 An XR cloud computing system for providing real-time XR content services utilizing microservice-based cloud-native in a multi-cloud environment, characterized in that, in claim 1, the content interaction unit comprises a bitstream transmitter that transmits the bitstream to the user XR device using an ultra-low latency transmission protocol, and a user interaction receiver that receives the user interaction and transmits it to the content rendering unit. Claim 7 An XR cloud computing system for providing real-time XR content services utilizing microservice-based cloud-native in a multi-cloud environment, characterized in that, in claim 1, the cache unit comprises a data input / output API, a data reader that reads the data type of the data to be stored, and an in-memory storage that stores the data in a structure with the fastest response speed according to the read data type. Claim 8 An XR cloud computing system for providing real-time XR content services utilizing microservice-based cloud-native in a multi-cloud environment, characterized in that, in claim 1, the XR device SDK part comprises a data decoder that decodes and restores the bitstream to construct the original multi-view XR content, a data renderer that renders the constructed multi-view XR content on the user XR device, and a user interaction transmitter that transmits user interactions to the XR cloud computing system.
Citation Information
Patent Citations
Method for providing XR contents and XR device for providing XR contents
US20210190530A1
Extended Reality Streaming with Integrated Copyright Protection
US20230401291A1
Cloud XR-based program virtualizing method
KR1020230054068A
Cross reality system with simplified programming of virtual content
WO2021118962A1
