Method, apparatus and storage medium for providing ar / mr application to 5g device

Abstract

A method is provided. The method includes selecting, with an augmented reality (AR) / mixed reality (MR) application, media content for playback, finding, with the AR / MR application, at least one fifth generation media streaming downlink (5GMSd) application server (AS) having an edge capability suitable for playback of the selected media content, selecting, with the AR / MR application, at least one 5GMSd AS for playback of the selected media content, and requesting, with the AR / MR application, playback of the selected media content through the selected at least one 5GMSd AS.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 232,540, filed August 12, 2021, and U.S. Provisional Application No. 63 / 232,544, filed August 12, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to fifth-generation (5G) augmented reality (AR) / mixed reality (MR), and more particularly to methods and apparatus for providing AR / MR applications to 5G devices. Background Technology

[0004] Third Generation Partner Program (3) rd The Generation Partnership Project (3GPP) TS26.501 defines the media streaming architecture for 5G networks. 3GPP provides a technical report on supporting augmented reality (AR) / mixed reality (MR) applications. 3GPP TR 26.998 defines support for glasses-based AR / MR devices in 5G networks. This considers two device categories: devices fully capable of decoding and playing complex AR / MR content (i.e., stand-alone AR (STAR)) and devices with smaller computing resources and / or smaller physical size (i.e., smaller battery) that can only run such applications when most of the computation is performed on a 5G edge server, network, or cloud rather than on the device (i.e., edge-dependent AR (EDGAR)).

[0005] The media session processor can select the edge server. However, this selection can occur before the AR / MR application determines a subset of the media components of the immersive scene to be streamed. Therefore, the required computational resources are affected by the AR / MR application options, which is not currently considered. Furthermore, the AR / MR application and media session processor should operate together or be operated by the AR / MR application provider via the 5G media streaming downlink (5GMSd) application function (AF), which is also not currently considered. Additionally, it is unclear how the AR / MR application selects the media components of the scene to stream to the device without receiving the complete scene map. Finally, the issue of scene updates is not addressed in the current workflow. Summary of the Invention

[0006] One aspect of this disclosure provides a method. The method may include: using an AR / MR application to select media content for playback; using the AR / MR application to locate at least one 5GMSd application server (AS) with edge capabilities suitable for playing the selected media content; using the AR / MR application to select at least one 5GMSd AS for playing the selected media content; and using the AR / MR application to request playback of the selected media content through the selected at least one 5GMSd AS.

[0007] One aspect of this disclosure provides an apparatus. The apparatus may include at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to the instructions of the program code. The program code includes: first selection code configured to cause the at least one processor to use an AR / MR application to select media content for playback; first discovery code configured to cause the at least one processor to use the AR / MR application to locate at least one 5GMSd AS with edge capabilities suitable for playing the selected media content; second selection code configured to cause the at least one processor to use the AR / MR application to select at least one 5GMSd AS for playing the selected media content; and first request code configured to cause the at least one processor to use the AR / MR application to request playback of the selected media content through the selected at least one 5GMSd AS.

[0008] One aspect of this disclosure provides a non-transitory computer-readable medium capable of storing instructions. The instructions include one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to: select media content for playback using an AR / MR application; locate at least one 5GMSd AS with edge capabilities suitable for playing the selected media content using the AR / MR application; select at least one 5GMSd AS for playback of the selected media content using the AR / MR application; and request playback of the selected media content via the selected at least one 5GMSd AS using the AR / MR application. Attached Figure Description

[0009] Further features, properties, and various advantages of the subject matter of this disclosure will become more apparent from the following detailed description and accompanying drawings, in which:

[0010] Figure 1 This is an environmental diagram illustrating how the methods, apparatus, and systems described herein can be implemented according to embodiments.

[0011] Figure 2yes Figure 1 Example component block diagram of one or more devices.

[0012] Figure 3 This is a media architecture diagram for media uplink streaming according to an embodiment.

[0013] Figure 4 This is a media architecture diagram for media downlink streaming according to an embodiment.

[0014] Figure 5A and Figure 5B This is a download architecture diagram of EDGAR 5GMSd according to an embodiment.

[0015] Figure 6A , Figure 6B , Figure 6C and Figure 6D This is an operation flowchart of 5G downlink streaming based on EDGAR according to an embodiment.

[0016] Figure 7A and Figure 7B This is an operation flowchart of 5G downlink streaming based on EDGAR according to an embodiment.

[0017] Figure 7C This is an operation flowchart of an EDGAR-based process for scene updating according to an embodiment.

[0018] Figure 8 This is a flowchart of the EDGAR-based 5G downlink streaming process according to an embodiment. Detailed Implementation

[0019] Figure 1 This is a diagram of an environment 100 in which the methods, apparatus, and systems described herein can be implemented according to embodiments. (See diagram for example.) Figure 1 As shown, environment 100 may include user equipment 110, platform 120, and network 130. Devices in environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0020] User equipment 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 120. For example, user equipment 110 may include computing devices (e.g., desktop computers, laptop computers, tablet computers, handheld computers, smart speakers, servers, etc.), mobile phones (e.g., smartphones, cordless phones, etc.), wearable devices (e.g., smart glasses or smartwatches), or similar devices. In some embodiments, user equipment 110 may receive information from and / or send information to platform 120.

[0021] Platform 120 includes one or more devices as described elsewhere herein. In some embodiments, platform 120 may include a cloud server or a group of cloud servers. In some embodiments, platform 120 may be designed to be modular, allowing components of the software to be loaded or unloaded as needed. This allows platform 120 to be easily and / or quickly reconfigured for different purposes.

[0022] In some implementations, as shown in the figure, platform 120 may be located in a cloud computing environment 122. It should be noted that although the implementations described herein place platform 120 in a cloud computing environment 122, in some implementations, platform 120 may also be non-cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0023] The cloud computing environment 122 includes an environment that houses the platform 120. The cloud computing environment 122 can provide computing, software, data access, storage, and other services that do not require the end user (e.g., user equipment 110) to know the physical location and configuration of the system(s) and / or device(s) housing the platform 120. As shown in the figure, the cloud computing environment 122 may include a set of computing resources 124 (the total computing resources are referred to as "multiple computing resources 124", and individual computing resources are referred to as "computing resources 124").

[0024] Computing resource 124 includes one or more personal computers, workstations, server devices, or other types of computing and / or communication devices. In some embodiments, computing resource 124 may house platform 120. Cloud resources may include computing instances executing in computing resource 124, storage devices provided by computing resource 124, data transmission devices provided by computing resource 124, etc. In some embodiments, computing resource 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0025] For example Figure 1 As shown, computing resources 124 include cloud resource groups, such as one or more applications (APP) 124-1, one or more virtual machines (VM) 124-2, virtualized storage (VS) 124-3, one or more hypervisors (HYP) 124-4, etc.

[0026] Application 124-1 includes one or more software applications that can be provided to or accessed by user device 110 and / or platform 120. Application 124-1 can prevent the installation and execution of software applications on user device 110. For example, application 124-1 may include software associated with platform 120 and / or any other software that can be provided by cloud computing environment 122. In some implementations, an application 124-1 may send information to or receive information from one or more other applications 124-1 via virtual machine 124-2.

[0027] Virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on its usage and its correspondence to any real machine. A system virtual machine can provide a complete system platform supporting the execution of a full operating system (OS). A process virtual machine can execute a single program and can support a single process. In some implementations, virtual machine 124-2 can execute on behalf of a user (e.g., user device 110) and can manage the basic functions of the cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.

[0028] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that utilize virtualization technology in the storage system or device of computing resource 124. In some implementations, the type of virtualization in the context of the storage system may include block virtualization and file virtualization. Block virtualization may refer to abstracting (or separating) logical storage from physical storage, enabling access to the storage system without regard to physical storage or heterogeneous architecture. This separation provides storage system administrators with flexibility in how to manage storage for end users. File virtualization can eliminate the dependency between data accessed at the file level and the location of physical storage files. This can optimize storage usage, server consolidation, and / or performance for non-disruptive file migration.

[0029] Hypervisor 124-4 provides hardware virtualization technology that enables multiple operating systems (e.g., "guest operating systems") to execute in parallel on a host computer such as computing resource 124. Hypervisor 124-4 can present a virtual operating platform to the guest operating system and manage the execution of the guest operating system. Multiple instances of various operating systems can share virtualized hardware resources.

[0030] Network 130 includes one or more wired and / or wireless networks. For example, network 130 may include cellular networks (e.g., 5G networks, Long-Term Evolution (LTE) networks, 3G networks, etc.). th Generation (3G) networks, code division multiple access (CDMA) networks, public land mobile networks (PLMN), local area networks (LAN), wide area networks (WAN), metropolitan area networks (MAN), telephone networks (e.g., public switched telephone network (PSTN)), private networks, ad hoc networks, intranets, the Internet, fiber-optic networks, and / or combinations of these or other types of networks.

[0031] As an example, Figure 1 The document provides the quantity and layout of the equipment and networks. In practice, it is consistent with... Figure 1 Compared to the equipment and / or network shown, there may be other equipment and / or networks, fewer equipment and / or networks, different equipment and / or networks, or equipment and / or networks with different arrangements. Furthermore, Figure 1 The two or more devices shown can be implemented in a single device, or, Figure 1 The single device shown can be implemented as multiple distributed devices. Alternatively or alternatively, a group of devices in environment 100 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in environment 100.

[0032] Figure 2 yes Figure 1 Example component block diagram of one or more devices. Device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, device 200 may include bus 210, processor 220, memory 230, storage component 240, input component 250, output component 260 and communication interface 270.

[0033] Bus 210 includes components enabling communication between components of device 200. Processor 220 is implemented in hardware, firmware, or a combination of hardware and software. Processor 220 is a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Accelerated Processing Unit (APU), microprocessor, microcontroller, Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), or other types of processing components. In some embodiments, processor 220 includes one or more processors that can be programmed to perform functions. Memory 230 includes random access memory (RAM), read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic storage, and / or optical storage) that store information and / or instructions for use by processor 220.

[0034] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state drives), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, and / or other types of non-transitory computer-readable media, and corresponding drives.

[0035] Input component 250 includes components that enable device 200 to receive information via, for example, user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, and / or microphone). Alternatively or additionally, input component 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, accelerometer, gyroscope, and / or actuator). Output component 260 includes components that provide output information from device 200 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0036] Communication interface 270 includes transceiver-like components (e.g., a transceiver and / or separate receiver and transmitter) that enable device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 270 allows device 200 to receive information from and / or provide information to other devices. For example, communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0037] Device 200 can perform one or more methods described herein. Device 200 can perform these methods when processor 220 executes software instructions stored on a non-transitory computer-readable medium such as memory 230 and / or storage component 240. Computer-readable medium is defined herein as a non-transitory memory device. A memory device includes storage space within a single physical storage device or storage space distributed across multiple physical storage devices.

[0038] Software instructions can be read into memory 230 and / or storage component 240 from other computer-readable media or devices via communication interface 270. When the software instructions stored in memory 230 and / or storage component 240 are executed, processor 220 may perform one or more methods described herein. Alternatively or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more methods described herein. Therefore, the implementations described herein are not limited to any particular combination of hardware circuitry and software.

[0039] As an example, Figure 2 The document provides the number and arrangement of components. In practice, it is consistent with... Figure 2 Compared to the components shown, there may be other components, fewer components, different components, or components arranged differently. Additionally or alternatively, a group of components of device 200 (e.g., one or more components) may perform one or more functions described as being performed by another group of components of device 200.

[0040] A 5GMS system can be a collection of application functions, application servers, and interfaces from the 5G media streaming architecture, supporting downlink media streaming services, uplink media streaming services, or both. A 5GMS application provider can be a party that interacts with the functions of the 5GMS system and provides 5GMS-aware applications that interact with the functions of the 5GMS system. A 5GMS-aware application can refer to an application provided by a 5GMS application provider in user equipment (UE), which contains the service logic of 5GMS application services and interacts with other 5GMS clients and network functions through interfaces and application programming interfaces (APIs) defined in the 5GMS architecture. A 5GMS client can refer to a UE function that is a 5GMSd client, a 5GMS uplink (5GMSu) client, or both a 5GMSd client and a 5GMSu client.

[0041] A 5GMSd client can refer to a UE function that includes at least a 5G media streaming player and a media session processor for downlink streaming, and can be accessed through an explicitly defined interface / API. A 5GMSu client can refer to an initiator of a 5GMSu service that can be accessed through an explicitly defined interface / API. A 5GMSu media streamer can refer to a UE function that enables uplink transmission of streaming content to the AS function of a 5GMS application provider, and that can interact with both the 5GMSu-aware application for media capture and subsequent streaming, and the media session processor for media session control.

[0042] Dynamic policies can refer to dynamic policy and charging control (PCC) rules used for uplink or downlink application media streams during a media session. An egest session can refer to an uplink media stream session from a 5GMS AS to a 5GMS Application Provider. An ingest session can refer to a session that uploads media content to a 5GMSd AS. A policy template can refer to a set of (semi-static) policy or control function (PCF) / network exposure function (NEF) API parameters specific to the 5GMS application provider, and the resulting PCC rules. A policy template ID identifies the desired policy template, which the 5GMSd AF uses to select the appropriate PCF / NEF API for the 5G system so that the PCF can compile the desired PCC rules. A media player entry point can refer to a document or pointer to a document that defines media presentation (e.g., a media presentation description (MPD) for dynamic adaptive streaming over HTTP (DASH) or a uniform resource locator (URL) for a video clip file). A media streamer entry point can refer to a pointer to an entry point that defines an uplink media streaming session (e.g., in the form of a URL). A presentation entry point can refer to a document that defines application presentation or a pointer to a document, such as an HTML5 document.

[0043] A provisioning session can refer to a data structure provided at the interface (M1d) by a 5GMSd application provider that configures 5GMSd features associated with a set of 5GMSd-aware applications. A 5GMSd media player can refer to a UE function that enables playback and rendering of media based on a media playback entry point and provides basic controls to the 5GMSd-aware application, such as play, pause, search, and stop. Server access information can refer to a set of parameters and addresses (including the addresses of the 5GMSd AF and 5GMSd AS) required to activate the reception of a streaming media session. Service and content lookup can refer to the functions and processes provided by the 5GMSd application provider to the 5GMSd-aware application, enabling end users to find available streaming media services and content providers and select specific services or content items for access. Service announcements can refer to processes between the 5GMSd-aware application and the 5GMSd application provider that allow the 5GMSd-aware application to obtain 5GMS service access information directly or by referencing this information.

[0044] A third-party player can refer to a part of an application that uses the API to perform selected 5GMSd functions to play media content. A third-party uplink streamer can refer to a part of an application that uses the API to perform selected 5GMSu functions to capture and stream media content.

[0045] Figure 3 This is a diagram of a media architecture 300 for media uplink streaming according to an embodiment. A 5GMSu application provider 301 can use 5GMSu for uplink streaming media services. The 5GMSu application provider 301 can provide a 5GMSu-aware application 302 on the UE 303 to utilize a 5GMSu client 304 and network functions using interfaces and APIs defined in 5GMSu. The 5GMSu AS can be an AS dedicated to 5G media uplink streaming. The 5GMSu client 304 can be an internal function of the UE 303 dedicated to 5G media uplink streaming.

[0046] 5GMSu AF 306 and 5GMSu AS 305 can be Data Network (DN) 307 functions. The operator's network can trust functions in a trusted DN. Therefore, an AF in a trusted DN can communicate directly with all 5G core functions. Functions in external DNs can only communicate with 5G core functions via NEF 308 using link 320.

[0047] Media architecture 300 can connect the internal functions of UE 303 and related network functions for 5G media uplink streaming. Therefore, media architecture 300 can include multiple functions. For example, the 5GMSu client 304 on UE 303 can be the initiator of 5GMSu services accessible via an interface / API. 5GMSu client 304 can include two sub-functions: a media session processor 309 and a media streamer 310. The media session processor 309 can communicate with the 5GMSu AF 306 to establish, control, and support the delivery of media sessions. The media session processor 309 can expose APIs usable by the 5GMSu-aware application 302. The media streamer 310 can communicate with the 5GMSu AS 305 to stream media content and provide services for media capture and streaming to the 5GMSu-aware application 302, as well as services for media session control to the media session processor 309. The 5GMSu-aware application 302 can control the 5GMSu client 304 by implementing external application or content service provider-specific logic and enabling the establishment of media sessions. For example, the 5GMSu AS 305 can accommodate 5G media functions and can be implemented, for example, as a content delivery network (CDN). The 5GMSu application provider 301 can be an external application or content-specific media function, such as using 5GMSu to stream media from the 5GMSu-aware application 302 for media storage, consumption, transcoding, and redistribution. The 5GMSu AF 306 can provide various control functions to the media session processor 309 on the UE 303 and / or to the 5GMSu application provider 301. The 5GMSu AF 306 can relay or initiate requests for processing by different PCFs 311 or interact with other network functions.

[0048] Media architecture 300 may include multiple different interfaces. For example, link 321 may refer to M1u, which may be a 5GMSu-provided API opened by 5GMSu AF 306 to provide access to and feedback from media architecture 300. Link 322 may refer to M2u, which may be a 5GMSu-published API opened by 5GMSu AS 305 and used when a 5GMSu AS 305 in a trusted DN such as DN 307 is selected to receive content from streaming media services. Link 323 may refer to M3u, which may be an internal API for exchanging information about the content hosted on a 5GMSu AS 305 within a trusted DN such as DN 307. Link 324 may refer to M4u, which may be a media uplink streaming API opened by 5GMSu AS 305 to media streamer 310 for streaming media content. Link 325 can refer to M5u, which can be a media session processing API exposed by 5GMSu AF 306 to the media session processor for media session processing, control, and assistance. This API also includes appropriate security mechanisms such as authorization and authentication. Link 326 can refer to M6u, which can be a UE 303 media session processing API exposed by media session processor 309 to 5GMSu-aware application 302 to utilize 5GMSu functionality. Link 327 can refer to M7u, which can be a UE media streaming API exposed by media streaming transmitter 310 to 5GMSu-aware application 302 and media session processor 309 to utilize media streaming transmitter 310. Link 328 can refer to M8u, which can be an application API used for exchanging information between 5GMSu-aware application 302 and 5GMSu application provider 301, such as providing service access information to 5GMSu-aware application 302. UE 303 can also be implemented in a self-contained manner, so that interfaces M6u326 and M7u327 are not exposed.

[0049] Figure 4 This is a diagram of a media architecture 400 for downlink streaming media according to an embodiment. A 5GMSd application provider 401 can use 5GMSd for downlink streaming media services. The 5GMSd application provider 401 can provide a 5GMSd-aware application 402 on the UE 403 to utilize a 5GMSd client 404 and network functions using the interfaces and APIs defined in 5GMSd. The 5GMSd AS can be an AS dedicated to 5G downlink streaming media. The 5GMSd client 404 can be an internal function of the UE 403 dedicated to 5G downlink streaming media.

[0050] 5GMSd AF 406 and 5GMSd AS 405 can be functions of DN 407. The operator's network can trust functions in a trusted DN. Therefore, an AF in a trusted DN can communicate directly with all 5G core functions. Functions in external DNs can only communicate with 5G core functions via NEF 408 using link 420.

[0051] Media architecture 400 can connect the internal functions of UE 403 and related network functions for 5G media downlink streaming. Therefore, media architecture 400 can include multiple functions. For example, a 5GMSd client 404 on UE 403 can be a receiver of 5GMSd services accessible via an interface / API. 5GMSd client 404 can include two sub-functions: a media session processor 409 and a media streamer 410. The media session processor 409 can communicate with the 5GMSd AF 406 to establish, control, and support the delivery of media sessions. The media session processor 409 can expose APIs that can be used by the 5GMSd-aware application 402. The media streamer 410 can communicate with the 5GMSd AS 405 to stream media content and provide services for media playback to the 5GMSd-aware application 402, as well as services for media session control to the media session processor 409. The 5GMSd-aware application 402 can control the 5GMSd client 404 by implementing external application or content service provider-specific logic and enabling the establishment of media sessions. The 5GMSd AS 405 can accommodate 5G media functions. The 5GMSd application provider 401 can be an external application or content-specific media function, such as using 5GMSd to create, encode, and format media for streaming media to the 5GMSd-aware application 402. The 5GMSd AF 406 can provide various control functions to the media session processor 409 on the UE 403 and / or to the 5GMSd application provider 401. The 5GMSd AF 406 can relay or initiate requests for processing by different PCFs 411 or interact with other network functions.

[0052] Media architecture 400 may include multiple different interfaces. For example, link 421 may refer to M1d, which may be a 5GMSd API provided by 5GMSd AF 406 to provide access to and feedback from media architecture 400. Link 422 may refer to M2d, which may be a 5GMSd ingestion API provided by 5GMSd AS 405 and used when a 5GMSd AS 405 in a trusted DN such as DN 407 is selected to receive content from streaming media services. Link 423 may refer to M3d, which may be an internal API for exchanging information about the content hosted on a 5GMSd AS 405 within a trusted DN such as DN 407. Link 424 may refer to M4d, which may be a media downlink streaming API provided by 5GMSd AS 405 to media streamer 410 for streaming media content. Link 425 can refer to M5d, which can be a media session processing API opened by 5GMSd AF 406 to the media session processor for media session processing, control, and assistance. This API also includes appropriate security mechanisms such as authorization and authentication. Link 426 can refer to M6d, which can be a UE 403 media session processing API opened by media session processor 409 to 5GMSd-aware application 402 to utilize 5GMSd functionality. Link 427 can refer to M7d, which can be a UE media player API opened by media streamer 410 to 5GMSd-aware application 402 and media session processor 409 to utilize media streamer 410. Link 428 can refer to M8d, which can be an application API used for exchanging information between 5GMSd-aware application 402 and 5GMSd application provider 401, such as providing service access information to 5GMSd-aware application 402.

[0053] Figure 5A and Figure 5BThis is a download architecture diagram of the EDGAR 5GMSd according to an embodiment. The AR EDGAR 5GMSd architecture 500 can also be applied to MR. Architecture 500 includes a 5G EDGAR UE 502, a cloud / edge server 504, and an AR / MR application provider 506. The 5G EDGAR UE 502 includes an AR runtime 508, a lightweight scene manager 510, a 5GMS client 512, and a basic AR / MR application 514. The AR runtime 508 includes a visual engine / simultaneous localization and mapping (SLAM) 516, a pose correction module 518, and a sound field mapping module 520. The lightweight scene manager 510 includes a basic scene graph handler (SGH) 522 and a synthesizer 524. The 5GMS client 512 includes a media session processor (including edge sessions) 526 and a media client 528. The media client 528 includes a scene description delivery module 530, a content delivery module 532, and a basic codec 534. The 5G EDGAR UE 502 includes a 5G lightweight system (Uu) 536 that communicates with a 5G system (gNb) 538 of a cloud / edge server 504.

[0054] The cloud / edge server 504 includes AR / MR applications 540 and media delivery functionality 542. Media delivery functionality 542 includes media AF 544 and media AS 546. Media AS 546 includes content delivery functionality 548, a scene description function (SDF) 550, a decoder 552, and an encoder 554. AR / MR applications 540 include an AR scene manager 556, AR functionality 564, a semantic awareness module 566, a social integration module 568, and a media asset database 570. The AR scene manager 556 includes a scene graphics generator 558, an immersive visual renderer 560, and an immersive audio renderer 562.

[0055] AR / MR application provider 506 includes immersive media server 574. Immersive media server 574 includes scene description module 576, manifest server 578, and clip server 580.

[0056] Media client 528 sends data (e.g., gestures and interactions) to content delivery function 548 via M4 interface 582. Content delivery function 548 sends data (e.g., pre-rendered media) to media client 528 via M4 interface 584. Media session processor 526 communicates with media AF 544 via M5d interface 586. Basic AR / MR application 514 communicates with AR / MR application 540 via M8 interface 588. Basic AR / MR application 514 can receive user input 591 and AR data 590 from AR runtime 508. AR runtime 508 can receive data from camera 592 and sensor 593, and can output the data to display 594 and speaker 595.

[0057] According to an embodiment, the SGH 522 of UE 502 can be capable of parsing a full scene graph (FSG) provided by AR / MR application 540. The SGH 522 can provide the AR / MR application 540 with a scene description having a list of media components and their descriptions. The AR / MR application 540 selects a subset / list of media components from the FSG and provides the subset of media components to SDF 550. SDF 550 is configured to generate a simplified scene graph (SSG) from the FSG by combining one or more media components and reducing the number of records in the FSG. SDF 550 is also configured to arrange media decoder 552 and media encoder 554, along with content delivery function 548, on the edges to prepare the corresponding media components used in the SSG. The SSG can be passed to UE 502. The SCH 522 of UE 502 is configured to parse the SSG and provide it to compositor 524 of UE 502. Compositor 524 is configured to synthesize a simplified scene graph, such as the SSG. UE 502 is configured to stream necessary media components from content delivery function 548, and UE 502 is configured to decode and synthesize media components in a simplified scenario.

[0058] Based on the FSG updated during a media session, the system according to an embodiment includes two capabilities. First, based on the SDF 550 being configured to convert the updated FSG into an updated SSG without losing any updates, the SDF 550 generates the updated SSG and delivers the updated SSG to the UE 502. If the updated FSG exceeds the current characteristics of the SSG, the AR / MR application 540 can be notified of the FSG update. Then, the AR / MR application 540 is configured to select a new subset of media components from the FSG. If the new subset is the same as the previous selection, no scene update is required. If the new subset includes new media components or modifications to existing media components, the AR / MR application 540 notifies the SDF 550 of the new subset. Based on the received new media subset, the SDF 550 is configured to generate a new SSG according to the updated FSG and provide the new SSG to the UE 502. SDF 550 is also configured to rearrange media decoder 552, media encoder 554, and content delivery function 548 on cloud / edge server 504 to prepare the corresponding media components for use in the updated SSG. UE 502's SCH 522 is configured to parse the updated SSG, as well as stream the corresponding media components, synthesize the media components, and play the synthesized media components.

[0059] Figure 6A , Figure 6B , Figure 6C and Figure 6D This is an operational flowchart of 5G downlink streaming based on EDGAR according to an embodiment. The system executing the operational flow of Figure 6 may include a basic AR / MR application 606, an AR runtime 608, a lightweight scene manager 610, a media client 612, and a media session processor 614, which may be part of an EDGAR UE 602 (furthermore, the media client 612 and media session processor 614 may be part of a media access function). The system also includes a 5GMSd AF / EES 616, a 5GMSd AS / edge application server (EAS) 618, and an AR / MR application 620, which may be part of a cloud / edge server 604. The system may also include an AR / MR application provider 622.

[0060] In operation 630, the system provides service announcements and content lookups. Service announcements can be triggered by AR / MR application 606. Service access information may include media player entries or references to service access information and can be provided via the M8d interface. In operation 632, the system selects the desired media content. In operation 634, the system initiates media playback. AR / MR application 606 can trigger media session processor 614 to start media playback and can provide media player entries to media client 612. In operation 636, the system performs service access information acquisition. When AR / MR application 606 only receives references to service access information, media session processor 614 interacts with 5GMSd AF 616 to obtain the complete service access information. In operation 638, AR / MR application 606 searches for 5GMSd AS with the desired edge capabilities. In operation 640, the system provides a list of one or more available 5GMSd AS / EAS.

[0061] In operation 642, AR / MR application 606 and media session processor 614 select the most suitable 5GMSd AS. In operation 644, AR / MR application 606 provides initial scene parameters through lightweight scene manager 610 and media client 612. In operation 646, 5GMSd AS 618 provides a new entry point for SSG. Alternatively, AR / MR application provider 622 can select or instantiate a 5GMSd for running AR / MR edge application 620, making operations 642, 644, and 646 optional. In operation 648, AR / MR application 606 requests scene manager 610 to play the SSG entry point.

[0062] In operation 650, the system provides a simplified entry point for the scenario to media client 612. In operation 652, the system establishes a transport session for the entry point. In operation 654, the system requests the entry point from 5GMSd AS 618. In operation 656, the system acknowledges the request. In operation 658, the system processes the entry point. In operation 660, the system provides an entry point reception notification to media session processor 614. In operation 662, the system configures the required Quality of Service (QoS) for one or more streaming media sessions. In operation 664, the system establishes a transport session for one or more delivery lists. In operation 666, the system requests one or more delivery lists from 5GMSd AS 618. In operation 668, the system acknowledges the request for one or more delivery lists.

[0063] In operation 670, the system processes one or more delivery manifests. In operation 671, the system provides the media session processor 614 with notification of receipt of one or more delivery manifests. In operation 672, the system captures the media pipeline. In operation 673, the system establishes a transport session for the content. In operation 674, the system provides the media session processor 614 with notification of the transport session. In operation 675, the system requests pre-rendered media and gesture information from 5GMSd AS 618. In operation 676, the system renders a two-dimensional view. In operation 677, the system provides pre-rendered media to the media client 612. In operation 678, the system decodes and processes the media data.

[0064] In operation 680, media client 612 provides media data to lightweight scene manager 610. In operation 682, the system performs compositing, pose correction, and media rendering. In operation 684, AR / MR application provider 622 provides a complete scene update to 5GMSd AS618. In operation 686, the system provides a simplified scene update.

[0065] Figure 7A and Figure 7B This is an operational flowchart of 5G downlink streaming based on EDGAR according to an embodiment. The system executing the operational flow of Figure 7 may include a basic AR / MR application 706, an AR runtime 708, a lightweight scene manager 710, a media client 712, and a media session processor 714, which may be part of an EDGAR UE 702 (furthermore, the media client 712 and media session processor 714 may be part of a media access function). The system also includes 5GMSdAF / EES 716, 5GMSdAS / EAS 718, and an AR / MR application 720, which may be part of a cloud / edge server 704. The system may also include an AR / MR application provider 722.

[0066] In operation 730, the system provides service announcements and content lookups. Service announcements can be triggered by AR / MR application 706. Service access information may include a media player entry or a reference to service access information and can be provided via the M8d interface. In operation 732, the system selects the desired media content. In operation 734, the system initiates media playback. AR / MR application 706 can trigger media session processor 714 to start media playback and can provide the media player entry to media client 712. In operation 736, the system performs service access information retrieval. When AR / MR application 706 only receives a reference to service access information, media session processor 714 interacts with 5GMSd AF 716 to obtain the complete service access information.

[0067] In operation 738, the system requests the FSG. Given a media entry point, AR / MR application 706 can request a complete scene description through scene manager 710. In operation 740, the system selects initial scene components. That is, AR / MR application 706 selects initial scene components and derives the required edge capabilities. In operation 742, AR / MR application 706 searches for a 5GMSd AS with the desired edge capabilities. In operation 742, the system provides a list of one or more available 5GMSd AS / EAS. In operation 744, the system provides a list of one or more available 5GMSd AS / EAS. In operation 746, AR / MR application 706 and media session processor 714 select the most suitable 5GMSd AS. In operation 748, AR / MR application 706 provides initial scene parameters through lightweight scene manager 710 and media client 712. In operation 750, 5GMSd AS 718 provides a new entry point for SSG. In operation 752, AR / MR application 706 requests scene manager 710 to play the SSG entry point.

[0068] Figure 7C This is a flowchart illustrating the operation of an EDGAR-based process for scene updates according to an embodiment. In operation 760, the AR / MR application provider 722 provides a full scene update during streaming. In operation 762, the 5GMSd AS 718 checks whether a simplified scene update can be performed. Based on the possibility of a simplified scene update, in operation 764, the 5GMSd AS 718 updates the simplified scene and provides it to the scene manager 710 of the UE 702.

[0069] Since a simplified scene update cannot be performed, the system executes operations 766 to 772. In operation 766, the 5GMSd AS 718 provides a complete scene update to the AR / MR application 706. In operation 768, the AR / MR application 706 selects initial scene components. In operation 770, the AR / MR application 706 provides the selected components to the 5GMSd AS 718. In operation 772, the 5GMSd AS 718 generates an updated simplified scene and provides this updated simplified scene to the scene manager 710 of the UE 702.

[0070] Figure 8This is a flowchart illustrating the process of EDGAR-based 5G downlink streaming according to an embodiment. In operation 802, the system uses an AR / MR application to select media content for playback. In operation 804, the system uses an AR / MR application to locate at least one 5GMSd AS with edge capabilities suitable for playing the selected media content. In operation 806, the system uses an AR / MR application to select at least one 5GMSd AS for playing the selected media content. In operation 808, the system uses an AR / MR application to request playback of the selected media content via the selected at least one 5GMSd AS.

[0071] Although Figure 8 An example block of process 800 is shown, but in some implementations, process 800 may include... Figure 8 The blocks shown are compared to additional blocks, fewer blocks, different blocks, or blocks with different arrangements. Alternatively or concurrently, two or more blocks of process 800 can be executed in parallel.

[0072] Therefore, according to the embodiment, the AR / MR application receives the FSG and determines which components of the scene graph are streamed (i.e., the AR / MR application is not only provided with the SSG). Edge resources are requested based on the processing requirement to convert the FSG and its media components into SSGs and their media components. Updates on the FSG can be processed according to the breadth of updates. If an update can be achieved by updating the SSG, the update occurs on the edge / cloud server. Otherwise, the FSG update is provided to the AR / MR application, and the AR / MR application can select from the updated FSGs. Therefore, updated SSGs are generated based on the selection made by the AR / MR application.

[0073] Furthermore, the proposed methods can be implemented using processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium to perform one or more of the proposed methods.

[0074] The above-mentioned technology can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media.

[0075] The embodiments of this disclosure can be used individually or in any combination in any order. Furthermore, each embodiment (and its methods) can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0076] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation of the precise forms disclosed. Modifications and alterations may be made based on the foregoing disclosure, or may be derived from practical implementation.

[0077] As used herein, the term component is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software.

[0078] Even if combinations of features are described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically described in the claims and / or not specifically disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes combinations of each dependent claim with all other claims in the claim set.

[0079] Unless explicitly stated otherwise, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, the indefinite article used herein is intended to include one or more items and may be used interchangeably with “one or more.” Additionally, the term “set” used herein is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more.” If the intention is to use an item, the term “a” or similar language is used. Furthermore, the terms “having,” “possessing,” “with,” etc., used herein are intended as open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.”

Claims

1. A method for providing augmented reality (AR) / mixed reality (MR) applications to 5G devices, characterized in that, The method includes: Select the media content to play; The AR / MR application requests a complete scene map of the selected media content and selects initial scene components based on the complete scene map. The AR / MR application locates at least one 5GMSd AS (5th generation media streaming downlink application server), which has edge capabilities suitable for playing selected media content. The edge capabilities are derived by the AR / MR application from the initial scene components and request edge resources based on the processing requirements of converting the complete scene map and the media components of the complete scene map into a simplified scene map and the media components of the simplified scene map. Using the AR / MR application to select at least one 5GMSd AS for playing selected media content; and The AR / MR application requests the playback of selected media content via at least one selected 5GMSd AS.

2. The method according to claim 1, characterized in that, The method also includes receiving full scene map updates from an AR / MR application provider using at least one selected 5GMSd AS.

3. The method according to claim 2, characterized in that, The method further includes: using at least one selected 5GMSd AS to determine whether the simplified scene graph of the selected media content is updatable.

4. The method according to claim 3, characterized in that, The method further includes: providing a simplified scene map update based on the full scene map update using at least one selected 5GMSd AS, based on the fact that the simplified scene map of the selected media content is updatable.

5. The method according to claim 3, characterized in that, The method further includes: based on the fact that the simplified scene graph determined by the selected media content cannot be updated: The complete scene map update is provided to the AR / MR application using at least one selected 5GMSd AS; Use the AR / MR application to select initial scene components; The AR / MR application provides selected initial scene components to at least one selected 5GMSd AS; and An updated simplified scene graph is generated using at least one selected 5GMSd AS, and the updated simplified scene graph is returned to the scene manager.

6. A device for providing augmented reality (AR) / mixed reality (MR) applications to 5G devices, characterized in that, The device includes: At least one memory is configured to store program code; and At least one processor is configured to read the program code to execute the method according to any one of claims 1 to 5.

7. A non-transitory computer-readable medium for storing instructions, characterized in that, When executed by one or more processors of the device, the instructions cause the one or more processors to perform the method according to any one of claims 1 to 5.

Images