Method and apparatus for transmitting 3D XR media data

Abstract

A method performed by a first terminal is provided. The method includes identifying a capability of the first terminal connected to at least one component device, establishing a session associated with an augmented reality (AR) service via a server based on the capability of the first terminal, performing pre-processing on three-dimensional (3D) media data acquired by the at least one component device, and transmitting the pre-processed 3D media data to a second terminal.

Description

Technical Field

[0001] This disclosure relates to methods and apparatus for providing extended reality (XR) experiences to users. More specifically, this disclosure relates to methods and apparatus for transmitting three-dimensional (3D) XR media data to another device. Background Technology

[0002] With recent advancements in communication and image processing technologies, services have emerged that provide extended reality (XR) experiences to users through various devices such as wearable or mobile devices. XR, or XR services, is a collective term encompassing virtual reality (VR), augmented reality (AR), mixed reality (MR), etc., referring to services that provide users with virtual environments or objects—either independently or in conjunction with real-world environments or objects—as if the virtual environment or objects were real or blended with the real world, allowing users to experience virtual environments or objects virtually generated by computing devices.

[0003] One method for providing XR services is to provide two-dimensional (2D) media content to a user via a display device. One method for providing 2D media content to a user via a display device includes capturing, encoding, decoding, and rendering 2D objects.

[0004] Furthermore, with the introduction of 3D media content that can be represented using point clouds or meshes, the possibility of providing 3D XR media content services to users using various devices has emerged. Devices used to provide 3D XR media content are not limited to 2D display devices; various display devices such as head-mounted displays (HMDs) or AR glasses can also be used. However, the processes for capturing, encoding, decoding, and rendering 3D objects may differ from those associated with 2D objects. For example, the process for handling 3D objects may be more complex than that for handling 2D objects, and the process for handling 2D objects may require little or no data preprocessing during capture or encoding operations. On the other hand, handling 3D objects may require data preprocessing, additional metadata, and sufficiently high signaling, processing, rendering, or display performance compared to handling 2D objects.

[0005] Therefore, there is a need to establish a method and process for providing 3D XR media content to users.

[0006] The above information is presented as background information only to aid in understanding this disclosure. No determination or assertion is made regarding whether any of the above content can be applied to this disclosure as prior art. Summary of the Invention

[0007] Solution

[0008] According to embodiments of this disclosure, a method and apparatus for performing augmented reality (AR) services between a first terminal and a second terminal are provided. Attached Figure Description

[0009] Figure 1A This is a schematic diagram illustrating a two-dimensional (2D) video stream and a 2D video call according to embodiments of the present disclosure;

[0010] Figure 1B This is a schematic diagram illustrating methods for providing virtual reality (VR) content and methods for providing augmented reality (AR) content according to embodiments of this disclosure;

[0011] Figure 2A This is a schematic diagram illustrating a method performed by a first user equipment (UE) to provide or receive three-dimensional (3D) extended reality (XR) media content to or from a second UE, according to embodiments of the present disclosure.

[0012] Figure 2B This is a schematic diagram illustrating a method performed by a first user to share XR media content with a second user, according to an embodiment of this disclosure.

[0013] Figure 3 This is a schematic diagram illustrating an XR service flow according to an embodiment of the present disclosure, in which objects existing in the environment of the first user are provided to a second user as 3D media objects;

[0014] Figure 4 This is a schematic diagram illustrating various device configurations that can be used to provide 3D XR media, according to embodiments of the present disclosure;

[0015] Figure 5 This is a schematic diagram illustrating the process of establishing an XR service session and providing XR services according to embodiments of the present disclosure;

[0016] Figure 6 This is a schematic diagram illustrating a user space set according to embodiments of the present disclosure;

[0017] Figure 7 This is a schematic diagram illustrating a stream for describing media data and metadata according to embodiments of the present disclosure;

[0018] Figure 8 This is a schematic diagram illustrating an XR media architecture for a UE according to embodiments of the present disclosure;

[0019] Figure 9 This is a schematic diagram illustrating a method for transmitting 3D XR media data to a second UE, performed by a first UE, according to an embodiment of the present disclosure.

[0020] Figure 10This is a schematic diagram illustrating the configuration of a UE or component device according to embodiments of this disclosure;

[0021] Figure 11 This is a schematic diagram illustrating the configuration of a server according to embodiments of the present disclosure;

[0022] Figure 12 This is a flowchart illustrating a method executed by a first terminal according to an embodiment of this disclosure; and

[0023] Figure 13 This is a flowchart illustrating a method performed by a second terminal according to an embodiment of the present disclosure. Detailed Implementation

[0024] Several aspects of this disclosure address at least the aforementioned problems and / or disadvantages, and provide at least the following advantages. Therefore, one aspect of this disclosure is to provide a method and apparatus for transmitting three-dimensional (3D) extended reality (XR) media data to another device.

[0025] Additional aspects will be set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practice of the presented embodiments.

[0026] According to one aspect of this disclosure, a method performed by a first terminal in a wireless communication system is provided. The method includes: identifying the capabilities of the first terminal connected to at least one component device; establishing a session associated with an augmented reality (AR) service via a server based on the capabilities of the first terminal; performing preprocessing on three-dimensional (3D) media data acquired by the at least one component device; and transmitting the preprocessed 3D media data to a second terminal.

[0027] According to another aspect of this disclosure, a method performed by a second terminal in a wireless communication system is provided. The method includes: identifying the capabilities of the second terminal connected to at least one component device; establishing a session associated with an augmented reality (AR) service via a server based on the capabilities of the second terminal; receiving three-dimensional (3D) media data from a first terminal; performing post-processing on the 3D media data; and rendering the post-processed 3D media data on the second terminal.

[0028] According to another aspect of this disclosure, a first terminal in a wireless communication system is provided. The first terminal includes a transceiver and at least one processor configured to identify the capabilities of the first terminal connected to at least one component device, establish a session associated with an augmented reality (AR) service via a server based on the capabilities of the first terminal, perform preprocessing on three-dimensional (3D) media data acquired via at least one component device, and transmit the preprocessed 3D media data to a second terminal via the transceiver.

[0029] According to another aspect of this disclosure, a second terminal in a wireless communication system is provided. The second terminal includes a transceiver and at least one processor configured to identify the capabilities of the second terminal connected to at least one component device, establish a session associated with an augmented reality (AR) service via a server based on the capabilities of the second terminal, receive three-dimensional (3D) media data from a first terminal via the transceiver, perform post-processing on the 3D media data, and render the post-processed 3D media data on the second terminal.

[0030] Other aspects, advantages, and distinctive features of this disclosure will become apparent to those skilled in the art from the following description of various embodiments disclosed in conjunction with the accompanying drawings.

[0031] The following description, provided with reference to the accompanying drawings, is intended to aid in a full understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. It includes various specific details to aid understanding, but these are to be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0032] The terms and words used in the following description and claims are not limited to their documentary meaning, but are used solely by the inventors to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is provided for illustrative purposes only and is not intended to limit the disclosure as defined by the appended claims and their equivalents.

[0033] It should be understood that the singular forms “a,” “an,” and “the” include plural indicators unless the context clearly specifies otherwise. Thus, for example, referring to “a component surface” includes referring to one or more such surfaces.

[0034] For the same reason, some components are exaggerated, omitted, or shown schematically in the accompanying drawings. Furthermore, the dimensions of each element do not perfectly reflect its actual size. Throughout the drawings, the same reference numerals always refer to the same or corresponding elements.

[0035] The advantages and features of this disclosure and its implementation methods will be more readily understood by referring to the following description of embodiments and accompanying drawings. However, this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided to make this disclosure comprehensive and complete, and to fully convey the concepts of this disclosure to those skilled in the art to which this disclosure pertains, and this disclosure will be defined only by the appended claims. Throughout this specification, the same reference numerals refer to the same elements.

[0036] Throughout the disclosure, the expression "at least one of a, b and c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b and c, or variations thereof.

[0037] Examples of terminals may include user equipment (UE), mobile station (MS), cellular phone, smartphone, computer, multimedia system capable of performing communication functions, etc.

[0038] In this disclosure, the controller may also be referred to as a processor.

[0039] Throughout the specification, a layer (or layer device) may also be referred to as an entity.

[0040] It will be understood that each block of the flowchart in the accompanying figures, and combinations of flowchart blocks, can be executed by computer program instructions. These computer program instructions can be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, and thus, the instructions executed by the processor of the computer or other programmable data processing apparatus create units for performing the functions specified in the flowchart blocks. The computer program instructions can also be stored in a computer-executable or computer-readable storage medium that can instruct the computer or another programmable data processing apparatus to perform functions in a particular manner; thus, the instructions stored in the computer-executable or computer-readable storage medium can produce items that include the meaning of instructions for performing the functions described in the flowchart blocks. The computer program instructions can also be loaded into a computer or another programmable data processing apparatus, and thus, when a series of operations are performed in the computer or other programmable data processing apparatus, the instructions for operating the computer or other programmable data processing apparatus by generating a process executed by the computer can provide operations for performing the functions described in the flowchart blocks.

[0041] Furthermore, each block may represent a module, segment, or portion of code comprising one or more executable instructions for performing specified logical functions(s). It should also be noted that in some alternative implementations, the functions mentioned in a block may appear out of order. For example, two blocks shown consecutively may execute substantially simultaneously, or these blocks may sometimes execute in reverse order according to their corresponding functions.

[0042] As used herein, the term "unit" refers to a software element or hardware element, such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC), that performs certain functions. However, the term "unit" is not limited to software or hardware. A "unit" may be configured to reside in an addressable storage medium or to operate one or more processors. Thus, the term "unit" can include, for example, elements such as software elements, object-oriented software elements, class elements, and task elements; processes; functions; attributes; procedures; subroutines; program code segments; drivers; firmware; microcode; circuits; data; databases; data structures; tables; arrays; and variables. The functionality provided by elements and "units" can be combined into a smaller number of elements and "units," or can be further divided into additional elements and "units." Furthermore, elements and "units" can be embodied in reproducing one or more central processing units (CPUs) in a device or secure multimedia card. Additionally, in embodiments of this disclosure, a "unit" may include one or more processors.

[0043] This disclosure can be applied to a variety of devices and XR services. For example, it can be applied to augmented reality (AR), AR wearable devices (e.g., AR glasses, head-mounted displays (HMDs), etc.), mobile AR wearable devices, standalone AR wearable devices, three-dimensional (3D) object modeling, 3D teleconferencing, session setup and establishment for XR services, cloud-assisted session management for providing XR services, virtual reality monitors (VRMs), mobile VR, television VR, etc. The fields of extended reality (XR) to which this disclosure can be applied may be determined in many different forms, not limited to the examples above.

[0044] In this disclosure, the term XR is a term that includes at least one of VR, AR, and Mixed Reality (MR). For example, AR glasses, AR objects, and VR services can be referred to as XR glasses, XR objects, and XR services, respectively.

[0045] In this disclosure, XR media content can include various types of media content. For example, XR media content can include 360-degree video content and 3D object-based media content (point clouds and meshes). In this disclosure, unless otherwise stated, XR media, XR media content, XR content, XR services, etc., all fall under the category of 3D content.

[0046] In this disclosure, "user's device" refers to one or more devices located around a user that acquire, process, transmit, or receive data to provide XR services to the user.

[0047] In this disclosure, "XR device" refers to a device that includes a display and provides XR content to a user via the display. The shape and properties of the display of an XR device can be determined in many different ways. For example, the display can be transparent, translucent, or opaque, and can be a flexible display, a foldable display, or a rigid display, wherein the display element is an organic light-emitting diode (OLED), an LED, a liquid crystal (LC), etc. The shape and properties of the display of an XR device can be determined in many different ways, not limited to the examples above. Furthermore, an XR device can be a wearable device that a user can wear (e.g., an HMD, XR glasses, etc.).

[0048] In this disclosure, "component apparatus" refers to an apparatus that performs at least one of the functions of "rendering," "visualization," and "capture" to provide XR services. Component apparatus can be a collective term for rendering apparatus, visual apparatus, and capture apparatus. Reference will be made below. Figure 4 Each function is described in detail. Component devices can be independent devices or device blocks included in another device.

[0049] Any one or more of various communication technologies can be used as communication technologies that can be used for communication between component devices and for communication between component devices and UE. For example, device-to-device (D2D) communication technologies such as Wi-Fi, Wi-Fi Direct, D2D communication, 5G sidelink, Bluetooth, tethering, and other short-range communication technologies can be used. The communication technologies that can be used for communication between component devices and for communication between component devices and UE can be determined in many different forms, not limited to the examples above.

[0050] In this disclosure, "UE" refers to a device having network capabilities (such as fifth-generation (5G) modem capabilities) to transmit data to or receive data from another user's device via a network. For example, a UE may communicate with another UE via a server and may include a communication module or communication application for communicating with the server or another UE.

[0051] As a communication technology that can be used for communication between UEs, any one or more of various communication technologies can be used. For example, a UE can communicate with other UEs by using communication technologies such as LTE or 5G that conform to the 3GPP standards or such as Wi-Fi. The communication technologies that can be used for communication between UEs can be determined in many different forms, not limited to the examples above.

[0052] In this disclosure, device names such as "XR device," "component device," and "UE" are used to logically categorize user devices according to their functions. Therefore, a device can be referred to by one or more device names. For example, when a first device includes a display and is capable of displaying XR content to a first user, transmitting data to or receiving data from a second user's UE, and capturing objects via a built-in camera, the first device may be referred to as any of the following: XR device, capturing device (component device), and UE.

[0053] The methods for providing 3D XR media content disclosed herein include the following:

[0054] —End-to-end (E2E) flow and architecture for XR services. E2E flow and architecture for XR services can include the distribution of media processing for services across multiple devices and can consist of processing entities in the cloud (one or more cloud servers, edge clouds, mobile edge computing (MEC) servers, etc.).

[0055] — Functional component architecture for terminals or UEs used for XR services.

[0056] —Supports UE architectures that include configurations of multiple devices (hereinafter referred to as multi-device configurations). For example, multiple devices can be connected to each other (or share networks) via wired tethering, wireless tethering, or other wired / wireless networks. In addition, multi-device configurations can include independent wearable devices (e.g., HMDs, XR glasses, etc.).

[0057] —The process of establishing and setting up UE sessions for various XR services on various devices.

[0058] — Necessary information for use cases that enable XR dialogue services (e.g., device pose information, reference point, device function type (e.g., vision, capture, etc.)) and media attributes (e.g., object size, etc.).

[0059] —Definitions used to determine the UE characteristics and capabilities for session establishment.

[0060] — Determine cloud assistance based on services, UE capabilities, and requirements (according to the UE or service management entity).

[0061] However, the above description is merely for ease of understanding, and the embodiments presented in this disclosure will be described throughout the disclosure. Hereinafter, methods and apparatus according to this disclosure will be described with reference to the accompanying drawings.

[0062] Figure 1A This is a schematic diagram illustrating a two-dimensional (2D) video stream 101 and a 2D video call according to embodiments of the present disclosure.

[0063] Reference Figure 1A In 2D video stream 101, 2D video content can be directly transmitted to the UE. 2D video call 102 is a service in which, when a first user using a first UE makes or receives a video call with a second user using a second UE, multiple images (or a single image) of the first user and / or the second user are displayed as 2D video images on the 2D displays of both the first and second UEs. In 2D video call 102, on the displays of both the first and second UEs, the 2D video images of the first user of the first UE and the second user of the second UE are simply superimposed on each other, but these two video images may be unrelated to each other.

[0064] Figure 1B This is a schematic diagram illustrating VR content provision method 103 and AR content provision method 104 according to embodiments of the present disclosure.

[0065] Reference Figure 1B In VR content provision method 103, a viewport can be created based on the UE's viewpoint, and VR content can be generated based on the viewport. A viewport refers to a polygon representing a spatial region, and objects can be rendered within the viewport. In other words, in VR content provision method 103, only objects included in the region viewed by the UE from its viewpoint can be generated as VR content. The generated VR content can be displayed on the UE's monitor or transmitted to another UE.

[0066] In AR content provisioning method 104, visual information can be generated based on the posture of the UE, and AR content can be generated based on the visual information. The visual information is information about the UE's surrounding environment. In other words, in AR content provisioning method 104, visual information can be generated based on the posture of the first UE, including information about the position and orientation of the first UE relative to its surrounding environment, and an image of the first UE or an image of objects around the first UE can be provided to the second UE by taking into account the first UE's surrounding environment. For example, when the first UE is lying on a sofa, the second UE can display an image of the first UE lying on the sofa or any object. In other words, in AR content provisioning method 104, the location (the location of the augmented object) of objects existing in the first UE's surrounding environment to be displayed on the second UE's display can be determined based on the visual information.

[0067] Compared to the use of 2D media content, this disclosure provides a method for enabling 360-degree video and 3D media-related services for XR applications. The 360-degree video and 3D media-related services disclosed herein may include XR dialogue services. XR dialogue services are services in which pre-generated or real-time AR objects, VR objects, etc., are provided to users in real-time during a real-time dialogue (e.g., a call) between users using an XR device. For XR dialogue services, additional pre-processing / post-processing may be required in the end-to-end (E2E) stream to support VR or AR applications. Furthermore, proper rendering and display of objects may require additional information related to the configuration and settings of the devices that may be used in the use cases of XR dialogue services.

[0068] For example, the requirements for metadata, as well as additional preprocessing and postprocessing, can be determined based on the following factors.

[0069] —Applications and use cases (from the user's perspective)

[0070] —Combinations of various devices and form factors that may be used in the use case (from both the user's and system's perspectives).

[0071] —Media coordination may be required due to E2E constraints or bottlenecks in the service chain (e.g., constraints on network bandwidth or device processing / functionality).

[0072] To support real-time services enabled by 360-degree video and 3D media (e.g., services such as XR conversational services, where content is captured and processed in real time and delivered to a network processing entity or another user), metadata (e.g., pose, camera type, etc.) may be needed for processing and displaying 360-degree video and 3D media. Furthermore, it may be necessary to optimize processing operations related to metadata across the entire E2E stream to meet the requirements of devices, services, and networks.

[0073] According to embodiments of this disclosure, a UE architecture for XR services is defined. This UE architecture enables various services (e.g., AR conversational services) with diverse requirements (media type, media quality, latency, etc.) across different device configurations (or settings) and capabilities (capture, vision, rendering, display, and processing capabilities, as well as processing speed / power) in various network capabilities and environments (e.g., variable network bandwidth, MEC / cloud processing capabilities, etc.).

[0074] Figure 2A This is a schematic diagram illustrating a method performed by a first UE to provide or receive 3D XR media content from a second UE, according to an embodiment of the present disclosure.

[0075] Reference Figure 2A The information related to the UE includes visual information indicating the UE's surrounding environment and information about XR objects included in the UE's surrounding environment. In order for the second UE to provide its user with a first XR object included in the first UE's surrounding environment as 3D XR media content, it may require information about the first XR object and first visual information about the first UE's surrounding environment. The first visual information can be used by the second UE to determine where to display the first XR object.

[0076] In the 3D XR content provision method 201 according to an embodiment of this disclosure, a first UE can provide a first spatial set information of the first UE to a second UE, enabling the second UE to obtain a first XR object and first visual information, and display the first XR object thereon. The first spatial set information is information about the space surrounding the first UE, and the second UE can use the first spatial set information to identify the surrounding environment of the first UE and display the first XR object. Furthermore, by considering the surrounding environments of both the first and second UEs, the second UE can further use the second spatial set information of the second UE to display the first XR object. Similarly, the first UE can receive second spatial set information from the second UE and display the second XR object based on the second visual information. Optionally, some or all of the visual information may not be shared between the UEs, and each UE can display the XR object based on the received spatial set information. Reference will be made below. Figure 6 Describe the spatial set information in detail.

[0077] Figure 2B This is a schematic diagram illustrating a method performed by a first user 211 to share XR media content with a second user 221, according to an embodiment of this disclosure. (Refer to...) Figure 2B The embodiments described in this disclosure are merely examples, and the methods for providing 3D XR media content according to this disclosure are not limited to... Figure 2B The embodiments of this disclosure are shown. Figure 2BAn example of using XR glasses as an XR device is shown. XR glasses can be transparent, semi-transparent, or opaque. Except when the XR glasses are opaque, the user of the XR glasses can directly see objects that actually exist in the user's field of view (FOV) through the lenses, and additionally see 3D media objects displayed by the XR glasses.

[0078] Reference Figure 2B , Figure 2B Figure 202 illustrates a scenario where an XR call (or AR call) is performed between a first user 211 and a second user 221. The XR call can be initiated by a call request and call response between a first UE 214 and a second UE 222. The first user 211 can see a 3D video object 215 and a shared object 216 representing the second user 221 via a first XR glasses 212, while the second user 221 can see a 3D video object 225 and a shared object 226 representing the first user 211 via a second XR glasses (or a second UE) 222.

[0079] The first XR glasses 212, the first camera 213, and the first UE 214 can exist around the first user 211 as devices for XR services. The first XR glasses 212 can render XR objects to be displayed on its display. Furthermore, the first XR glasses 212 can include a visual camera and can capture the surrounding environment 210 of the first user 211 using the visual camera. The first camera 213 can capture images of the first user 211 in real time and can be used to transmit real-time 3D images of the first user 211 to the second user 221. The first UE 214 can control XR calls with the second UE 222, receive and process data from the second UE 222 for transmission to the first XR glasses 212, and receive and process images captured from the first XR glasses 212 and the first camera 213 for transmission to the second UE 222. Similarly, the second XR glasses (or the second UE) 222 and the second camera 223 can exist around the second user 221 as devices for XR services within the second user environment 220. The difference between this configuration and the configuration of the devices surrounding the first user 211 is that the second XR glasses also function as a second UE 222 capable of transmitting data to and receiving data from the first UE 214 and managing and processing various data slices.

[0080] Shared object 216 or 226 can be an object that actually exists around the first user 211 or the second user 221, or an object that is virtually created or shared by the first user 211 or the second user 221. Furthermore, the first user 211 or the second user 221 can manipulate (or interact with) shared object 216 or 226. For example, the second user 221 can move or rotate shared object 226 displayed on the second XR glasses (or the second UE 222), and correspondingly, shared object 216 can also be moved or rotated on the display of the first XR glasses 212.

[0081] In such Figure 2B In the case of performing an XR call as shown in 202 and 203, the following characteristics may exist. At least some of the following characteristics may differ from... Figure 1A Features of 2D video calls.

[0082] —One or more 3D media objects can be delivered (and / or shared) between two users. Objects can be captured in real time or pre-captured before an AR call.

[0083] Users can view 3D media objects via XR devices.

[0084] —When rendered on an XR device, 3D media objects can be realistically enhanced into the user's environment or background.

[0085] User interaction is possible, such as rotating and placing 3D media objects by the user within the user's environment.

[0086] — 3D media objects can be pre-made and shared between users in a call (e.g., like shared object 216), or captured and delivered in a live manner (e.g., like a 3D video object representing a first user 211 or a second user 221).

[0087] —The user's UE may include one or more hardware devices with different processing functions and capabilities, or may be connected to one or more hardware devices. For example, one or more hardware devices may include a capture camera, a vision camera, rendering XR glasses, a mobile device that performs specific processing and has 5G capabilities, etc.

[0088] —One or more hardware devices may be located in different locations in the user environment, and the location of the hardware devices may be static or dynamic.

[0089] The media processing required for the service can be distributed across other devices and entities within the E2E stream (such as cloud and MEC servers).

[0090] According to embodiments of this disclosure, in order to enhance and display 3D media objects in a user environment, XR glasses, a UE, and a camera can exchange information with each other. A second UE 222 can obtain information about a second user 221, information for generating a second spatial set, and information about objects (e.g., the second user 221) existing within the second spatial set via a second camera 223. A first UE 214 can receive information from the second UE 222 about the second spatial set surrounding the second UE 222 and information about the second user 221. The first UE 214 can transmit the information received from the second UE 222 to the first XR glasses 212 after processing or not processing the received information. The first XR glasses 212 can enhance and display 3D media objects and shared objects 216 for the second user 221 in the environment of the first user 211 based on the received information using a display. Information about the surrounding environment of the first user 211 obtained from the first camera 213 can also be used to enable the first XR glasses 212 to enhance and display 3D media objects. In the same manner, the second XR device can enhance and display the first user 211 and the shared object 226 in the environment of the second user 221.

[0091] Figure 2B Reference numeral 203 in the accompanying drawings represents the field of view of the first user 211 wearing the first XR glasses 212. A table as a shared object 216 and a 3D video object 215 representing the second user 221 can be displayed on the monitor of the first XR glasses 212. Furthermore, XR calls can be made between three or more users. For example, refer to... Figure 2B In the XR call 203, a total of five users, including the first user 211, participate, and the second user 221, the third user, and the fourth user are displayed on the first XR glasses 212. Because the table as a shared object 216 and the fifth user 217 are actually present around the first user 211, the table and the fifth user 217 can be directly visible to the first user 211 without being displayed on the transparent or semi-transparent display of the first XR glasses 212.

[0092] In the following text, reference will be made to Figures 3 to 11 The description based on this disclosure is for providing various 3D XR services (such as references) Figure 2B The method for providing 3D XR media content (described in the XR call). Figure 3 This is a diagram illustrating the media flow for providing XR services. Figure 4 Various device configurations for providing XR services are shown. Figure 5 The establishment of an XR service session for providing XR services is illustrated. Figure 6 The user space set and user space set parameters used to provide XR services are shown. Figure 7This shows the flow of media data and metadata in an XR service session. Figure 8 The XR media architecture for a UE used to provide XR services is shown. Figure 9 This is a flowchart of the XR service provision method. Figure 10 and Figure 11 The configuration of the device used to provide XR services is shown.

[0093] Figure 3 This is a schematic diagram illustrating an XR service flow according to an embodiment of the present disclosure, in which an object existing in the environment of a first user (hereinafter referred to as "first object") is provided to a second user as a 3D media object.

[0094] Reference Figure 3 The 3D media process 300 can provide a first object to a second user. The 3D media process 300 can be executed by a device of a first user including a first UE and a device of a second user including a second UE, and some of the 3D media processes 300 can be executed by a cloud or MEC server. For example, capture 310 can be executed by the first user's device, and at least some of 3D modeling 320, XR encoding 330, and XR formatting 340 can be executed by the first user's device or a cloud or MEC server. At least one of XR decoding 360 or XR rendering 370 can be executed by the second user's device or a cloud or MEC server. Therefore, XR service session management may be required to distribute service processing among entities, i.e., to determine in a distributed manner which entity will execute a specific process in the 3D media process 300. For example, based on at least one of the capabilities of a component device, the capabilities of a UE, or the capabilities of an XR device, the UE can determine to request a cloud or MEC server to perform processing for at least some of the 3D media processes 300 (i.e., cloud-assisted processing). Furthermore, the cloud or MEC server can receive device capability reports from the UE and, based on at least one of the component device capabilities, the UE capabilities, and the XR device capabilities, determine whether at least some of the 3D media processes 300 are performing cloud-assisted processing. Criteria used to evaluate the capabilities of a user device may include the device's storage capacity, the number of operations per second that can be processed, the number of clock cycles of the processing device, or information regarding whether dedicated hardware devices for a specific processing are included. Criteria used to evaluate the capabilities of a user device may be determined in a variety of different forms, not limited to the examples above.

[0095] The following is an example of detailed operations in the 3D Media Process 300.

[0096] —Capture (or capture) 310: The act of capturing content (e.g., scenes, objects, combinations of both, etc., depending on the service application) in real time via one or more cameras. One or more cameras may include not only RGB cameras (outputting 2D video) but also cameras capable of capturing depth attributes and other attributes such as reflectivity that can be used to capture data needed for 3D modeling (e.g., depth maps). The attributes that can be captured by a camera are not limited to the examples above and can include a variety of other attributes. Furthermore, in addition to the captured data, processing of the data captured by the camera may require other data available during capture (e.g., intrinsic and extrinsic parameters of the camera).

[0097] —3D Modeling 320: The data output in Capture 310 can be used to perform 3D modeling to generate and output 3D model data bitstream content. 3D model data bitstreams, such as polygon file format (PLY) data, can represent 3D media data as point clouds or meshes. For example, the data output in Capture 310 can be processed into layer data as follows.

[0098] Multiple RGB + depth -> a single PLY representing one object.

[0099] Multiple RGB + depth -> multiple PLYs (multiple object sub-parts) -> a single PLY representing one object.

[0100] —XR Encoding 330: The output from 3D modeling 320 can be encoded to compress large amounts of raw data. Various encoding techniques can be used to perform point cloud encoding or mesh encoding, such as video-based point cloud compression (V-PCC) based on Moving Pictures Expert Group (MPEG) video, Google Draco, etc. Encoding can be lossy or lossless. To support decoding of the compressed data, a decoder corresponding to the encoder used in XR Encoding 330 may be necessary.

[0101] —XR Formatting (or Encapsulation) 340: For data transmission utilizing networks such as 5G, the compressed data output during the XR encoding process 330 may require formatting and / or encapsulation. For example, the MPEG International Organization for Standardization Basic File Format (ISOBMFF) for file encapsulation, the MPEG Media Transport Protocol (MMTP) payload format, and the Real-Time Transport Protocol (RTP) payload format for preparing data before delivery can be used as formatting technologies.

[0102] —Delivery 350: By using Hypertext Transfer Protocol (HTTP), RTP, HTTP-based dynamic adaptive streaming over HTTP (DASH), MPEG media transport (MMT), or other delivery mechanisms, compressed and formatted media can be delivered to the second UE using 5G networks.

[0103] —XR Decoding 360: Compressed data can be received by the XR decoding entity, and the XR decoding entity can decapsulate and decode the compressed bitstream to recover the uncompressed PLY bitstream.

[0104] —XR Rendering 370: After XR Decoding 360, the 3D data bitstream can be transmitted to the renderer. The renderer can render a 2D viewport of the 3D data based on the intent of the first user or the first UE received from the first UE, or the pose information of the second user using the second UE (e.g., user offset position, pose, orientation, view frustum, and viewport). The intent of the first user or the first UE can be delivered to the second UE, for example, through some metadata.

[0105] Figure 3 The 3D media process 300 shown is merely an example of an XR 3D service stream, and 3D media can be provided through a media process that is slightly different from 3D media process 300. Furthermore, the first and second UEs may each include one or more component devices, or may be connected to one or more component devices. For example, one or more component devices may be connected to or network-shared with the first or second UE using Bluetooth, Wi-Fi Direct, 5G bypass, or other communication technologies.

[0106] Figure 4 This is a schematic diagram illustrating various device configurations that can be used to provide 3D XR media, according to embodiments of the present disclosure.

[0107] Before describing the device configuration, the syntax for each device according to embodiments of this disclosure will be described. Each device can be specified using the following syntax or identifiers.

[0108] —UE ID, device description, and device functionality type description (UEx: devicedescription: devicefunctionalitytypedescription)

[0109] Here, when each device has the network capability to transmit data to and receive data from another user's device (hereinafter referred to as data network capability), the corresponding device has a unique UE ID; otherwise, the syntax for the corresponding device may include the UE ID of another device that has data network capability in a device connected to that device in a wired or wireless manner. For example, in Figure 4 In the first device configuration 401, since only the mobile phone has data network capabilities, the UE ID of all devices around the user is "UE1", which serves as the UE ID of the mobile phone. On the other hand, in the third device configuration 403, since the independent XR glasses, the third camera, and the mobile phone each have data network capabilities, they can each have "UE1", "UE2", and "UE3" as their UE IDs, respectively. According to embodiments of this disclosure, when the UE is able to access the 5G system, at least one of the following can be used as the UE ID: subscription permanent identifier (SUPI), permanent equipment identifier (PEI), and 5G global unique temporary identifier (5G-GUTI). Furthermore, the correlation between the UE ID and each of the SUPI, PEI, and 5G-GUTI generated using a separate algorithm can be determined according to a preset algorithm, and this correlation can be provided to the UE. This correlation can be determined, for example, by the UE or a server, and can be provided to the UE or the server.

[0110] According to embodiments of this disclosure, the “device function type description” corresponding to the device can be categorized based on roles in the device configuration and can be defined as follows.

[0111] —Rendering: A device corresponding to the rendering function type can render XR objects on a display. The rendering device can render XR objects using metadata / necessary information for XR rendering-related functional processing. Rendering-related functional processing can include, for example, 2D / 3D media decoding, post-processing, rendering, and rendering to a 2D / 3D display. Necessary information for XR rendering can include not only media data but also the pose information of the rendering device itself.

[0112] —Vision: Devices corresponding to vision function types can acquire and provide information about the user's surrounding environment (i.e., visual information) to enable accurate rendering of 2D or 3D media for XR services. For example, a vision device can use an RGB camera or other cameras to acquire basic input data for computer vision processing, such as simultaneous localization and mapping (SLAM), thereby identifying and analyzing the user's surroundings. To realistically overlay an XR environment onto the user's environment, accurate analysis of the user's surroundings and 3D media objects may be required. Use cases for realistic overlay representations could include, for example, placing a 3D media point cloud of a dog (a 3D media object) on a floor surface (the user's surroundings) or on a sofa in the user's living room (the user's surroundings).

[0113] —Capture: The device corresponding to the capture function type can acquire and provide basic input data for capturing 3D objects in the user environment (such as 3D models of the user's head, body or other objects).

[0114] Figure 4 Three device configurations are shown according to embodiments of the present disclosure, which are capable of providing 3D media content to a user via an XR device. Figure 4 The relative positions of the various devices shown with respect to each other can be static or dynamic.

[0115] Reference Figure 4The first device configuration 401 consists of XR glasses shared with a mobile phone via a network, a first camera included in the XR glasses, and three external cameras. The vision camera UE1:camera1:vision (UE1:Camera 1:vision) can be located on or inside the XR glasses (rendering device). The capture cameras UE1:camera3:capturing (UE1:Camera 3:capturing), UE1:camera4:capturing (UE1:Camera 4:capturing), and UE1:camera5:capturing (UE1:Camera 5:capturing) can be located outside the XR glasses to capture objects around the user. The mobile phone, with data network capability, can connect to the XR glasses via a wired connection or according to a wireless communication method (e.g., Bluetooth, network sharing, etc.). Furthermore, the mobile phone can render XR objects on its display (and thus can be identified as "UE1:phone:rendering (UE1:phone:rendering)") and capture objects around the mobile phone via its camera (and thus can be identified as "UE1:camera2:capturing (UE1:camera 2:capturing)").

[0116] The second device configuration 402 consists of standalone XR glasses, a first camera included in the standalone XR glasses, a dockable second camera, and an external camera. Unlike the first device configuration 401, standalone XR glasses are so named because they have data network capabilities to transmit and receive data from another user's device without the need for a separate mobile phone. The dockable second camera is a camera that can be detached from the standalone XR glasses. In other words, the first camera can be attached to or included as a component of the standalone XR glasses, while the second camera can be attached to or detached from the standalone XR glasses for placement in different locations. The first camera can perform visual functions, and the second camera can perform both visual and capture functions.

[0117] The third device configuration 403 consists of a standalone XR glasses, two external cameras, and a mobile phone. The third device configuration 403 comprises multiple devices (standalone XR glasses, a third camera, and a mobile phone) with data network capabilities. Therefore, each device with data network capabilities can transmit XR service-related data to a target destination (e.g., another user's device or a server) via other devices, or directly to the target destination without passing through other devices.

[0118] also, Figure 4 Only three examples of various device configurations are shown, and this disclosure is not limited to... Figure 4Examples are provided, and various other device configurations can be included. For instance, the presence or absence of data network capability for each device can be determined in various ways. Even when the first device has data network capability, it can still transmit data to a second device that also has data network capability, and the second device can process the received data and transmit it to a server or another device. Furthermore, the number of devices that can be included in the device configuration is not limited to... Figure 4 Examples are given and defined in many different forms. The types of functions (capture, vision, and rendering) that each device possesses can also be defined in many different forms.

[0119] Figure 5 This is a schematic diagram illustrating the process of establishing an XR service session and providing XR services according to embodiments of the present disclosure.

[0120] Reference Figure 5 UE 51 can communicate with one or more component devices 52 belonging to the UE user environment and XR service provider 53. An XR service session can be established based on the communication between the UE, one or more component devices, and the XR service provider. A user of the UE can use the established XR service session to transmit XR 3D media content to or receive XR 3D media content from another UE user in real time. The XR service provider may include at least one server and transmit XR service-related data or metadata to the UE. For example, the XR service provider may include a cloud, MEC server, etc.

[0121] In operation 501, each component device may transmit its device description to the UE as an initial capability report. One or more component devices may include, for example, AR glasses, a camera, etc. An initial capability report may be transmitted to the UE when the corresponding component device is initially installed / connected to it.

[0122] In Operation 502, the UE may request an XR service provider to transmit information associated with an XR service list. For example, when a user of the UE requests or receives an XR call request from another user via the UE, a request for an XR service list may be initiated. The UE may assume that the XR services can provide one or more representations of an object or scene based on device capabilities or network capabilities.

[0123] In Operation 503, the XR service provider may provide the UE with a list of XR services in response. The XR service list may include capability requirements for each XR service. The XR services that can be included in the XR service list can be determined in various different forms. For example, XR services may include XR conferencing, AR conferencing, video calling, etc. Furthermore, XR services may include multiple services with different capability requirements for the same type of service (e.g., XR calling) (e.g., high-capacity XR calling service and low-capacity XR calling service). Additionally, for a given XR service, the XR service provider may perform network media processing to support UEs with insufficient processing capabilities. For example, the XR service provider may perform processing such as encoding or decoding XR media data on behalf of the UE and transmit the resulting data to the UE. The XR service list may also include information about whether network-assisted media processing is available or required for each XR service.

[0124] In Operation 504, the UE can request a device status report from each component device.

[0125] In operation 505, the corresponding component device can transmit a device status report to the UE. For example, the device status report may include the following device status information or device capability information:

[0126] —The physical location and orientation of the device (e.g., camera pose).

[0127] —The device's hardware capabilities (e.g., for cameras, RGB resolution, depth resolution, and FOV; for XR glasses, encoder and decoder functions, 3D modeling capabilities, display resolution, display FOV, etc.)

[0128] In operation 506, based on the initial capability report received in operation 501, the XR service list received in operation 503, and the device status report received in operation 505, the UE can select at least one XR service from the XR service list. In operation 505, the UE can collect device status reports received from one or more component devices and select an XR service from the XR service list that has capability requirements matching the status or capability of each component device.

[0129] In operation 507, based on the initial capability report received in operation 501 and the device status report received in operation 505, the UE can determine the capability information and status information of the corresponding component device related to the selected XR service, and transmit the determined component device capability information and status information as a device capability report to the XR service provider. The device capability report may include camera information, the processing performance of the component device, the location and orientation information of the component device, etc. According to embodiments of this disclosure, the device capability report may include user space set parameters. The UE can determine the user space set parameters based on the initial capability report received in operation 501 and the device status report received in operation 505. The following will refer to... Figure 6 The syntax and semantics for the user space set parameters are described in detail. Furthermore, based on at least one of the capabilities of the corresponding component device, the capabilities of the UE, and the capabilities of the XR device, the UE can determine whether to request the XR service provider to perform processing (i.e., network-assisted processing) on ​​at least some of the 3D media processes associated with the selected XR service. According to embodiments of this disclosure, the device capability report may include information for requesting network-assisted processing for at least some 3D media processes associated with the selected XR service.

[0130] In operation 508, the XR service provider may provide the UE with device configuration information and a service entry point (e.g., a list in the form of HTTP-based Dynamic Adaptive Streaming (DASH) media presentation descriptions (MPDs). The device configuration information may include operational configuration information of the component devices associated with the selected XR service (e.g., display resolution, uplink media profile, necessary metadata, etc.). The service entry point may include identification information (e.g., address) of the data network accessible to the UE for receiving the selected XR service. Furthermore, based on the UE's request or the user device's capabilities (component device capabilities, UE capabilities, or XR device capabilities) included in the device capability report received in operation 507, the XR service provider may determine to execute at least some 3D media processes associated with the selected XR service. According to embodiments of this disclosure, in operation 508, the XR service provider may transmit information to the UE about which 3D media processes the network will support.

[0131] In operation 509, the UE can transmit device configuration information to each component device.

[0132] Each component device can transmit a configuration acknowledgment (ACK) response to the UE (operation 510). The configuration ACK response may include details indicating that the corresponding component device has been configured or is able to configure itself based on the received device configuration information. In addition, the component device can transmit media data and metadata required for the session of the selected XR service to the UE.

[0133] In operation 511, the UE can establish an XR service session by accessing the service entry point received in operation 508. When the XR service session is established, in operation 512, the UE and the XR service provider can continue uplink / downlink streaming of media data and metadata.

[0134] According to embodiments of this disclosure, operations 501 to 505 can be performed only when the UE initially connects to each component device. In other words, after the initial connection, the UE establishes a first XR service session to provide the user with a first XR service, and when a second XR service session needs to be established after the first XR service session ends, operations 501 to 505 can be skipped. Furthermore, the device description initially reported in operation 501 can be changed to be reported in operation 505.

[0135] Not only due to the importance of the physical environment itself, but also because of the diversity of user device configurations that depend on the user's physical environment, a high-quality XR experience may require device capability information and metadata related to the user environment of the component device. This disclosure defines the device capability information and metadata related to the user environment of the component device required in an XR service session. Entities participating in an XR service session can use the device capability information and metadata to provide XR services to users.

[0136] Furthermore, this disclosure proposes a "user space set" for considering the user environment when defining device capability information and metadata. The user space set can be a set of information including at least one of the following: information about the location and orientation of various devices located around the user and used to provide XR services; device capability information; or information about the physical environment surrounding the user. Various devices located around the user can be used together with the physical environment surrounding the user to define the user space set. Each user can have a user space set. In other words, a user space set corresponding to each user can exist.

[0137] Figure 6 This is a schematic diagram illustrating a user space set according to embodiments of the present disclosure.

[0138] According to embodiments of this disclosure, a user space set may include various parameters (hereinafter referred to as "user space set parameters") indicating the user's surrounding environment. The user space set may include information about the space and information about various devices located around the user and used to provide XR services. At least some devices (i.e., UEs) participating in an XR service session may obtain or process information required to provide XR services based on the various parameters included in the user space set. For example, a UE may receive captured data or visual data from a nearby camera and process the received captured data or visual data based on the user space set. The processed data may be transmitted along with the user space set to a server or another UE and may be used to provide other users with 3D media data about the user's surrounding environment.

[0139] Reference Figure 6 A user space set can be represented using a right-handed Cartesian coordinate system, where the origin is defined as the reference point and the x-axis direction is defined as the reference orientation of the space set. However, this is merely an example, and user space sets can be represented according to various other representation techniques used to represent 3D space. In this disclosure, for convenience, a right-handed Cartesian coordinate system will be used as an example of a representation technique to describe user space sets. Figure 6 For example, in the user space set there may be XR glasses 601, a first capture camera 602 and a second capture camera 603, which perform functions as a UE, visual functions and rendering functions.

[0140] Furthermore, the user space set may include one or more subspace sets. According to one embodiment of this disclosure, a visual subspace set defines a space in which 3D media is rendered and realistically enhanced, allowing the user to experience the 3D media as a realistic part of a scene / background existing within the visual subspace. One or more visual subspace sets may exist within a single user space set. Visual subspace sets may be implemented using one or more visual cameras, whose fields of view (FOV) may overlap or not. Additionally, there may be a visual subspace set corresponding to each component device performing a visual function.

[0141] According to embodiments of this disclosure, a capture subspace set defines a space in which a real 3D object can be captured stereoscopically by one or more capture cameras. When only a portion of a real 3D object exists within the capture subspace set, only that portion of the real 3D object can be captured. One or more capture subspace sets can exist within a single user space set. A capture subspace set can be implemented using one or more capture cameras, whose fields of view (FOV) may overlap or not. Furthermore, there can be a capture subspace set corresponding to each component device performing the capture function.

[0142] Furthermore, although for convenience, the user space set and subspace set are described below as being in the shape of cuboids, their shapes are not limited to cuboids and can be determined in many different forms. In addition, the user space set or subspace set can be static or dynamically changing. For example, the shape, size, and configuration of the user space set or subspace set may be modified due to various reasons (e.g., repositioning of the user or device), and the position of the user space set or subspace set may change.

[0143] According to embodiments of this disclosure, a subspace set can be provided for each device or each functional type of the device. For example, a visual subspace set 610 can exist for XR glasses 601, a first capture subspace set 620 can exist for the first capture camera 602, and a second capture subspace set 630 can exist for the second capture camera 603. Additionally, XR glasses 601 can also perform a capture function, and in this case, a separate capture subspace set can also be provided for XR glasses 601. Position and orientation information in user space set 600 can be determined relative to reference point 605 of user space set 600. Furthermore, position and orientation information in subspace sets can be determined relative to reference points in subspace sets, and reference points in subspace sets can be determined relative to reference point 605 of user space set 600. For example, reference point 611 in visual subspace set 610, reference point 621 in first capture subspace set 620, and reference point 631 in second capture subspace set 630 can be determined relative to reference point 605 in user space set 600.

[0144] According to embodiments of this disclosure, the information required to provide XR services may include at least one of the following:

[0145] —Number of devices concentrated in user space

[0146] —Description of each device

[0147] —Device function type for each device. The device function type can be used as the basis for employing the pose information of each device. The device function type for each device may include at least one of rendering, vision, and capture.

[0148] —The position and orientation of each device (e.g., the position and orientation in front of the camera). In other words, the pose of each device relative to a reference point defined in the user space set.

[0149] —Reference point. A reference point can be one of the following:

[0150] o Based on real-world coordinates, such as Global Positioning System (GPS) coordinates.

[0151] o is bound to a reference point of a device in the user space set.

[0152] —Indication of whether the position and / or orientation of the device is static or dynamic;

[0153] —Regarding camera devices:

[0154] o FOV / perspective

[0155] Minimum and maximum values ​​of the depth sensitivity of the o sensor

[0156] o intrinsic parameters

[0157] external parameters

[0158] According to embodiments of this disclosure, user space set parameters may include the necessary information for the XR service as described above, and can be represented using the following syntax. First, user space set parameters can be categorized into a first parameter group and a second parameter group. According to embodiments of this disclosure, the second parameter group may or may not be obtained based on the first parameter group.

[0159] According to embodiments of this disclosure, the syntax for representing a first parameter group is as follows. The first parameter group may include all or some of the parameters described below. In other words, some parameters may be omitted. Furthermore, the following syntax for the first parameter group is merely an example; parameters with the same or similar semantics to those in the first parameter group may also be represented using different syntax.

[0160]

[0161]

[0162] According to embodiments of this disclosure, the semantics of each parameter represented by the above syntax are as follows.

[0163] ——num_components: It specifies the number of components (component devices) in the user space set.

[0164] —initial_position_x, initial_position_y, initial_position_z: These specify the x, y, and z coordinate values, which correspond to the coordinates of the component device's initial position relative to the user space set reference point. The units representing the coordinate values ​​can be, for example, centimeters or millimeters, but are not limited to these, and can be determined in many different forms. When the component device is used as a reference device in the user space set (when the component device's device_id matches the device_id value specified as space_set_reference_type=1 in the SpaceSetReference structure), all three coordinate values ​​are set to 0.

[0165] —initial_orientation_x, initial_orientation_y, initial_orientation_z, initial_orientation_w: These specify the x, y, z, and w elements of an orientation quaternion (or Hamiltonian) that indicates the initial orientation of the component assembly. w is the real part of the quaternion, and x, y, and z are the imaginary parts. When the component assembly is used as a reference device in a user space set, the values ​​of these parameters define a unit quaternion with zero rotation in the orientation of the component assembly. In this case, initial_orientation_x can indicate the x-axis direction of the space set coordinate system (e.g., a right-handed Cartesian coordinate system), and initial_orientation_y can indicate the vertically upward y-axis direction.

[0166] —space_setreference_type: This specifies how the reference point, i.e., the origin (0, 0, 0), and reference orientation are defined in the user space set. All other attitude information used for the first parameter set can be defined using the reference point as the origin. The reference orientation can define the x-axis direction of the space set coordinate system (e.g., a right-handed Cartesian coordinate system). When the value of space_set_reference_type is 0, the reference point (ground plane) and reference orientation can be defined as real-world GPS coordinates and real-world orientation, respectively. For component devices with the value of space_setreference_type set to 1, the component device's attitude (position coordinates and orientation) can be used as the reference point and reference orientation for the user space set.

[0167] — gps_latitude, gps_longitude: These specify the latitude and longitude lines for the GPS coordinates of the reference point (origin) of the user's spatial coordinate system in decimal degrees (DD).

[0168] —world_orientation: This specifies the world compass orientation in degrees, defined as the reference orientation of the spatial coordinate system (e.g., the x-axis of a right-handed Cartesian coordinate system) (0 degrees corresponds to true north in the real world). The coordinate system can be a right-handed Cartesian coordinate system with its y-axis perpendicular to the x-axis and pointing upwards. The default direction (x-axis) can be true north.

[0169] —device_id: It specifies a unique identifier for the component device.

[0170] —dynamic_reference: A flag that specifies whether the reference point in the user space set is static (when the flag value is 0) or dynamic (when the flag value is 1).

[0171] —device_description: This specifies the description of the component device. The description of the component device can be specified as 1) a description from a predefined list (e.g., "0 = glasses, 1 = mobile phone, 2 = camera") or 2) a description string entry.

[0172] —dynamic_pose: A flag that specifies whether the pose of the component device is static (when the flag value is 0) or dynamic (when the flag value is 1).

[0173] —num_functionalities: This specifies the number of functions (defined by pose_functionality_type) that use the component device and its pose information. A component device identified by a device_id can include one or more functions. In other words, a component device can include only one function, both capture and vision functions, both capture and playback functions, both vision and playback functions, or all of the capture, vision, and playback functions.

[0174] —pose_functionality_type: This specifies the pose functionality type of the component. A value of 0 indicates a pose functionality used for rendering, a value of 1 indicates a pose functionality used for vision, and a value of 2 indicates a pose functionality used for capturing.

[0175] —hor_field_of_view, ver_field_of_view: These specify the horizontal and vertical FOV (field of view) capture or viewing capabilities of a component device (such as a camera), respectively. The unit of FOV can be, for example, radians.

[0176] —minimum_depth, maximum_depth: These specify the minimum and maximum depth capture or vision capabilities of a component device (e.g., a camera) for the specified function. The unit of depth can be, for example, millimeters.

[0177] —IntrinsicParameters() and ExtrinsicParameters(): These specify the internal parameter list and external parameter list for each component device (camera), respectively. For example, internal parameters are parameters for the camera device itself and can include focal length, principal point, tilt coefficient, etc., while external parameters are parameters used to describe the transformation relationship between the camera coordinate system and the real-world coordinate system and can include rotation or translation parameters between the two coordinate systems.

[0178] Next, according to embodiments of this disclosure, the syntax for representing the second parameter group is as follows. The second parameter group may include all or some of the parameters described below. In other words, some parameters may be omitted. Furthermore, the syntax for the second parameter group below is merely an example, and parameters having the same or similar semantics as the parameters in the second parameter group may also be represented according to different syntaxes.

[0179]

[0180]

[0181] According to embodiments of this disclosure, the semantics of each parameter represented by the above syntax are as follows:

[0182] — `space setsize_cubible_dx`, `space setsize_cubible_dy`, `space setsize_cubible_dz`: These specify the size of a user space set in the form of a cuboid along the x, y, and z axes of a Cartesian coordinate system. For example, when `space_set_reference_type = 0`, the reference point in the user space set can be the center of the cuboid; when `space_set_reference_type = 1`, it can be the center of the cuboid's base. However, this is just an example; the location of the reference point can be determined in many different ways.

[0183] — `visionspacesize_cubible_dx`, `visionspacesize_cubible_dy`, `visionspacesize_cubible_dz`: These specify the dimensions of a set of visual subspaces in cuboid form along the x, y, and z axes of a Cartesian coordinate system. The dimensions along the x, y, and z axes are specified relative to a reference point within the visual subspace set. The reference point within the visual subspace set can be defined by `SubSpaceReferencePointStruct()` included in the visual subspace structure. For example, the reference point (or anchor point) within the visual subspace set can be determined by the edge of the cuboid representing the visual subspace set that is closest to the reference point in the user space set. Anchor points within the visual subspace set can be determined in various different forms, not limited to the examples above.

[0184] ——capturesubspacesize_cubible_dx, capturesubspacesize_cubible_dy, capturesubspacesize_cubible_dz:

[0185] These specify the dimensions of a cuboid-shaped set of snapping subspaces in the x, y, and z axes of a Cartesian coordinate system. The dimensions in the x, y, and z axes are specified relative to a reference point within the snapping subspace set. The reference point within the snapping subspace set can be defined by the `SubSpaceReferencePointStruct()` method included in the snapping subspace structure. For example, the reference point (or anchor point) within the snapping subspace set can be determined by the edge of the cuboid representing the snapping subspace that is closest to the reference point in the user space set. The anchor point within the snapping subspace set can be determined in many different forms, not limited to the examples above.

[0186] Next, according to embodiments of this disclosure, a third set of parameters representing 3D media objects captured by a capture camera in a user space set is described. The third set of parameters can be determined based on at least one of the first or second set of parameters in the user space set.

[0187] According to embodiments of this disclosure, the syntax for representing a third parameter group is as follows. The third parameter group may include all or some of the parameters described below. In other words, some parameters may be omitted. Furthermore, the syntax for the third parameter group below is merely an example, and parameters having the same or similar semantics as parameters in the third parameter group may also be represented according to different syntaxes.

[0188]

[0189] According to embodiments of this disclosure, the semantics of each parameter represented by the above syntax are as follows:

[0190] —real_size_dx, real_size_dy, real_size_dz: These specify the actual size of the 3D media in the x, y, and z directions, respectively, corresponding to the encoded cuboid used to represent the 3D media data (e.g., a 10-bit bounding box for V-PCC compressed point clouds). The unit of measurement can be, for example, millimeters.

[0191] —object_default_orientation_x, object_default_orientation_y, object_default_orientation_z, object_default_orientation_w: These specify elements of an orientation quaternion representing the default rendering orientation of a 3D media object relative to the encoded cuboid used to represent 3D media data (e.g., a 10-bit bounding box used for V-PCC compressed point clouds). For V-PCC encoded data, the default rendering orientation can match the pi_front[d] supplemental enhancement information (SEI) message (in the V-PCC specification, pi_front[d] indicates the d-axis value of a unit vector representing the forward direction of the reconstructed point cloud sequence in units of 2–16). When pi_front[d] is not present, it can be assumed that the default rendering orientation represents a unit vector (0.0, 1.0, 0.0).

[0192] — min_rendering_distance specifies the minimum distance between the user's display and the 3D media object at which the 3D media object can be rendered and presented to the user. The distance unit can be, for example, centimeters or millimeters.

[0193] — `max_rendering_distance` specifies the maximum distance between the user's display and the 3D media object at which the 3D media object can be rendered and presented to the user. The distance unit can be, for example, centimeters or millimeters.

[0194] —default_rendering_distance specifies the default rendering distance between the user's display and the 3D media object, at which the 3D media object is rendered and presented to the user during initial playback. For example, the distance unit can be centimeters or millimeters.

[0195] Reference Figure 6The first parameter group, second parameter group, or third parameter group described are shared among the first UE, the server, and / or the second UE as user space set parameters or subspace set parameters, enabling the first UE / second UE or the server to understand the space around the second UE / first UE and process objects around the second UE / first UE to control their display as 3D XR media objects on the XR device.

[0196] Figure 7 This is a schematic diagram illustrating a stream of media data and metadata according to embodiments of the present disclosure.

[0197] Figure 7 This illustrates the flow of media data and metadata, the flow of user space set parameters, and the flow of additional media metadata (such as object size, default orientation, or some or all of the user space set parameters) used for XR services, between entities participating in an XR service session. (See reference...) Figure 7 For convenience, cloud, server, MEC server, etc., are collectively referred to as cloud. According to embodiments of this disclosure, it is possible to... Figure 7 The metadata stream shown is used to select all or some of the parameters included in the first, second, or third parameter group and transmit them between UEs.

[0198] According to embodiments of this disclosure, the following syntax can be used to describe the flow of media data and metadata.

[0199] —UE1: source device -> UE2: target device (UE1: source device -> UE2: target device)

[0200] Here, although each user can have one or more UEs (i.e., devices with network capabilities (e.g., 5G modem capabilities) to transmit data to and receive data from another user's device via the network), for the sake of convenience, it is assumed that each of the first user and the second user has one UE. Therefore, UE1 and UE2 refer to the first UE of the first user and the second UE of the second user, respectively. According to embodiments of this disclosure, each stream of media data and metadata described using the syntax is as follows.

[0201] (Operation 701) First user's posture information

[0202] -UE1: glasses->cloud [Purpose: Split rendering]

[0203] -UE1: glasses->UE2: phone [Purpose: capture and delivery of view-related parts) / rate adaptation]

[0204] -user_pose_parameter:SpaceSetReferenceStruct(pose_functionality_type=0)

[0205] Split rendering is a process that performs some rendering operations in the cloud.

[0206] (Operation 701a) Second user's attitude information

[0207] -UE2: glasses->UE1: phone [Purpose: capture and delivery of view-related parts) / rate adaptation]

[0208] -UE2: glasses->cloud [Purpose: Split rendering]

[0209] -user_pose_parameters:SpaceSetReferenceStruct(pose_functionality_type=0)

[0210] (Operation 702) 3D / 2D Media Data

[0211] -UE2:phone->UE1:phone[3D data]

[0212] -cloud->UE1: glasses[2D data, purpose: split rendering]

[0213] (Operation 702a) 2D Media Data

[0214] -UE1: camera->UE1: phone

[0215] -UE1: camera->cloud [Purpose: 3D modeling in the cloud]

[0216] (Operation 702b) 3D Media Data

[0217] -UE1:phone->UE2:phone

[0218] -cloud->UE2:phone

[0219] (Operation 703) Visual information of the first user

[0220] -UE1: phone -> UE2: phone / glasses [Purpose: To support rendering and rate adaptation in UE2]

[0221] -UE1: phone->cloud [Purpose: To support cloud-based 3D modeling and split rendering]

[0222] -vision_cam_parameters: SpaceSetReferenceStruct(pose_functionality_type=1)[unprocessed data or first parameter group]

[0223] -space_set_size: spaceset size struct() [vision - data to be processed]

[0224] -space_set_reference_point: SpaceSetReferenceStruct() [vision - data being processed]

[0225] -light_source_direction[vision-data being processed]

[0226] -augmentation_type[vision-data processed]

[0227] (Operation 703a) Visual information of the second user: It can be inferred from the visual information of the first user obtained in Operation 703 by replacing UE1 and UE2 with each other.

[0228] (Operation 704) 3D Modeling Parameters

[0229] -UE1: camera->UE1: phone [Transferring information between user devices]

[0230] -UE1: camera / phone -> cloud [Purpose: Cloud 3D modeling]

[0231] -capture_cam:SpaceSetReferenceStruct(pose_functionality_type=2)

[0232] -Intrinsic_param:Intrinsic parameters()

[0233] -Extrinsic_param:Extrinsic parameters()

[0234] (Operation 705) 3D Model Information

[0235] -UE1: phone -> UE2: phone / glasses [When performing 3D modeling in UE]

[0236] -Cloud->UE2: Phone / Glasses [When performing 3D modeling in the cloud]

[0237] -Object size, default orientation, default rendering size, priority (object size, default orientation, default rendering size, priority)

[0238] When a first user's movement within their spatial set needs to be correctly mapped to a second user's spatial set (by scaling or not scaling), the first user's pose information (or spatial set information), such as their first or second parameter set, can be directly transferred to the second user and used for rendering. Furthermore, when an object is shared between two users and visible to both, both users can accurately determine from which user's FOV (field of view, direction, and distance) they are viewing the shared object using knowledge of the other user's pose and visual information (spatial set, etc.). Sharing each user's pose information can be useful in real-time use cases, such as when two users need to view a shared object at the same distance and angle.

[0239] Figure 8 This is a schematic diagram illustrating an XR media architecture for a UE according to embodiments of the present disclosure.

[0240] Reference Figure 8 This illustrates an XR media architecture including an XR interaction controller 830 with a first UE 81. The first UE 81 of the first user can transmit and receive XR service-related information to and from a server (or second UE) 82 via one or more interfaces. The server 82 can be, for example, a cloud, an MEC server, a data network entry point, etc. The first UE 81 can transmit data to the server 82, or transmit data to the second UE directly or via the server 82.

[0241] Each entity in the XR media architecture of the first UE 81 can be a logical entity or a hardware entity. Logical entities can be implemented through various hardware configurations. The XR interaction controller 830 can process and control the pose information of component devices (rendering devices, visual cameras, capture cameras, etc.). The pose information of the component devices can be used by entities such as the XR media player 840 or the XR media generator 850. Furthermore, when pose information is required for part of the transmission or part of the rendering within the XR session service, at least a portion of the pose information (processed or unprocessed) can be transmitted to the server 82 or directly to the second UE. The function of each entity in the XR media architecture according to embodiments of this disclosure is as follows.

[0242] ——XR Sensing Application 810: It can control other XR entities in the XR media architecture.

[0243] —XR Media Session Processor 820: Through communication with the server (or second UE) 82, it can perform capability negotiation for XR service configuration, establish XR sessions, and control (manage) and support XR sessions. For example, capability negotiation may involve determining the quality level of XR services that can be supported by the user device's capabilities and XR service requirements, based on the user device's capabilities between the UE and the server, or determining which processes in the 3D media process related to the XR service are supported by the user device's capabilities, while the remaining processes are executed in the server.

[0244] —XR Interaction Controller 830: It can manage interaction-based services by communicating with a server (or a second UE) 82. The XR Interaction Controller 830 can provide relevant data to the XR Sensing Application 810 for interaction control, to the XR Media Session Processor 820 for interaction reporting control, to the XR Media Player 840 for vision-based interactive playback, and to the XR Media Generator 850 for media generation.

[0245] —XR Media Player 840: It can receive XR media content by communicating with the server (or second UE) 82. The XR Media Player 840 can provide relevant data to the XR Sensing Application 810 for media playback (media access, unpacking, depackaging, decoding, rendering, etc.), provide relevant data to the XR Media Session Processor 820 for media session control, and provide relevant data to the XR Interaction Controller 830 for session interaction.

[0246] —XR Media Generator 850: It can generate XR media content by communicating with a server (or a second UE) 82. The XR Media Generator 850 can provide relevant data to the XR Sensing Application 810 for media generation (capture, 3D modeling and preprocessing, encoding, encapsulation, packaging, etc.), provide relevant data to the XR Media Session Processor 820 for media session control, and provide relevant data to the XR Interaction Controller 830 for XR media generation interaction.

[0247] According to embodiments of this disclosure, interface parameters (including metadata that can be shared within the interface) that can be transmitted between entities in an XR media architecture are as follows. However, the following are merely examples of possible metadata. For example, XR media processing, XR media streaming, XR services, etc., according to embodiments of this disclosure, can be implemented via interface parameters transmitted between entities.

[0248] • Parameters of the first interface (801): Metadata required for data processing in server 82. Since server 82 can support both remote rendering and remote content creation, both metadata about the first user and metadata about the second user can be included.

[0249] -User pose information (purpose: remote rendering):

[0250] >NonDerivedParameters(pose_functionality_type=0);

[0251] -Vision camera information (used for remote rendering):

[0252] >NonDerivedParameters(pose_functionality_type=1);

[0253] -Capture camera information (used for cloud-based 3D modeling and coding):

[0254] >NonDerivedParameters(pose_functionality_type=2);

[0255] - User space set information (used for remote rendering / 3D modeling):

[0256] SpaceSetReferenceStruct()

[0257] SpaceSetSizeStruct()

[0258] >VisionSub Space Struct()

[0259] >CaptureSubSpaceStruct()

[0260] >SubSpaceReferencePointStruct()

[0261] -Media object capture information (Uplink: for when most processing is done on the device; Downlink: for when most rendering is done on the device)

[0262] >ObjectSizeStruct()

[0263] >DefaultOrientationStruct()

[0264] >DefaultRenderingParamStruct()

[0265] The parameters of the second interface (802) are: metadata transmitted between the XR interaction controller 830 and the XR media player 840. The metadata is typically metadata information related to a second user. However, the XR media player 840 may not necessarily need to reproduce the metadata related to the second user. According to embodiments of this disclosure, the XR media player 840 may typically not have spatial set-related processing capabilities and may have visual information processing capabilities. However, processing capabilities can be flexibly shared between the XR interaction controller 830 and the XR media player 840. In other words, information is shared between the XR interaction controller 830 and the XR media player 840, allowing data that cannot be processed individually to be processed collaboratively.

[0266] -User pose information (used for media playback):

[0267] >NonDerivedParameters(pose_functionality_type=0);

[0268] -Vision camera information (for media playback):

[0269] >NonDerivedParameters(pose_functionality_type=1);

[0270] -User space settings information (used for media playback):

[0271] SpaceSetReferenceStruct()

[0272] SpaceSetSizeStruct()

[0273] >VisionSub Space Struct()

[0274] >(CaptureSubSpaceStruct())

[0275] >SubSpaceReferencePointStruct()

[0276] -Media object capture information (Uplink: for when most processing is done on the user device rather than on the server; Downlink: for when most rendering is done on the user device rather than on the server)

[0277] >ObjectSizeStruct()

[0278] >DefaultOrientationStruct()

[0279] >DefaultRenderingParamStruct()

[0280] Parameters of the third interface (803): According to embodiments of this disclosure, the XR media generator 850 may not have powerful processing capabilities. Therefore, the XR media generator 850 can offload 3D media generation and encoding, etc. According to embodiments of this disclosure, metadata can be directly transmitted to the server (or the second UE) 82 via the fifth interface (805), or it can be transmitted to the server via the first interface (801) after passing through the XR interaction controller 830 via the third interface (803). The pose information and visual information of the second user input to the XR media generator 850 via the first interface (801) and the third interface (803) can be used to perform viewing-based partial capture, generation, delivery, or rendering of the media data of the second user for the first user.

[0281] -Capture camera information (used for remote 3D modeling and coding, etc.):

[0282] >NonDerivedParameters(pose_functionality_type=2);

[0283] - User space set information (optional):

[0284] SpaceSetReferenceStruct()

[0285] SpaceSetSizeStruct()

[0286] >(VisionSubSpaceStruct())

[0287] >CaptureSubSpaceStruct()

[0288] >SubSpaceReferencePointStruct()

[0289] -Media object capture information (if all processing is performed by XR Media Generator 850)

[0290] >ObjectSizeStruct()

[0291] >DefaultOrientationStruct()

[0292] >DefaultRenderingParamStruct()

[0293] The parameters of the fourth interface (804) are: the list of media typically received, such as DASH MPD.

[0294] Parameters of the fifth interface (805): When latency is critical, specific metadata can be directly transmitted between the XR media generator 850 and the server (or second UE) 82. In other words, media data can be transmitted directly to the server (or second UE) 82 via the fifth interface (805) without going through the XR interaction controller 830.

[0295] Figure 9 This is a schematic diagram illustrating a method for transmitting 3D XR media data to a second UE, performed by a first UE, according to an embodiment of the present disclosure.

[0296] Reference Figure 9 In operation 910, the first UE can receive a capability and status report about at least one component device from at least one component device.

[0297] In operation 920, the first UE can transmit a device capability report about the XR service to the server based on capability and status reports.

[0298] In operation 930, the first UE can receive device configuration information for XR services from the server.

[0299] In operation 940, the first UE can establish an XR service session based on device configuration information.

[0300] In operation 950, the first UE can process 3D media data and metadata related to the XR service obtained by controlling at least one component device.

[0301] In operation 960, the first UE can transmit the processed 3D media data and metadata to the second UE via the server.

[0302] Figure 10 This is a schematic diagram illustrating the configuration of a UE or component device according to embodiments of this disclosure. (Refer to...) Figure 1A , Figure 1B , Figure 2A , Figure 2B and Figures 3 to 9 The described UE, XR device, or component device may each have, for example: Figure 10 The configuration shown. Optionally, some component devices may include those with... Figure 10 The components in the system are different (e.g., camera, low-power processor, display, short-range communication module, etc.). For ease of description, the following text will use the UE as an example.

[0303] Reference Figure 10 The UE may include a transceiver 1020, a memory 1030, and a processor 1010. However, the components of the UE are not limited to these. For example, the UE may include more or fewer components than those described above. For example, the UE may not include the memory 1030. Furthermore, the transceiver 1020, the memory 1030, and the processor 1010 may be implemented as a single chip. In addition, the processor 1010 may include one or more processors.

[0304] The receiver and transmitter are collectively referred to as transceiver 1020, and can transmit and receive signals to and from a server, component device, XR device, or other UE. For example, transceiver 1020 can transmit and receive control signals, media data, and metadata. To achieve this, transceiver 1020 may include an RF transmitter for up-converting and amplifying the frequency of the signal to be transmitted, and an RF receiver for low-noise amplification of the received signal and down-converting its frequency. However, this is merely an example of transceiver 1020, and the components of transceiver 1020 are not limited to RF transmitters and RF receivers.

[0305] In addition, transceiver 1020 can receive signals via a radio channel and output the signals to processor 1010, and transmit signals output from processor 1010 via a radio channel.

[0306] The memory 1030 can store data and programs required for UE operation. Furthermore, the memory 1030 can store control information or data included in signals received by the UE. The memory 1030 may include storage media such as read-only memory (ROM), random access memory (RAM), hard disk, compact disc (CD)-ROM, and digital versatile disc (DVD), or combinations thereof. Alternatively, the memory 1030 may not exist separately, but may be included within the processor 1010.

[0307] Processor 1010 can control a series of processes, enabling the UE to operate according to embodiments of this disclosure. For example, processor 1010 can receive control signals, media data, and metadata via transceiver 1020, and process the received control signals, media data, and metadata. Additionally, processor 1010 can transmit the processed control signals, media data, and metadata via transceiver 1020. Processor 1010 may include multiple processors and execute programs stored in memory 1030 to perform operations on components controlling the UE.

[0308] Figure 11 This is a schematic diagram illustrating the configuration of a server according to embodiments of the present disclosure. (Refer to...) Figure 1A , Figure 1B , Figure 2A , Figure 2B and Figures 3 to 9 The described cloud, server, or MEC server may have the following characteristics: Figure 11 The configuration shown is illustrated below. For ease of description, the following text will use a server as an example.

[0309] Reference Figure 11 The server may include a transceiver 1120, a memory 1130, and a processor 1110. However, the components of the server are not limited to these. For example, the server may include more or fewer components than described above. For example, the server may not include the memory 1130. Furthermore, the transceiver 1120, memory 1130, and processor 1110 may be implemented as a single chip. In addition, the processor 1110 may include one or more processors.

[0310] The receiver and transmitter are collectively referred to as transceiver 1120, and can transmit and receive signals to and from a UE, component device, XR device, or another server. For example, transceiver 1120 can transmit and receive control signals, media data, and metadata. To achieve this, transceiver 1120 may include an RF transmitter for up-converting and amplifying the frequency of the signal to be transmitted, and an RF receiver for low-noise amplification of the received signal and down-converting its frequency. However, this is merely an example of transceiver 1120, and the components of transceiver 1120 are not limited to RF transmitters and RF receivers.

[0311] In addition, transceiver 1120 can receive signals via a radio channel and output the signals to processor 1110, and transmit signals output from processor 1110 via a radio channel.

[0312] The memory 1130 can store data and programs required for UE operation. Furthermore, the memory 1130 can store control information or data included in signals received by the server. The memory 1130 may include storage media such as ROM, RAM, hard disk, CD-ROM, and DVD, or combinations thereof. Alternatively, the memory 1130 may not exist separately, but may be included within the processor 1110.

[0313] Processor 1110 can control a series of processes, enabling the server to operate according to embodiments of this disclosure. For example, processor 1110 can receive control signals, media data, and metadata via transceiver 1120, and process the received control signals, media data, and metadata. Additionally, processor 1110 can transmit the processed control signals, media data, and metadata via transceiver 1120. Processor 1110 may include multiple processors and execute programs stored in memory 1130 to perform operations on components controlling the server.

[0314] Figure 12 This is a flowchart illustrating a method performed by a first terminal according to an embodiment of the present disclosure.

[0315] Reference Figure 12 In operation 1201, the first terminal can identify the capability of the first terminal to be connected to at least one component device. For example, the at least one component device may include at least one of a camera, a speaker, a display, and a sensor.

[0316] In operation 1203, the first terminal can establish a session associated with an augmented reality (AR) service via a server based on its capabilities. For example, the first terminal can communicate with the server to establish a session, and the AR service can include AR calls between the first terminal and a second terminal. In this embodiment, the type and configuration of the session are identified based on the capabilities of the first terminal. During session establishment, the format associated with the 3D media data is determined.

[0317] In operation 1205, the first terminal may perform preprocessing on the 3D media data acquired by at least one component device. For example, preprocessing may include format conversion. In an embodiment, the preprocessed 3D media data is encoded before being transmitted to the second terminal.

[0318] In operation 1207, the first terminal can transmit pre-processed 3D media data to the second terminal in real time.

[0319] Figure 13 This is a flowchart illustrating a method performed by a second terminal according to an embodiment of the present disclosure.

[0320] Reference Figure 13 In operation 1301, the second terminal can recognize the capability of a second terminal connected to at least one component device. For example, the at least one component device may include at least one of a camera, a sensor, a display, and a speaker.

[0321] In operation 1303, the second terminal can establish a session associated with an augmented reality (AR) service via a server based on its capabilities. For example, the second terminal can communicate with the server to establish a session, and the AR service can include AR calls between the first and second terminals. In this embodiment, the type and configuration of the session are identified based on the capabilities of the second terminal. During session establishment, the format associated with the 3D media data is determined.

[0322] In operation 1305, the second terminal can receive 3D media data from the first terminal in real time.

[0323] In operation 1307, the second terminal can perform post-processing on the 3D media data. For example, post-processing may include format conversion.

[0324] In operation 1309, the second terminal can render the post-processed 3D media data on the second terminal. In this embodiment, the post-processed 3D media data is decoded before rendering.

[0325] According to embodiments of this disclosure, a method for transmitting 3D DAR media data to a second UE, performed by a first user equipment (UE), includes: receiving a capability and status report about at least one component device from at least one component device; transmitting a device capability report about an XR service to a server based on the capability and status report; receiving device configuration information for the XR service from the server; establishing an XR service session based on the device configuration information; processing 3D media data and metadata related to the XR service obtained by controlling at least one component device; and transmitting the processed 3D media data and metadata to the second UE via the server.

[0326] The at least one component may include: one or more visual camera devices configured to acquire 3D information about the surrounding environment of a first user of the first UE; one or more capture camera devices configured to acquire 3D information about objects around the first user; a rendering device configured to render 3D media data related to the XR service of the second UE; and an XR device for displaying the rendered 3D media data.

[0327] Capability and status reports may include at least one of the following: location information, orientation information, and hardware capability information of at least one component device.

[0328] The device capability report may include user space set parameters, and the user space set parameters may include information about the space surrounding the first user of the first UE and information about the location and orientation of at least one component device in the surrounding space.

[0329] The user space set parameters may include at least one subspace set parameter, and the at least one subspace set parameter may include at least one of one or more visual subspace set parameters or one or more capture subspace set parameters.

[0330] One or more visual subspace set parameters can represent the target space, wherein one or more visual camera devices from at least one component device acquire 3D information about the surrounding environment of the first user, and one or more capture subspace set parameters can represent the target space, wherein one or more capture camera devices from at least one component device acquire 3D information about objects around the first user.

[0331] The method may further include: receiving from a server a list of multiple XR services including requirement information for each XR service; selecting one or more XR services from the list of multiple XR services based on capability and status reports; and transmitting to the server a device capability report on the selected one or more XR services.

[0332] The method may also include transmitting a request to the server for at least some of a plurality of 3D media processes for processing 3D media data to be executed by the server, based on capability and status reports and demand information about one or more selected XR services.

[0333] The method may also include receiving information from the server for configuring at least some of a plurality of 3D media processes for processing 3D media data to be processed by the server.

[0334] The method may further include: receiving from the second UE 3D media data associated with a second user of the second UE and user space set parameters associated with the second user; generating a 3D media object by processing the 3D media data associated with the second user based on the user space set parameters associated with the second user; and controlling the display of the first UE or XR device to display the 3D media object.

[0335] According to another embodiment of this disclosure, a first UE for transmitting 3D XR media data to a second UE includes: a transceiver; and at least one processor configured to: control the transceiver to receive capability and status reports about at least one component device from at least one component device; based on the capability and status reports, control the transceiver to transmit device capability reports about XR services to a server; control the transceiver to receive device configuration information for XR services from the server; establish an XR service session based on the device configuration information; process 3D media data and metadata related to XR services obtained by controlling at least one component device; and control the transceiver to transmit the processed 3D media data and metadata to the second UE via the server.

[0336] The methods of the embodiments of this disclosure described in the appended claims or their specification can be implemented in hardware, software, or a combination of hardware and software.

[0337] When the method is implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured to be executed by one or more processors within an electronic device. The one or more programs may include instructions that cause the electronic device to perform the method according to the embodiments of this disclosure described in the claims or their specification. Furthermore, a computer program product storing one or more programs may be provided.

[0338] These programs (software modules or software) can be stored in RAM, non-volatile memory including flash memory, ROM, electrically erasable programmable ROM (EEPROM), disk storage devices, CD-ROMs, DVDs or other types of optical storage devices, and magnetic tape. Optionally, the programs can be stored in a memory configured as a combination of some or all of the memories. Furthermore, multiple such memories may be included.

[0339] Furthermore, the program can be stored in an attachable storage device, which can be accessed via a communication network such as the Internet, intranet, local area network (LAN), wide area network (WLAN), or storage area network (SAN), or a combination thereof. The storage device can access means for performing operations according to embodiments of this disclosure via an external port. Additionally, a separate storage device on the communication network can also access the means for performing operations according to embodiments of this disclosure.

[0340] In the specific embodiments described above, the components included in this disclosure are expressed in a singular or plural form according to the embodiments presented herein. However, the choice of singular or plural representation is suitable for the circumstances presented for ease of description, and this disclosure is not limited to singular or plural forms. Elements expressed in plural form may be configured as a single element, or elements expressed in singular form may be configured as multiple elements.

[0341] The embodiments of this disclosure presented in the specification and accompanying drawings are provided only as specific examples to facilitate the description of the technical details according to this disclosure and to aid in understanding the disclosure, and are not intended to limit the scope of the disclosure. In other words, other modifications can obviously be made based on the technical spirit of this disclosure by those skilled in the art. Furthermore, the embodiments of this disclosure can be combined with each other for operation when needed. For example, parts of the embodiments of this disclosure and other embodiments of this disclosure can be combined with each other to enable operation of a UE, component devices, XR devices, and servers. In addition, the embodiments of this disclosure are also applicable to other communication systems, and other modifications based on the technical spirit of the embodiments of this disclosure are also possible.

[0342] Although this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. A method performed by a first terminal in a wireless communication system, the method comprising: The ability to identify at least one component device connected to the first terminal; Receive information from the server indicating the list of Extended Reality XR services and the configuration information of the XR services; Based on the capabilities of the at least one component device and the configuration information of the XR service, determine the XR service in the XR service list; The server receives information indicating the entry point corresponding to the XR service, wherein the information indicating the entry point includes identification information of the data network accessible to the first terminal to receive the XR service; Establish a session for the XR service, where the XR service session is associated with the entry point; as well as At least one processing is performed on the three-dimensional (3D) media data of the XR service, wherein the processed 3D media data is transmitted to a second terminal.

2. The method according to claim 1, wherein, The 3D media data is further processed by the cloud based on the capabilities of the first terminal.

3. The method according to claim 2, further comprising: Based on the capabilities of the first terminal, the processing to be performed by the cloud is determined.

4. The method according to claim 1, wherein, The 3D media data is acquired by the at least one component device.

5. The method according to claim 1, wherein, The format associated with the 3D media data is determined during the session establishment.

6. The method according to claim 1, wherein, The XR service includes XR calls between the first terminal and the second terminal.

7. The method according to claim 1, wherein, The at least one component device includes a camera.

8. The method according to claim 1, wherein, The 3D media data is transmitted in real time.

9. A method performed by a second terminal in a wireless communication system, the method comprising: The ability to identify at least one component device connected to the second terminal; Receive information from the server indicating the list of Extended Reality XR services and the configuration information of the XR services; Based on the capabilities of the at least one component device and the configuration information of the XR service, determine the XR service in the XR service list; The server receives information indicating the entry point corresponding to the XR service, wherein the information indicating the entry point includes identification information of the data network accessible to the first terminal to receive the XR service; Establish a session for the XR service, where the XR service session is associated with the entry point; Receive three-dimensional (3D) media data from the first terminal; Perform at least one process on the 3D media data; as well as The processed 3D media data is rendered on the second terminal.

10. The method according to claim 9, wherein, The 3D media data is further processed by the cloud based on the capabilities of the second terminal.

11. The method of claim 10, further comprising: Based on the capabilities of the second terminal, the processing to be performed by the cloud is determined.

12. The method according to claim 9, wherein, The format associated with the 3D media data is determined during the session establishment.

13. A first terminal in a wireless communication system, the first terminal comprising: transceiver; as well as At least one processor is configured as follows: The ability to identify at least one component device connected to the first terminal. The transceiver receives information from the server indicating the list of Extended Reality (XR) services and configuration information for the XR services. Based on the capabilities of the at least one component device and the configuration information of the XR service, determine the XR service in the XR service list. The transceiver receives information from the server indicating the entry point corresponding to the XR service. This information includes identification information of the data network accessible to the first terminal for receiving the XR service. Establish a session for the XR service, where the XR service session is associated with the entry point, and At least one processing is performed on the three-dimensional (3D) media data of the XR service, wherein the processed 3D media data is transmitted to a second terminal.

Images