Device and method for spatial synchronization between xr maps

Abstract

Spatial synchronization between extended reality (XR) graphs is provided. An example method includes determining a first coordinate transform between a first coordinate system that is a global coordinate system of a primary graph and a second coordinate system of a first node in the primary graph, determining a second coordinate transform between the second coordinate system and a third coordinate system of a second node in a secondary graph, determining a third coordinate transform between the first coordinate system and the third coordinate system based on the first coordinate transform and the second coordinate transform, and transmitting the third coordinate transform and an identifier of the second node to a Tenderer of the secondary graph. The Tenderer uses this information to determine a fifth coordinate transform P B’ B between the first coordinate system and the fourth coordinate system. The Tenderer can then convert the coordinates of a node or trackable object into the coordinate system of the secondary graph.

Description

Cross-references to related applications

[0001] This application claims priority to European Patent Application No. 24305458.2, filed on 27 March 2024, the entire disclosure of which is incorporated herein by reference. Background Technology

[0002] AR applications require seamlessly integrating virtual content into the user's real-world environment. Such applications can leverage the concepts of anchors, traceable objects, and XR space described in 3GPP TS26.119 "Device Media Capabilities for Augmented Reality Services (Rev. 18)". An anchor is a virtual element whose position, orientation, scale, and other attributes are expressed in a traceable space defined by a traceable object. The position, orientation, scale, and other attributes of a virtual asset are expressed relative to the anchor. A traceable object is a real-world object that can be traced by the XR runtime. Each traceable object provides a local reference space, also called the traceable space, where anchors can be expressed. XR space is the reference frame in which 3D coordinates are expressed. Examples of anchors, traceable objects, and XR space are shown below. Figure 1 The illustration shows a traceable object 102, an anchor point 104, and a virtual asset 106. Summary of the Invention

[0003] A method according to some embodiments includes: determining a first coordinate transformation P between a first coordinate system B, which serves as a global coordinate system of a main graph, and a second coordinate system Bref of a first node in the main graph. B Bref Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ Based on the first and second coordinate transformations, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'. align ; and transmit the third coordinate transformation and the identifier of the second node to the renderer associated with the auxiliary graph.

[0004] An apparatus according to some embodiments includes one or more processors configured to at least perform: determining a first coordinate transformation P between a first coordinate system B, which serves as a global coordinate system of a master graph, and a second coordinate system Bref of a first node in the master graph. B Bref Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ Based on the first and second coordinate transformations, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'.align ; and transmit the third coordinate transformation and the identifier of the second node to the renderer associated with the auxiliary graph.

[0005] A computer-readable medium according to some embodiments stores instructions for performing a method, the method comprising: determining a first coordinate transformation P between a first coordinate system B, which serves as a global coordinate system of a main graph, and a second coordinate system Bref of a first node in the main graph. B Bref Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ Based on the first and second coordinate transformations, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'. align ; and transmit the third coordinate transformation and the identifier of the second node to the renderer associated with the auxiliary graph.

[0006] In some embodiments, the third coordinate transformation and the identifier of the second node are transmitted via a configuration method of the application programming interface.

[0007] Some embodiments also include: determining a fourth coordinate transformation P between the third coordinate system Bref' and the fourth coordinate system B' by a renderer associated with the auxiliary graph. B’ Bref’ The fourth coordinate system is the global coordinate system of the auxiliary graph; and the fifth coordinate transformation P, based on the third and fourth coordinate transformations, determines the relationship between the first and fourth coordinate systems. B’ B .

[0008] Some embodiments also include: obtaining the pose of at least one mapped node or trackable object in a first coordinate system B by a renderer associated with the auxiliary graph; transforming the pose to a fourth coordinate system B' using a fifth coordinate transformation; and rendering at least a portion of the auxiliary graph using the transformed pose.

[0009] In some embodiments, the pose of a mapping node or trackable object is provided through a configuration method of an application programming interface.

[0010] Some embodiments also include: selecting a first node in the main graph, wherein the first node is selected as a camera node.

[0011] Some embodiments also include: transmitting the identifier of the second node to the renderer associated with the auxiliary graph.

[0012] In some embodiments, the renderer associated with the secondary image is an audio renderer.

[0013] Some embodiments also include providing a list of traceable object identifiers to the renderer associated with the secondary graph.

[0014] In some embodiments, the update pose of at least one mapping node or trackable object is provided to the renderer associated with the auxiliary graph periodically or in response to a predefined event.

[0015] In some embodiments, the updated pose of a mapping node or traceable object is provided through an update method of an application programming interface.

[0016] In some embodiments, the updated pose is provided in the first coordinate system B, and the renderer associated with the auxiliary graph uses a fifth coordinate transformation to transform the pose to the fourth coordinate system B'.

[0017] Some embodiments also include rendering at least a portion of the auxiliary graph using a transformed updated pose. Attached Figure Description

[0018] Figure 1 An extended reality (XR) space for augmented reality (AR) applications is shown, including anchor points, trackable objects, and virtual assets.

[0019] Figure 2 An MPEG node mapping extension for synchronizing visual and audio graphs is shown.

[0020] Figure 3 This is a flowchart illustrating the configuration phase process performed for each external graph according to some embodiments.

[0021] Figure 4 The spatial relationship between two graphs extended using node mapping according to some embodiments is illustrated schematically.

[0022] Figure 5 The spatial transformation determination for graph alignment is illustrated schematically according to some embodiments.

[0023] Figure 6 The diagram illustrates graph synchronization in a hierarchical layout using traceable objects, anchors, and node mappings according to some embodiments.

[0024] Figure 7 The diagram illustrates graph synchronization in a non-hierarchical layout using traceable object, anchor, and node mappings according to some embodiments.

[0025] Figure 8 This is a flowchart illustrating the runtime phase methods performed for each external graph according to some embodiments.

[0026] Figure 9 This is a flowchart of a spatial synchronization method between graphs, as performed in some embodiments.

[0027] Figure 10 This is a functional block diagram of a device on which the embodiments described herein can be implemented. As an example, the renderer for the main diagram and / or auxiliary diagram can be, for example... Figure 10 Implemented on the device shown. Detailed Implementation

[0028] Overview of graph synchronization support in MPEG-I scene description Within the framework of MPEG-I scene description, the m66705 contribution (MPEG-I WG3 m66705, a generic API for rendering engines, January 2024) introduced a generic application programming interface (API) called the Render Control API (summarized in Table 2) and a node mapping glTF extension (summarized in Table 1) in an attempt to provide temporal and spatial synchronization between external parts / graphs and the main graph. Figure 2 An example of using this extension to provide synchronization between the main graph 202 (represented here by a virtual car) and the auxiliary graph 204 is shown. In one example, the main graph 202 may be associated with a visual renderer, while the auxiliary graph 204 may be associated with an audio renderer. For example, different nodes 206, 208, and 210 of the auxiliary graph may represent different audio sources, and these different audio sources may be expected to coordinate with different features of the main graph 202.

[0029]

[0030] Table 1: Semantics of MPEG Node Mapping glTF Extension

[0031] Table 2: Rendering Control API Problems solved in some embodiments The features of the rendering control API and the glTF node mapping extension are not yet fully developed. The current version does not define how spatial synchronization is achieved between the main graph and outer graphs with the node mapping extension shown in Table 1. As an example, it is unclear how bounding boxes are used for XR spatial alignment in the configure() method. In another example, it is unclear how a simple list of anchor points can be used in the configure() method to place virtual content in the same position in both the main graph and all outer graphs. In yet another example, it is unclear how the transformation matrix is ​​computed in the update() method to maintain spatial synchronization between the main graph and outer graphs.

[0032] Exemplary embodiments provide a method and apparatus for spatially synchronizing several external graphs with a main graph that constitutes an extended reality (XR) application. Exemplary embodiments also correspondingly provide configuration for a rendering control API and a node mapping glTF extension. Some embodiments can be described as including a configuration phase and a runtime phase.

[0033] Exemplary configuration phase In an exemplary embodiment, a configuration phase is performed in response to the device receiving a new XR scene description. This phase may include, for example: Figure 3 The process shown can be performed for each external graph. In the exemplary configuration process, a scene description file is obtained (302). The selection of node mappings for references used for spatial synchronization is performed (304). The spatial transformation for alignment of XR spaces between graphs is determined (306). If the scene is an AR scene (308), the configuration of the XR runtime is performed for each traceable object (310). Subsequently, spatial synchronization between graphs is initialized (314).

[0034] In some embodiments, the selection of referenced node mappings can be performed as follows: MPEG node mapping glTF extensions can be added to several nodes of the main graph. When constructing the graph, the content creator can ensure spatial consistency between the nodes of the main graph with node mapping glTF extensions and the corresponding nodes of the external graph identified by the referenceId parameter of the node mapping glTF extension.

[0035] The main graph and the outer graph have dedicated global XR spaces. Each node in the graph has translation, rotation, and scaling (TRS) parameters that define a local XR space relative to its parent node, such as... Figure 4 As shown.

[0036] In an exemplary embodiment, when nested nodes of the outer graph are mapped, all their parent nodes are also mapped to ensure that the entire hierarchical structure is spatially synchronized.

[0037] During the configuration phase, as described above, a node mapping that will serve as a reference for spatial synchronization among all defined node mappings of each outer graph is selected (304). This selection can be performed in different ways in different embodiments. In one embodiment, the reference node mapping can be randomly selected among all node mappings. In a second embodiment, the reference node mapping can be randomly selected among all node mappings having the root node of the outer graph. In a third embodiment, the content creator can identify the reference node mapping by including parameters such as the boolean value isReferenceNodemapping in the semantics of the MPEG node mapping glTF extension, as described below. This selection can be guided by spatial precision criteria (e.g., the 3D vertices of the relevant node geometry have higher precision than other mapping nodes). In a fourth embodiment, the reference node mapping is a camera node, because spatial consistency of cameras across graphs is desirable for rendering.

[0038] In some embodiments, the determination of the spatial transformation for graph alignment can be performed as follows. This part of the configuration phase can correspond to the determination (306) of the spatial transformation between the global XR space of the main graph and the global space of each outer graph. This spatial transformation corresponds to matrix P. B’ B And it can be described by which the input 3D coordinates represented in the global XR space B of the main graph can be transformed to the 3D coordinates represented in the global space B' of the outer graph through equation (1):

[0039] Based on node mapping extension, the matrix P can be executed. B’ B The calculation. Figure 5 The calculation process is illustrated schematically. For example... Figure 5 As shown, the nodes in the main graph have referenced node mappings (selected as described above). ref The local XR space B is defined. ref and the mapping transformation matrix P Bref Bref’ (Table 1 shows the transformation parameters for the node mapping glTF extension). Nodes in the external graph. ref' The referenceId parameter is referenced by the node mapping glTF extended from Table 1.

[0040] For any point P:

[0041]

[0042] Then, in cases (2) and (3):

[0043] And finally, in cases (1) and (4):

[0044] Referring to formula (5), it can be noted that without information exchange, the renderer of the main image does not know matrix P. B’ Bref’ This defines the coordinate transformation between the coordinate system (B') of the auxiliary graph and the coordinate system (Bref') of the nodes in the auxiliary graph. Conversely, without information exchange, the renderer of the outer graph is unaware of the matrix product [P]. Bref Bref’ ] -1 [P B Bref ] -1 It is also unknown which node in the external graph. ref' The referenceId parameter of the referenced node mapping is referenced. In an exemplary embodiment, the matrix product and the referenceId of the reference (e.g., the identifier of the node in the outer graph corresponding to the selected node mapping of the reference in the main graph) are provided to the outer graph in the configure() method of the rendering control API, which is known to the renderer of the main graph.

[0045] For clarity, this matrix product can be called the alignment matrix P. align

[0046] In an exemplary embodiment, upon receiving the alignment matrix P align After obtaining the referenceId, the renderer of the outer graph can calculate and store the spatial transformation matrix P, which will be used to express the pose in the global space B' of the outer graph using formula (1). B’ B .

[0047] In some embodiments, the configuration of the XR runtime for each traceable object can be performed as follows. This process can be performed for AR applications that define traceable objects and anchors in the semantics of the main graph and outer graph. In some embodiments, hierarchical combinations of shared traceable object, anchor, and node mappings are supported, such as... Figure 6 As shown. In some embodiments, a non-hierarchical combination of shared traceable objects, anchors, and node mappings is supported, such as... Figure 7 As shown.

[0048] For each defined traceable object, the device's XR runtime can be configured to detect and track these objects during runtime. This configuration can be done using the Khronos OpenXR API. For example, the ground can be set as a traceable object using `xrCreateReferenceSpace()` with an `XR_REFERENCE_SPACE_TYPE_STAGE` value of the `XrReferenceSpaceType` enumeration. A traceable object associated with an arbitrary free space point can be set in the user's real-world environment using `xrCreateSpatialAnchorFB()` and `xrCreateSpatialAnchorMSFT()`. A 2D marker can be set as a traceable object using `xrCreateMarkerSpaceVARJO()`. These methods create and return a handle to the XR space of the traceable object.

[0049] In some embodiments, the initialization of spatial synchronization between graphs can be performed as follows. This process initializes spatial synchronization between the main graph and the outer graph. It also provides the outer graph with the initial pose of the nodes of the outer graph (referred to as "mapped" nodes), identified by the referenceId parameter extended by the node mapping glTF.

[0050] Because configuring the XR runtime for each trackable object in an AR application is only completed in the previous step, the initial pose of the trackable object may not yet be available. Conversely, the process also provides the initial pose of the trackable object.

[0051] For each mapping node, the pose can be expressed in the global XR space B of the main graph using formula (2), where (x ref ',y ref ',z ref ') Bref’ = (0,0,0) (that is, in B) ref’ The origin of the mapping nodes expressed in XR space.

[0052] In some embodiments, in order to obtain the pose of a trackable object, the Khronos OpenXR xrLocateSpace() method can be used to retrieve XrPosef information (the location and orientation of the XR space origin of the trackable object) from the XrSpaceLocation structure with the following input parameters: the XR space of the trackable object to be located, the base / parent space representing the pose (e.g., the global XR space B of the main graph), and the time providing the pose (e.g., the current time).

[0053] All these initial poses are computed and expressed in the global XR space B of the main graph before the configure() method of the rendering control API is called. These initial poses are then calculated by each outer graph using equation (1), and by equations (5) and (6) based on the referenceId (the identifier of the node of the outer graph corresponding to the selected node mapping of the reference of the main graph). B’ B To convert to the relevant global XR space B'.

[0054] Exemplary runtime phase In an exemplary embodiment, the runtime phase is executed per frame (e.g., per rendering cycle). In some embodiments, this phase may be as follows: Figure 8 Execute as shown. Figure 8 The method can be performed on a per-external-graph basis. For each frame (802), checks (804, 806) are performed to determine if spatial synchronization updates are needed. If so, the poses of the mapped nodes are computed (808). If the scene is an AR scene (810), the poses of the trackable objects are retrieved from the XR runtime (814) for each trackable object (812). The spatial synchronization of the graph is then updated (816).

[0055] In some embodiments, checks may be performed to determine whether / when to perform spatial synchronization updates. The occurrence of synchronization updates may be specified by the content creator. In some embodiments, updates may be periodic, such as per frame or per N frames. In some embodiments, updates may be event-based, such as based on trigger activations (e.g., visibility, proximity, collision, user input) that may be defined in the interactive semantics of the main graph (e.g., in MPEG_scene_interactivity, MPEG_node_interactivity glTF extensions). In some embodiments, updates may be based on a logical combination of periodicity and / or event occurrence.

[0056] In some embodiments, if identified by the content creator, these semantics are added to the semantics of the MPEG node mapping glTF extension of the referenced mapping node, or to the glTF scene extension.

[0057] In some embodiments, synchronous updates occur on a per-frame basis and do not provide additional semantics.

[0058] In some embodiments, the calculation of the pose of the mapped node can be performed as follows. This process is similar to the calculation of the initial pose of the node in the external graph identified by the referenceId parameter of the node mapping glTF extension.

[0059] For each mapping node, the pose is updated and expressed in the global XR space B of the main graph using formula (2), where (x ref ',y ref ',z ref ') Bref’ = (0,0,0) (that is, in XR space B) ref’ The origin of the mapping node expressed in the text.

[0060] In some embodiments, the pose of a trackable object can be retrieved as follows. This process can be performed only for AR applications where trackable objects and AR anchors are defined semantically in the main and outer graphs.

[0061] During runtime, the pose of a trackable object can change (e.g., predictions may become more accurate and / or the trackable object may have a new pose in the user's environment). This process can be similar to the retrieval of the trackable object's pose as described herein. All these trackable object poses can be represented in the global XR space B of the master graph.

[0062] In some embodiments, spatial synchronization between update graphs can be performed as follows: The poses of mapped nodes and trackable objects are computed and represented in the global XR space B of the main graph before the update() method of the rendering control API is invoked.

[0063] Then, these poses are rendered by the renderer of each outer graph using Equation (1) and utilizing the spatial transformation matrix P calculated from Equation (5). B’ B Transform to the relevant global XR space B'.

[0064] Exemplary rendering control API and node mapping semantics In some embodiments, the rendering control API may be implemented as follows.

[0065] The configure() and update() methods of the rendering control API can be configured to adapt to the spatial synchronization methods described above. An example of the configure() method is described in Table 3. An example of the update() method is provided in Table 4.

[0066]

[0067] Table 3: Exemplary configure() methods of the rendering control API.

[0068]

[0069] Table 4: Exemplary update() methods of the rendering control API In some embodiments, node mapping semantics can be implemented as follows. The semantics of the MPEG node mapping extension (Table 5) can be configured based on the spatial synchronization method described in the foregoing sections.

[0070]

[0071] Table 5: Exemplary MPEG node mapping extensions.

[0072] Methods according to some embodiments, such as Figure 9 As shown. At position 902, the first coordinate transformation P between the first coordinate system B, which serves as the global coordinate system of the main graph, and the second coordinate system Bref of the first node in the main graph is determined. B Bref At position 904, determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ At position 906, based on the first and second coordinate transformations, the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref' is determined. align At point 908, the identifier of the third coordinate transformation and the second node (e.g., referenceId) is transmitted (e.g., via API) to the renderer associated with the auxiliary graph. This transmission can occur during the configuration phase.

[0073] In some embodiments, at 910, the renderer associated with the auxiliary graph determines the fourth coordinate transformation P between the third coordinate system Bref' and the fourth coordinate system B'. B’ Bref’ The fourth coordinate system is the global coordinate system of the auxiliary graph. At position 912, based on the third and fourth coordinate transformations, the fifth coordinate transformation P between the first and fourth coordinate systems is determined. B’ B .

[0074] In some embodiments, at 914, the renderer associated with the secondary map acquires the pose of at least one mapped node or trackable object in a first coordinate system B. At 916, the pose is transformed to a fourth coordinate system B' using a fifth coordinate transformation. At 918, at least a portion of the secondary map is rendered using the transformed pose. As described in more detail above, updated poses of nodes and / or trackable objects can be provided to the renderer associated with the secondary map during an XR session, and these poses can also be rendered using the fifth coordinate transformation P. B’ B Transform it and use it as appropriate for rendering auxiliary images.

[0075] It should be noted that when describing any transformation as being transmitted, determined, derived, calculated, or otherwise used, this includes transmitting, determining, deriving, calculating, or otherwise using the inverse transformation of the coordinate transformation, since the coordinate transformation and its inverse transformation can be calculated from each other. For example, an exemplary embodiment would include providing a transformation [P] to a renderer associated with an auxiliary map. align ] -1 Instead of P align This is because it provides enough information to describe the transformation.

[0076] Exemplary system hardware. Exemplary embodiments of a device configured to implement the embodiments described herein may use, for example, Figure 10 The system will be implemented through systems such as [the [system name]]. Figure 10 This is a block diagram illustrating examples of systems implementing various aspects and embodiments. System 1000 may be embodied as an apparatus including the various components described below and configured to perform one or more aspects described in this document. Examples of such apparatus include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0077] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon to implement various aspects, such as those described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040 that may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0078] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device can include one or both encoding and decoding modules. Alternatively, the encoder / decoder module 1030 may be implemented as a separate element of system 1000, or it may be incorporated within processor 1010 as a combination of hardware and software known to those skilled in the art.

[0079] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described herein may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, a main image, one or more auxiliary images, poses, coordinate transformations, and intermediate or final results from equations, formulas, operations, and operational logic processing.

[0080] In some embodiments, the memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Various Video Coding, i.e., a new standard developed by JVET (Joint Video Experts Group)).

[0081] Inputs to the components of system 1000 can be provided through various input devices, as indicated in box 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal.

[0082] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select desired data packets. The RF section of various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners performing various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to the desired frequency band. Various embodiments rearrange the order of the above (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as insert amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0083] Additionally, USB and / or HDMI terminals may include corresponding interface processors for connecting system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 1010 as needed. Similarly, aspects of USB or HDMI interface processing may be implemented, as needed, within a separate interface IC or within processor 1010. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010 and an encoder / decoder 1030 operating in conjunction with memory and storage elements, to process the data stream as needed for presentation on an output device.

[0084] Various components of system 1000 can be housed within an integrated housing. Within the integrated housing, the various components can be interconnected and transmit data between them using suitable connection means 1140 (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0085] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0086] In various embodiments, data is streamed to or otherwise provided to system 1000 using a wireless network, such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 1000, delivering data via an HDMI connection to input box 1130. Still other embodiments use an RF connection to input box 1130 to provide streaming data to system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0087] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be used in televisions, tablets, laptops, mobile phones, or other devices. Display 1100 can also be integrated with other components (e.g., in a smartphone) or standalone (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 1120 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), an optical disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, an optical disc player performs the function of playing the output of system 1000.

[0088] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that implement device-to-device control with or without user intervention are used to transmit control signals between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120. Output devices may be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 may be integrated into a single unit along with other components of system 1000 in an electronic device such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0089] For example, if the RF input section 1130 is part of a separate set-top box, the display 1100 and speaker 1110 can alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0090] The embodiments can be executed by processor 1010 or by computer software implemented by hardware or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 1010 can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0091] Additional examples. A method according to some embodiments includes: determining a first coordinate transformation P between a first coordinate system B, which serves as a global coordinate system of a main graph, and a second coordinate system Bref of a first node in the main graph. B Bref Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ Based on the first and second coordinate transformations, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'. align; and transmit the third coordinate transformation and the identifier of the second node to the renderer associated with the auxiliary graph.

[0092] An apparatus according to some embodiments includes one or more processors configured to at least perform: determining a first coordinate transformation P between a first coordinate system B, which serves as a global coordinate system of a master graph, and a second coordinate system Bref of a first node in the master graph. B Bref Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ Based on the first and second coordinate transformations, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'. align ; and transmit the third coordinate transformation and the identifier of the second node to the renderer associated with the auxiliary graph.

[0093] A computer-readable medium according to some embodiments stores instructions for performing a method, the method comprising: determining a first coordinate transformation P between a first coordinate system B, which serves as a global coordinate system of a main graph, and a second coordinate system Bref of a first node in the main graph. B Bref Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ Based on the first and second coordinate transformations, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'. align ; and transmit the third coordinate transformation and the identifier of the second node to the renderer associated with the auxiliary graph.

[0094] In some embodiments, the third coordinate transformation and the identifier of the second node are transmitted via a configuration method of the application programming interface.

[0095] Some embodiments also include: determining a fourth coordinate transformation P between the third coordinate system Bref' and the fourth coordinate system B' by a renderer associated with the auxiliary graph. B’ Bref’ The fourth coordinate system is the global coordinate system of the auxiliary graph; and the fifth coordinate transformation P between the first and fourth coordinate systems is determined based on the third and fourth coordinate transformations. B’ B .

[0096] Some embodiments also include: obtaining the pose of at least one mapped node or trackable object in a first coordinate system B by a renderer associated with the auxiliary graph; transforming the pose to a fourth coordinate system B' using a fifth coordinate transformation; and rendering at least a portion of the auxiliary graph using the transformed pose.

[0097] In some embodiments, the pose of a mapping node or trackable object is provided through a configuration method of an application programming interface.

[0098] Some embodiments also include: selecting a first node in the main graph, wherein the first node is selected as a camera node.

[0099] In some embodiments, the renderer associated with the secondary image is an audio renderer.

[0100] Some embodiments also include providing a list of traceable object identifiers to the renderer associated with the secondary graph.

[0101] In some embodiments, the update pose of at least one mapping node or trackable object is provided to the renderer associated with the auxiliary graph periodically or in response to a predefined event.

[0102] In some embodiments, the updated pose of a mapping node or traceable object is provided through an update method of an application programming interface.

[0103] In some embodiments, the updated pose is provided in the first coordinate system B, and the renderer associated with the auxiliary graph uses a fifth coordinate transformation to transform the pose to the fourth coordinate system B'.

[0104] Some embodiments also include rendering at least a portion of the auxiliary graph using a transformed updated pose.

[0105] This disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in a specific manner and are generally described in a way that may sound limiting, at least for the purpose of illustrating the various features. However, this is for the purpose of clarity of description and does not limit the disclosure or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in previous documents.

[0106] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments have been specifically shown, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation.

[0107] This document describes various methods, each of which includes one or more steps or actions for implementing the described method. The order and / or use of specific steps and / or actions may be modified or combined unless the correct operation of the method requires a specific order. Additionally, in various embodiments, terms such as "first," "second," etc., may be used to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply a sequence of operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding, but may occur, for example, before, during, or in a time period overlapping with the second decoding.

[0108] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0109] The embodiments described herein can be implemented by computer software, either by a processor or other hardware or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0110] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0111] The embodiments and aspects described herein may be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single embodiment (e.g., discussed only as a method), embodiments of the discussed features may be implemented in other forms (e.g., apparatuses or programs). Apparatuses may be implemented, for example, with suitable hardware, software, and firmware. Methods may be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0112] The references to "an embodiment" or "an embodiment" or "an implementation" or "implementation", as well as other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in a manifestation" or "in an implementation" or "in an implementation", as well as any other variations, appearing throughout this disclosure, do not necessarily refer to the same embodiment.

[0113] Additionally, this disclosure may relate to “determining” various types of information. Determining information may include one or more of the following: for example, estimated information, calculated information, predicted information, or information retrieved from memory.

[0114] Furthermore, this disclosure may relate to “accessing” various types of information. Accessing information may include one or more of the following: for example, receiving information, retrieving information (e.g., retrieving from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0115] Additionally, this disclosure may relate to "receiving" various types of information. Like "access," the intention to receive is a broad term. Receiving information may include one or more of the following: for example, accessing information or retrieving information (e.g., retrieving from memory). Furthermore, "receiving" is generally referred to in one or more ways during operation, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0116] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As yet another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” this wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to as many as the items listed.

[0117] Furthermore, as used herein, the term “signal” specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a specific parameter among several parameters for region-based filter parameter selection for artifact removal filtering. In this way, in embodiments, the same parameter is used at both the encoder and decoder sides. Thus, for example, the encoder can send (explicitly signal) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without sending (implicitly signal) to simply allow the decoder to know and select the specific parameter. Bit savings are implemented in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to send information to the corresponding decoder. Although the verb form of the word “signal” has been referred to above, the word “signal” can also be used as a noun herein.

[0118] Implementation methods can generate various signals that are formatted to carry information, such as information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementation methods. For example, the signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0119] Several embodiments are described. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, means, or aspects individually or in any combination across various claim classes and types: A bitstream or signal that includes one or more of the described syntax elements or their variants.

[0120] A bitstream or signal, which includes a syntax that conveys information generated according to any embodiment of the described embodiments.

[0121] Create and / or send and / or receive and / or decode bitstreams or signals, which include one or more of the described syntax elements or their variants.

[0122] Create and / or send and / or receive and / or decode according to any of the described embodiments.

[0123] The method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the described embodiments.

[0124] It should be noted that the various hardware elements, one or more of which are described in the embodiments, are referred to as “modules”, which implement (i.e., perform, execute, etc.) the various functions described herein in connection with the respective modules. As used herein, a module includes hardware considered suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, and it should be noted that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium such as RAM, ROM, etc.

[0125] Although the features and elements have been described above in specific combinations, each feature or element may be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM discs and digital multifunction disks (DVDs). The processor associated with the software can be used to implement a wireless transceiver for a WTRU, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Obtain scene information describing a scene, the scene including a main graph and at least one auxiliary graph, wherein the auxiliary graph includes at least one node, and wherein the scene information includes timing information associated with the node, the timing information describing the timing of spatial synchronization between the main graph and the node; Determine at least one coordinate transformation between the first coordinate system of the main graph and the second coordinate system of the node in the auxiliary graph; The coordinate transformation is transmitted to the renderer associated with the auxiliary graph; as well as Based on the timing information, the nodes in the auxiliary graph are spatially synchronized with the main graph.

2. An apparatus comprising one or more processors, said one or more processors being configured to perform at least: Obtain scene information describing a scene, the scene including a main graph and at least one auxiliary graph, wherein the auxiliary graph includes at least one node, and wherein the scene information includes timing information associated with the node, the timing information describing the timing of spatial synchronization between the main graph and the node; Determine at least one coordinate transformation between the first coordinate system of the main graph and the second coordinate system of the node in the auxiliary graph; The coordinate transformation is transmitted to the renderer associated with the auxiliary graph; as well as Based on the timing information, the nodes in the auxiliary graph are spatially synchronized with the main graph.

3. The method of claim 1 or the apparatus of claim 2, wherein the timing information is provided in the MPEG node mapping extension.

4. The method of claim 1 or the method of claim 3 as dependent on claim 1, or the device of claim 2 or the device of claim 3 as dependent on claim 2, wherein the timing information includes information indicating that synchronization is performed in the configuration step, and wherein, In response to the timing information, the spatial synchronization is performed in the configuration step.

5. The method of claim 1 or the method of claims 3 to 4 of claim 1, or the device of claim 2 or the device of claims 3 to 4 of claim 2, wherein the scene information comprises an array of triggers, wherein the timing information identifies the index of the trigger in the array, and wherein the spatial synchronization is performed in response to the activation of the trigger having the identified index.

6. The method of claim 1 or the method of claims 3 to 5 of claim 1, or the device of claim 2 or the device of claims 3 to 5 of claim 2, wherein the timing information includes information indicating the number of frames N, and wherein, In response to the timing information, spatial synchronization is performed every N frames.

7. The method of claim 1 or the method of claims 3 to 6 of claim 1, or the device of claim 2 or the device of claims 3 to 6 of claim 2, wherein the timing information includes a logical expression identifying at least one condition, and wherein the spatial synchronization is performed in response to the satisfaction of the logical expression.

8. A method comprising: Determine the first coordinate transformation P between the first coordinate system B, which serves as the global coordinate system of the main graph, and the second coordinate system Bref of the first node in the main graph. B Bref ; Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ ; Based on the first coordinate transformation and the second coordinate transformation, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'. align ;as well as The third coordinate transformation and the identifier of the second node are transmitted to the renderer associated with the auxiliary graph.

9. An apparatus comprising one or more processors, said one or more processors being configured to perform at least: Determine the first coordinate transformation P between the first coordinate system B, which serves as the global coordinate system of the main graph, and the second coordinate system Bref of the first node in the main graph. B Bref ; Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ ; Based on the first coordinate transformation and the second coordinate transformation, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'. align ;as well as The third coordinate transformation and the identifier of the second node are transmitted to the renderer associated with the auxiliary graph.

10. A computer-readable medium storing instructions for performing a method, the method comprising: Determine the first coordinate transformation P between the first coordinate system B, which serves as the global coordinate system of the main graph, and the second coordinate system Bref of the first node in the main graph. B Bref ; Determine the second coordinate transformation P between the second coordinate system Bref of the first node in the main graph and the third coordinate system Bref' of the second node in the auxiliary graph. Bref Bref’ ; Based on the first coordinate transformation and the second coordinate transformation, determine the third coordinate transformation P between the first coordinate system B and the third coordinate system Bref'. align ;as well as The third coordinate transformation and the identifier of the second node are transmitted to the renderer associated with the auxiliary graph.

11. The method of claim 8, the device of claim 9, or the computer-readable medium of claim 10, wherein the third coordinate transformation and the identifier of the second node are transmitted via a configuration method of an application programming interface.

12. The method of claim 8 or the method of claim 11 which is a derivative of claim 8, the device of claim 9 or the device of claim 11 which is a derivative of claim 9, or the computer-readable medium of claim 10 or the computer-readable medium of claim 11 which is a derivative of claim 10, further comprising: The fourth coordinate transformation P between the third coordinate system Bref' and the fourth coordinate system B' is determined by the renderer associated with the auxiliary graph. B’ Bref’ The fourth coordinate system is the global coordinate system of the auxiliary graph; and Based on the third coordinate transformation and the fourth coordinate transformation, a fifth coordinate transformation P between the first coordinate system and the fourth coordinate system is determined. B’ B .

13. The method of claim 12 as dependent on claim 8, the apparatus of claim 12 as dependent on claim 9, or the computer-readable medium of claim 12 as dependent on claim 10, further comprising: The renderer associated with the auxiliary graph obtains the pose of at least one mapping node or trackable object in the first coordinate system B; The pose is transformed to the fourth coordinate system B' using the fifth coordinate transformation; as well as At least a portion of the auxiliary image is rendered using the transformed pose.

14. The method of claim 13 as dependent on claim 8, the device of claim 13 as dependent on claim 9, or the computer-readable medium of claim 13 as dependent on claim 10, wherein the pose of the mapping node or the trackable object is provided by a configuration method of an application programming interface.

15. The method of claim 8 or the method of claims 11 to 14 of claim 8, the device of claim 9 or the device of claims 11 to 14 of claim 9, or the computer-readable medium of claim 10 or the computer-readable medium of claims 11 to 14 of claim 10, further comprising: Select the first node in the main graph, wherein the first node is selected as the camera node.

16. The method of claim 8 or the method of claims 11 to 15 of claim 8, the device of claim 9 or the device of claims 11 to 15 of claim 9, or the computer-readable medium of claim 10 or the computer-readable medium of claims 11 to 15 of claim 10, wherein the renderer associated with the auxiliary diagram is an audio renderer.

17. The method of claim 8 or the method of claims 11 to 16 of claim 8, the device of claim 9 or the device of claims 11 to 16 of claim 9, or the computer-readable medium of claim 10 or the computer-readable medium of claims 11 to 16 of claim 10, further comprising: Provide the renderer associated with the auxiliary graph with a list of traceable object identifiers.

18. The method of claim 8 or the method of claims 11 to 17 of claim 8, the device of claim 9 or the device of claims 11 to 17 of claim 9, or the computer-readable medium of claim 10 or the computer-readable medium of claims 11 to 17 of claim 10, wherein the update pose of at least one mapping node or trackable object is provided periodically or in response to a predefined event to the renderer associated with the auxiliary graph.

19. The method of claim 18 as dependent on claim 8, the device of claim 18 as dependent on claim 9, or the computer-readable medium of claim 18 as dependent on claim 10, wherein the updated pose of the mapping node or the trackable object is provided via an update method of an application programming interface.

20. The method of claim 18 or 19 of claim 8, the apparatus of claim 18 or 19 of claim 9, or the computer-readable medium of claim 18 or 19 of claim 10, wherein the updated pose is provided in the first coordinate system B, and wherein the renderer associated with the auxiliary graph transforms the updated pose to the fourth coordinate system B' using the fifth coordinate transformation.

21. The method of claim 8 or the method of claims 11 to 20 of claim 8, the device of claim 9 or the device of claims 11 to 20 of claim 9, or the computer-readable medium of claim 10 or the computer-readable medium of claims 11 to 20 of claim 10, further comprising: At least a portion of the auxiliary graph is rendered using a transformed updated pose.

Images