Method for sharing spatial anchor

Abstract

Methods and apparatuses for Augmented Reality Scene experience are provided wherein, a set of at least two trackable objects is signaled in a scene description file describing the augmented reality scene, in associated with at least one anchor object, wherein the at least two trackable objects are signaled in the set according to a given order. The scene description file is transmitted to one or more devices which can select a first supported trackable object in the set for setting the at least one anchor object in the augmented reality scene.

Description

[0001] METHOD FOR SHARING SPATIAL ANCHOR

[0002] This application claims the priority to European Application No. 23306521.8, filed on 14 September 2023, which is incorporated herein by reference in its entirety.

[0003] TECHNICAL FIELD

[0004] At least one of the present embodiments generally relates to augmented reality and more particularly to the anchors used for positioning in a virtual environment. BACKGROUND

[0005] Augmented reality (AR) is a concept and a set of technologies for merging real and virtual elements to produce visualizations where physical and digital objects co-exist and interact in real time. AR visualizations require a means to see augmented virtual elements as a part of the physical view. This can be implemented using an augmented reality terminal (AR terminal) equipped with a camera and a display, which captures video from the user's environment and combines this captured information with virtual elements on a display. Examples of such devices are such as smartphones, tablets or head-mounted displays. 3D models and animations are the most obvious virtual elements to be visualized in AR. However, AR objects can more generally be any digital information for which spatiality (3D position and orientation in space) gives added value, for example pictures, videos, graphics, text, and audio. AR visualizations can be seen correctly from different viewpoints, so that when a user changes his / her viewpoint, virtual elements stay or act as if they would be part of the physical scene. This requires tracking technologies for deriving 3D properties of the environment to produce AR content, and when viewing the content, for tracking the position of the AR terminal with respect to the environment. The AR terminal’s position can be tracked, for example by tracking known objects or visual features in the AR terminal's video stream and / or using one or more sensors. Before the AR objects can be augmented into physical reality, their positions must be defined with respect to the physical environment. A particular challenge of augmented reality is when multiple users access the same AR scene and thus can interact together though this virtual environment. A precise and reliable positioning of AR terminals is a critical aspect of an AR system since such feature is mandatory to enjoy the AR experience.

[0006] SUMMARY

[0007] According to a first aspect, a method for generating a scene description file describing an augmented reality scene is provided. The method comprises signaling in the scene description file, in associated with at least one anchor object, a set of at least two trackable objects, herein the at least two trackable objects are signaled in the set according to a given order, and transmitting the scene description file to one or more devices. In a variant, the given order depends on one of an accuracy associated with a type of trackable objects, a tracking or detection complexity of trackable objects, a memory usage for tracking or detecting trackable objects.

[0008] In another aspect, a method for setting at least one anchor object in an augmented reality scene is provided. The method comprises receiving, by a user’s device, a scene description file describing the augmented reality scene, for at least one anchor object in the scene description file, selecting a trackable object in a set of at least two trackable objects associated with the at least one anchor object in the scene description file, wherein the selecting is based a given criteria, and setting the at least one anchor using the selected trackable object. In a variant, selecting the trackable object in the set comprises parsing the set of trackable objects in an order provided by the given criteria, and selecting a first parsed trackable object that is supported by the user’s device.

[0009] In another variant, the given criteria includes at least one of: one or more capabilities of the user’s device, an accuracy associated with a type of the trackable objects, a tracking or detection complexity of trackable objects, a memory usage for tracking or detecting trackable objects, a given order according to which the at least two trackable objects are signaled in the set.

[0010] According to another aspect, a method for adding at least one new trackable object to a scene description file in association with at least one anchor object is provided. The method comprises receiving, by a user’s device, a scene description file describing an augmented reality scene, obtaining at least one anchor object from the scene description file, the at least one anchor object being associated with at least one trackable object in the scene description file, responsive to a determination that a type of the at least one new trackable object is not present in the scene description file in association with at least one anchor object, creating the at least one new trackable object in association with the at least one anchor, and providing the at least one new trackable object to one or mode devices.

[0011] According to another aspect, a method for updating a scene description file describing an augmented reality scene is provided. The method comprises receiving, from a user’s device, data relating to the at least one first trackable object associated with at least one anchor object, the at least one anchor object being present in the scene description file in association with at least one second trackable object, signaling, in the scene description file, the at least one first trackable object in association with the at least one anchor object using the received data, and providing the updated scene description file to one or more users’ device.

[0012] According to another aspect, an apparatus comprising one or more processors configured to implement any of the method mentioned above is provided.

[0013] Further embodiments that can be used alone or in combination are described herein. One or more embodiments also provide a computer program comprising instructions which when executed by one or more processors cause the one or more processors to perform the methods according to any of the embodiments described herein. One or more of the present embodiments also provide a non-transitory computer readable medium and / or a computer readable storage medium having stored thereon instructions for implementing steps of the methods described herein.

[0014] One or more embodiments also provide a computer readable storage medium having stored thereon a bitstream generated according to the methods described herein. One or more embodiments also provide a method and apparatus for transmitting or receiving the bitstream generated according to the methods described above.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG. 1 illustrates a block diagram of a system within which aspects of the present embodiments may be implemented.

[0017] FIG. 2 illustrates a block diagram of an example of an augmented reality system in which various aspects and embodiments are implemented.

[0018] FIG. 3 illustrates an example of a method for generating a scene description for an augmented reality scene according to one embodiment.

[0019] FIG. 4 illustrates an example of a method for setting an anchor in an augmented reality scene according to one embodiment.

[0020] FIG. 5 illustrates an example of a method for adding trackables to a scene description for an augmented reality scene according to one embodiment.

[0021] FIG. 6 illustrates an example of a method for updating a scene description for an augmented reality scene according to one embodiment.

[0022] FIG. 7 illustrates an example of spatial relationships between trackable, AR anchor and virtual asset.

[0023] FIG. 8 shows examples of trackable objects.

[0024] FIG. 9 illustrates an example of a marker-based AR scene visualization (image courtesy of Ikea).

[0025] FIG. 10 shows an example of a geospatial AR scene visualization, from https: / / martech.org.

[0026] FIG. 1 1 shows an example of an AR scene visualization that uses multiple markers, from https: / / stemkoski.github.io / AR.js-examples / index.html.

[0027] FIG. 12 shows an example of an AR scene with multiple trackables associated with anchors, from [ETSIJ.

[0028] FIG. 13 illustrates an example of feature points for a feature-based trackable.

[0029] FIG. 14 illustrates an example of geo spatial anchor, from Google.

[0030] FIG. 15 illustrates an example of a reference scene according to one embodiment. FIG. 16 illustrates an example of transforms associated to a set of trackables according to one embodiment.

[0031] FIG. 17 illustrates an example of the reference scene after selecting of a supported trackable, according to one embodiment.

[0032] FIG. 18 shows an example of a reference space for 2D marker.

[0033] FIG. 19 shows an example of a method for setting an anchor according to another embodiment.

[0034] FIG. 20 shows an example of transforms and trackables set associated with an anchor at runtime, according to an embodiment.

[0035] FIG. 21 illustrates an example of a method for adding new trackables to a scene description according to another example.

[0036] FIG. 22 illustrates a block diagram of a system within which aspects of the present embodiments may be implemented, according to another embodiment.

[0037] FIG. 23 shows two remote devices communicating over a communication network in accordance with an example of the present principles.

[0038] FIG. 24 shows the syntax of a signal in accordance with an example of the present principles.

[0039] DETAILED DESCRIPTION

[0040] This application describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the application or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.

[0041] The aspects described and contemplated in this application can be implemented in many different forms. Figs 1 and 22 below provide some embodiments, but other embodiments are contemplated and the discussion of Figs 1 and 22 does not limit the breadth of the implementations. At least one of the aspects generally relates to AR scene visualization, and at least one other aspect generally relates to transmitting data representative of the AR scene. These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for implementing any of the methods described, and / or a computer readable storage medium having stored thereon the data representative of the AR scene generated according to any of the methods described.

[0042] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0043] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singly or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 100 is configured to implement one or more of the aspects described in this application.

[0044] The system 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 110 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device, and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples.

[0045] System 100 includes an encoder / decoder module 130 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 110 as a combination of hardware and software as known to those skilled in the art. Program code to be loaded onto processor 1 10 or encoder / decoder 130 to perform the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 1 10. In accordance with various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during the performance of the processes described in this application. Such stored items may include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0046] In some embodiments, memory inside of the processor 110 and / or the encoder / decoder module 130 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television.

[0047] The input to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in FIG. 1 , include composite video.

[0048] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various of these functions, including, for example, down converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, down converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements may include inserting elements in between existing elements, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0049] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 110 as necessary. Similarly, aspects of USB or HDMI interface processing may be implemented within separate interface ICs or within processor 1 10 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder / decoder 130 operating in combination with the memory and storage elements to process the data stream as necessary for presentation on an output device.

[0050] Various elements of system 100 may be provided within an integrated housing, Within the integrated housing, the various elements may be interconnected and transmit data therebetween using suitable connection arrangement 115, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0051] The system 100 includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network card and the communication channel 190 may be implemented, for example, within a wired and / or a wireless medium.

[0052] Data is streamed to the system 100, in various embodiments, using a Wi-Fi network such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received over the communications channel 190 and the communications interface 150 which are adapted for Wi-Fi communications. The communications channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 100 using a set-top box that delivers the data over the HDMI connection of the input block 105. Still other embodiments provide streamed data to the system 100 using the RF connection of the input block 105. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.

[0053] The system 100 may provide an output signal to various output devices, including a display 165, speakers 175, and other peripheral devices 185. The display 165 of various embodiments includes one or more of, for example, a touchscreen display, an organic lightemitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 165 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 185 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide a function based on the output of the system 100. For example, a disk player performs the function of playing the output of the system 100.

[0054] In various embodiments, control signals are communicated between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communications protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices may be connected to system 100 using the communications channel 190 via the communications interface 150. The display 165 and speakers 175 may be integrated in a single unit with the other components of system 100 in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.

[0055] The display 165 and speaker 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display 165 and speakers 175 are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0056] The embodiments can be carried out by computer software implemented by the processor 1 10 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 120 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1 10 can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0057] FIG. 2 illustrates a block diagram of an example of a system in which various aspects and embodiments can be implemented. Such system is designed to allow shared (i.e. collaborative) augmented reality experience which is one of next challenges for AR applications. Multiple users - here Alice, Bob and Charlie - can simultaneously view and interact with virtual objects from their position in an AR scene which is a shared augmented scene taking place in a real-world 3D environment. Modifications in the AR scene can be visible in real time to every user. A digital representation of the AR scene 220 is handled by an AR controller 210 that manages also the coordination of the interactions between users in the virtual environment. To enjoy the AR scene, users will join other users in the shared augmented space using an AR terminal (200A, 200B, 200C). The AR terminal displays the virtual objects of the AR scene superimposed to the view of the real-world environment. To ensure consistent interactions with the AR scene, all AR terminals must be continuously localized in the same world frame coordinate system. AR terminals and the AR controller are coupled together through a communication network 250. This network is preferably wireless to provide mobility to the AR terminals. In a collaborative experience using the system of FIG. 2, the virtual objects are shared between all the users. Each user may use his own AR terminal to display the AR scene.

[0058] For AR applications, the spatial relationships between objects of the scene and objects in the real world have to be described. Such kind of spatial relationships can be performed using AR anchors. Anchor is a reference defined by a pose (position and rotation), and it is associated with one or more trackables. A trackable is a physical reference in the real world from which some features can be extracted and detected by an AR device. Trackables provide a Coordinate Reference System in which the pose of the anchor can be expressed as illustrated in FIG. 7. In FIG. 7, the detection of the trackable triggers to positioning of the anchor having the pose TRS#1 in the trackable coordinate reference system. The detection of the trackable triggers the display of the corresponding virtual content (virtual chest asset associated to the anchor with a pose TRS#2 relatively to the anchor). The virtual content is then positioned relatively to the pose of the trackable. If the trackable moves, then the virtual content will follow it. FIG. 8 illustrates different trackables that can be used in describing AR scene, for example in [ETSI] : ETSI GS ARF 004-2 V1.1. 12021 -08: Augmented Reality Framework Interoperability Requirements for AR components, systems and services, part 2: World storage and AR authoring functions.

[0059] Marker-based AR (Image and Fiducial Trackables) remains the most popular method among app developers. The virtual content is positioned in the real world with respect to a specific image known by the application, called a 2D marker (QR code, natural image, etc.). This is for example used for AR educational books, where virtual 3D objects appear onto pages containing distinctive pictures. The picture is first recognized by the application, then the corresponding 3D content is rendered, with a position which is defined relatively to the image. The image is continuously tracked by the application and the 3D content follows its position. Typically, if the user moves the image around, then the virtual world will “stick” to the real image. An example of a virtual content positioned relatively to a fiducial marker is illustrated in FIG. 9.

[0060] Similarly, 3D markers (Mesh, CAD, Map Trackables) are commonly used in industrial AR applications to position virtual 3D content (assembly instructions for example) relative to a reference object from the real environment (part of an engine for instance). The 3D marker is a 3D model of the reference object, typically a 3D mesh. It is used by the application to detect the presence and the pose of the reference object in the real environment.

[0061] There are many cases, especially in outdoor AR scenarios, where geospatial anchoring is necessary. In this case, the positioning of the scene objects is known in advance in the real world. Geospatial anchoring is also key for outdoor location-based services such as those providing directions or touristic information, or in geocaching applications. In this case, the AR space which is used to position the scene content is an external geospatial coordinate system which is globally defined independently from the runtime application. An example of such a geospatial AR scene visualization is illustrated on FIG. 10.

[0062] It frequently happens that one single virtual object is attached to several independent markers. For instance, if a content producer wants to associate a virtual content (e.g. a 3D horse) to several images (e.g. a set of predefined horse images), then it is required to allow specifying several images as possible 2D markers within the scene description. The object is then positioned in the real world each time one of these images is recognized by the application.

[0063] Furthermore, in an AR context, it is also common to have a single scene containing a collection of several independent virtual objects to be independently positioned in the real-world environment. For example, one might want to place a virtual car to one place, and a virtual house to another place, each being attached to a distinct marker. In this type of scene, the relative position of the scene objects with respect to each other is described using a hierarchy relation which is not meaningful anymore. In another example, let’s consider a scene describing a gaming set made of a virtual board, a set of virtual characters and a set of textual information. When used within an AR application, the board might need to be attached to a table, the 3D characters might need to be attached to specific physical playing cards, and the text information might need to be attached to the headset. To cover this type of usage, each part of the scene needs to specify its own AR space, taking over the predefined transformation hierarchy. An example of an AR scene visualization based on multiple markers respectively positioning independent virtual objects is illustrated on FIG. 1 1 .

[0064] FIG. 12 shows an example of AR Scene with multiples trackables associated with anchors. A world anchor, also named anchor in the present document, represents a fixed position in relation to one or more elements of the real world (trackables). It has a Coordinate Reference System in which AR Assets stay spatially registered. Anchors can be combined in a graph, thus giving the notion of world anchor ([ETSI]). The relative position of AR Assets, Trackables and World Anchors are defined by 3D Transforms. A 3D Transform defines the relative position and orientation between Trackables, World Anchors or AR Assets. An AR Asset shall be attached to one or more World Anchors or Trackables. A World Anchor shall be attached to one or more World Anchors or T rackables. T rackables are detected in the real world through their features. Linking an AR Asset directly to a Trackable is the easiest way to enrich the World Graph as illustrated by the example of FIG. 9. However, when the use case requires more complex augmentations of the world, World Anchors are introduced in the scene description to help organize AR Assets hierarchically.

[0065] In the marker-based example illustrated on FIG. 12 and provided by the document [ETSI], an industrial machine will be augmented in order to provide contextually-relevant information. The user's position relative to the machine is determined by recognition of different 2D markers. The left part of FIG. 12 illustrates the real world, while the right part of FIG. 12 illustrates the world graph (i.e. the virtual scene description). As shown in FIG. 12, in a first step, at authoring time, the AR experience designer defines the places where AR Assets will be attached. These positions will be defined as World Anchors in the World Graph. World Anchors are identified with a unique ID (ID:W-0R, ID:W-0S on FIG. 12).

[0066] In a second step, AR Assets are positioned in the Coordinate Reference System of the World Anchors using respectively a 3D transform 3DT. In the example of FIG. 12, a gauge and a word “online” are positioned as AR assets.

[0067] In a third step, trackables are added in the World Graph and positioned relative to the associated World Anchors using a 3D transform (3DT). In the example of FIG. 12, three 2D- marker trackables are positioned (ID: T-OX, ID: T-OY, ID: T-OZ). Each World Anchor is linked to every trackable the designer decides to involve in the computation of the pose of the associated World Anchor in the real world. The first AR Asset (the gauge) can be triggered only if Trackable T-OX is detected by the AR device. The other AR Asset (the word Online) can be triggered by any of the 3 Trackables or a combination of them detected by the AR device.

[0068] Another kind of trackable that can be used for placing anchor in the AR scene is the feature based trackable for spatial anchors. Spatial anchors work with an image captured by the device's camera, using image features and their relative position in 3D space to recognize a real-world location. In this context, the trackable is a set of features detected in the image captured by the camera. An example of such features is illustrated on FIG. 13. When the set of features is detected, a corresponding anchor is created in the AR space at the corresponding location.

[0069] Geospatial trackables are a further type of trackables that allow to place virtual content in the real world. An example of an AR scene using geospatial anchors is illustrated on FIG. 14.

[0070] In order for any AR devices to be part of a shared AR experience and to provide interoperability between AR devices, a scene description format has been standardized Information technology - Coded representation of immersive media - part 14: Scene description, FDIS) to describe an AR scene. In the context, the MPEG_anchors extension (provided in [Amd2] Potential improvements of ISO / IEC 23090-14 CDAM 2: Support for Haptics, Augmented Reality, Avatars, Interactivity, MPEG-I Audio, and Lighting N00861), introduces the trackable and MPEG anchor entities.

[0071] The MPEG-anchors extension defines the followings list of trackables:

[0072] Table 8.1-4- Definition of the Trackable type

[0073] Table 1 : MPEG SD trackables type

[0074] The semantics of a trackable are provided below:

[0075] Table 8.1-3- Definition of the Trackable object

[0076] TRACKABLE APPLICATION corresponds to features based trackables. As feature based trackables are attached to device, such trackables are associated to vendor extension. As seen above, anchoring is very useful in AR experience, this allows to obtain a locally stable virtual scene. Persistent anchor (i.e. the anchor is stored and reused later) can be used by AR devices which have close technical specifications. However, depending on the type of trackable 2D or 3D marker, feature-based trackable, ...), the positioning accuracy is not the same. Two users of a shared AR scene that use different trackables for placing a same virtual object using an anchor associated to each one of the trackables can see a same object at locations slightly different due to a lack in accuracy of the positioning, thus leading to a bad AR shared experience.

[0077] Basically, an anchor type is bound to a framework, this is especially true for spatial anchors. Spatial anchors are vendor dependent, that is they are defined and implemented by the app provisioner or the device maker.

[0078] In a shared experience (with multiple AR devices), each AR device involved in the shared AR scene receives the scene description file. The information included in the scene description file is used for anchoring the AR device to the shared AR scene. In the general case, it is assumed that any device is able to handle every trackable type specified in the scene description file while this is not the case.

[0079] More specifically, when spatial anchor is used, that is the anchor is associated to a feature based trackable, if the AR device is not from a same vendor as the one that has defined the feature-based trackable that is used for anchoring, the AR device will not be able to handle correctly the feature-based trackable and thus won’t be able to anchor to the shared AR scene or to locate correctly the virtual asset.

[0080] Also, even when the AR device manages to handle the feature-based trackable associated with the anchor, it may happen that the AR device does not handle the trackable in a similar way as the AR device of the vendor that has specified the trackable. Thus, the anchoring may not be sufficiently accurate, thus yielding to poor AR experience for the user of the AR device. The inaccurate anchoring of one AR device involved in an AR shared scene can even impact the AR experience of other users involved in the same shared AR scene through the users’ interaction.

[0081] Some embodiments provide for enhancing interoperability of AR devices involved in a shared AR scene.

[0082] According to an embodiment, when building a reference scene, a user or authoring user defines in the scene description file an anchor attached to a set of different trackables. This would enhance the interoperability of a given AR shared experience between any types of AR devices. In the set, the Trackables are sorted according to a given accuracy criteria. For example, an ordered list based on accuracy of the trackable is based on the type of trackables. The more accurate type of trackable is the feature-based trackable. The ordered list based on accuracy can be: feature-based trackable, Marker-2D (e.g image), Marker-3D, plane. By accuracy, it should be understood the positioning accuracy of the anchor when using the trackable. If known, the transform between the anchor and the trackable is also transmitted in the description file. Each trackable associated with the anchor has its own transform for placing the anchor.

[0083] When the scene description is received by a user’s device participating to the shared AR experience, the set of trackables associated to an anchor is parsed and a first trackable supported by the AR device is selected. The set of trackables is parsed according to a given order which can be the one according to which the trackables are signaled in the set or another order. This embodiment provides alternatives trackables to the AR device that can used for anchoring when for example a feature-based trackable is not supported by the AR device.

[0084] FIG. 3 illustrates a method 300 for generating a scene description file describing an augmented reality scene according to this embodiment. At 310, a set of at least two trackable objects is signaled in the scene description file, in associated with at least one anchor object. In the present document, trackable and trackable object refers to the same entity “trackable” as defined above. Anchor object refers to the same entity “anchor” or “world anchor” as defined above.

[0085] In the embodiment mentioned above, the at least two trackable objects are signaled in the set according to a given order. Then, at 320, the scene description file is transmitted to one or more devices for participating in a shared AR scene.

[0086] FIG. 15 shows an example of an AR reference scene wherein the anchor is associated with set of possible trackables: Anchor with spatial features (F), Anchor with trackable image (Image), Anchor with trackable plane (Plane).

[0087] In a variant, a transform associated respectively to each one of said at least two trackable objects of the set is signaled in the scene description file.

[0088] For a spatial anchor (the anchor associated with the set of spatial features F), there is not necessary a specific transform between the set of features (F) and the anchor. By default, one could be provided as identity. This allows to maintain consistency between trackables and anchors. For spatial anchor, anchor and trackable are often superimposed.

[0089] For the remaining part of the possibles trackables (Marker 2D, marker 3D, Plane, ....), a transform with respect to the anchor is defined. In the example of the AR scene of FIG. 15, example of transforms associated with the above cited trackables are shown on FIG. 16.

[0090] The given order in which the trackable objects are signaled in the set can be based on an accuracy associated to a type of the trackable objects, or a complexity for tracking or detecting the type of trackable objects or a memory usage for tracking or detecting the trackable objects.

[0091] FIG. 4 illustrates an example of a method 400 for setting an augmented reality scene using at least one anchor. At 410, a scene description file describing the augmented reality scene is received by a user’s device. For example, the scene description file is generated according to the method as described in FIG. 3. The scene description file received by the user’s AR device comprises for at least one anchor a list or a set of trackables associated to the anchor in which it can select one trackable for anchoring. The list can be an ordered list, for example ordered according to positioning accuracy of the trackables. In another variant, the list is not ordered but the user’s AR device parses the trackables of the list according to a given criteria. At 420, for at least one anchor object in the scene description file, the user’s device selects a trackable object in a set of at least two trackable objects associated with the at least one anchor object in the scene description file. The user’s device selects the first trackable supported by the user’s device in the set. In this embodiment, as said above, the set of trackable objects is parsed in an order defined according to a given criteria. The given criteria can be the same as the one used in Method 300 for signaling the trackable objects or another criteria. In an example, the given criteria is based on the type of the trackables and takes into account the positioning accuracy associated to each type of the trackables At 430, the user’s device sets the at least one anchor using the selected trackable object.

[0092] FIG. 17 shows an example of the AR scene set using the received scene description after the selection of the first supported trackable. In the example, the first supported trackable is the image / marker 2D.

[0093] In another embodiment, a method for adding at least one new trackable object to a scene description file in association with at least one anchor object is provided. In the embodiment, the anchor object can be defined in the scene description file in association with a single trackable or with multiple trackables. An aim of this embodiment is to enrich by the user’s device the set of trackables associated to an anchor object. An example of such a method 500 is illustrated on FIG. 5. At 510, a scene description file describing an augmented reality scene is received by a user’s device. At 520, at least one anchor object is obtained from the scene description file. The at least one anchor object is associated with at least one trackable object in the scene description file. The user’s device sets the AR scene using the at least one anchor object. Then, at 530, a new trackable object is obtained by the user’s device. The new trackable is associated with the at least one anchor object. At 540, it is determined whether the new trackable object is present or not in the scene description file in association with at least one anchor object. In a variant, it is determined whether a type of the new trackable object is present or not in the scene description file in association with the at least one anchor object.

[0094] At 550, if the new trackable object or if the type of the new trackable object is not present, the new trackable object is added in the scene description file in association with the at least one anchor. This is done for example by sending the corresponding data to the server that sent the scene description file. In this way, the new trackable object can be provided to one or more AR devices sharing the AR scene.

[0095] Correlatively, a method for updating a scene description file describing an augmented reality scene is provided. The method can be performed by the server that originally generated or provided the scene description file or by a user’s device if the scene description file has been generated by a user’s device. An example of the method 600 for updating a scene description file is illustrated on FIG. 6. The scene description file comprises at least one anchor object in association with at least one trackable object. For example, the scene description file could have been generated according to the method illustrated on FIG. 3, but any other method could also be used. At 610, data relating to at least one other trackable object associated with the at least one anchor object is received from a user’s device. At 620, the at least one other trackable object is signaled in the scene description file in association with the at least one anchor object using the received data, and at 630, the updated scene description file is provided to one or more users’ device.

[0096] Further details on the above given embodiments are provided below. As explained above, in a first step, the virtual scene is created by a scene creator followed by the formatting of the virtual scene in a scene description file. In a second step, the scene description is provided to one or more AR devices. In this second step, the scene description file is received, parsed and a first supported trackable is selected among the set of defined trackables for a given anchor.

[0097] The creation of the reference scene or the AR scene is further described below, it comprises the implementation of the trackable set according to an embodiment. The content creator is responsible for creating the initial reference scene. For each trackable to use in the AR scene, two scenarios arise: either the trackable exists, or it must be created from existing features. For each new trackable type that is added, some parameters should be specified to allow precisely placing the anchor at the suggested place. The AR scene is transmitted in a format that will allow users to use it. In the following, MPEG_SD file format is used as support for transmitting the AR scene.

[0098] The set of trackables (spatial, marker 3D, marker 2D, plane ....) used for a given anchor is defined in a common reference space. This is required to be able to establish the different transform between trackables and anchor.

[0099] Regarding the transformation between the trackable and the anchor, local reference space (the reference space attached to the trackable) must be known to all users.

[0100] The document Extension MPEG_anchor of gITF format (N00861) defines the local reference space for trackables. Each trackable provides a local reference space in which an anchor pose can be expressed. This local reference space is right-handed and depends on the type of trackable. For example, reference space for Marker 2D is illustrated on FIG. 18.

[0101] The virtual scene is described thanks to a scene description file such that it can then be exported / transmitted to AR devices of other users.

[0102] If the trackable is already present in the AR scene but not defined as such, the use of the trackable for the given anchor in the scene description file only needs a formatting to add it to the scene description file.

[0103] If the type of trackable is not spatial, a transform between the trackable and the anchor is computed. This transform is transmitted as metadata and then added to the scene description file.

[0104] Examples of action to take for adding a trackable depends on the type of the trackable.

[0105] For feature-based trackable (spatial anchor), the trackable is a set of features. This type of trackable depends on the AR devices vendor or maker. The set of features and how to detect them depends on the vendor of the AR device. An identity transform is defined between the trackable and the anchor. The type of spatial anchor can be added as metadata. For spatial anchor, the type depends on the vendor (Apple, Microsoft, Google ). Generally, a spatial trackable defined by one vendor of an AR device is not understood by an AR device from another vendor.

[0106] For Marker 2D trackable, the trackable can be an image. To be tracked, a printed version of this image must be present in the real scene. If a representation of the real image is available in the gITF graph, only the transform between the trackable and the anchor must be computed, the size of the image is included in the mesh parameters. In gITF format, a node refers a mesh, the mesh refers a material, the material refers a texture. Pixels of the image are stored in a buffer.

[0107] If a representation of the real image is not available, the application may build it. For that, a mesh is created from the real image by selecting 4 corners of the image (for example using ray casting mechanism). The texture can be extracted from a cropped snapshot of the real scene (ideally taken vertically to avoid deformation). The transform between the trackable and the anchor must be computed, the size of the image is included in the mesh parameters.

[0108] For a Marker 3D trackable, the trackable can be an any 3D real object. To be tracked, the object must be present in the real scene. If a representation of the real object is available, only the transform between the trackable and the anchor must be computed, 3D data are included in the mesh parameters. In gITF format, a node refers a mesh, the mesh refers a material, the material refers a texture. Vertex and texture are stored in a buffer. If a representation of the real object is not available, it’s possible to build it from a scan. Recent AR Devices allows this (for example, iPhone 14 with ARKit framework). The transform between the trackable and the anchor must be computed. 3D data are included in the mesh parameters.

[0109] For a Plane trackable, the trackable can be the floor, a wall, the ceiling. To be tracked, the plane object must be visible in the scene. The plane is extracted by the framework .

[0110] (Microsoft, Apple, Google) used by the AR device. The transform between the trackable and the anchor must be computed. 3D data are included in the mesh parameters.

[0111] When the set of trackables is defined, all the data is included in a formatted file (scene description file) to be transmitted to AR devices.

[0112] The user that has created a scene description file allows an export to a server to share the scene description file with other AR devices.

[0113] As part of a scene description, parameters may be added in a standard specification, as shown in the following example based on MPEG-I Scene Description. MPEG-I Scene Description is a framework Information technology - Coded representation of immersive media - part 14: Scene description, FDIS) that uses the Khronos gITF extension mechanism to support additional scene description features. The Amd2 document contains an extension relative to anchor “MPEG_anchor”. Embodiments provided herein extend the existing “MPEG_anchor” extension.

[0114] At gITF level, there is a set of trackables, so trackables can be added in it.

[0115] In the MPEG_anchor_extension, an anchor refers a single trackable ID. According to an embodiment provided herein, the trackable ID is replaced by a set or an array of trackable IDs. In other words, instead of signaling one indicator of a trackable per anchor (i.e. one trackable ID per anchor), multiple indicators (trackable IDs) are signaled per anchor. In this way, an AR device receiving the Scene Description File can have alternative trackables for a given anchor when one of the trackable referred by the anchor is not supported by the AR device.

[0116] In a variant, the order of the elements in the trackable array associated to a given anchor gives the order in which the trackables must be considered by the AR devices.

[0117] As said above, a transform (TRS) must be associated to each trackable to retrieve the anchor position. In a variant, a set of transforms (TRS) is added at the gITF level and a mapping property is added in the anchor element at the gITF level.

[0118] The syntax and semantics of the MPEG_anchor extension at the gITF level are provided below in Table 8.1.1 from Amd2 which is modified according to the variants provided herein. Modifications are shown underlined. Table 8.1 -1 - Common objects of the MPEG_anchor extensions provided at the gITF file level

[0119] The syntax and semantics of the Anchor object are provided below in Table 8.1.2 from Amd2 wherein modifications are shown underlined. Table 8.1 -2 - Definition of the Anchor object

[0120] Below is provided an example of the scene description file according to an embodiment wherein addition / modification with respect to the state of the art of the gITF file are shown underlined: "extensions": {

[0121] "MPEG_anchor": { "anchors": [

[0122] { "trackables": [0,1 ,21, " transforms": [0,1 ,21,

[0123] "requiresAnchoring": true, "actions":[0]

[0124] "trackables": [

[0125] {

[0126] "type": 7

[0127] },

[0128] {

[0129] "type": 4,

[0130] " markerNode ":4

[0131] },

[0132] {

[0133] "type": 0

[0134] “transforms”: [

[0135] {

[0136] [1 .0,0.0, 0.0, 0.0, 0.0,1 .0,0.0, 0.0, 0.0, 0.0,1 .0,0.0,0.0,0.0,0.0,1.01

[0137] },

[0138] {

[0139] [1 .0,0.0, 0.0, 0.5, 0.0,1 .0,0.0, 0.4, 0.0, 0.0,1 .0,0.7, 0.0, 0.0, 0.0,1 .0]

[0140] },

[0141] {

[0142] [ 1 .0,0.0, 0.0, 0.7, 0.0,1 .0,0.0, 0.8, 0.0, 0.0,1 .0,0.9, 0.0, 0.0, 0.0,1 .0]

[0143] } },

[0144] FIG. 19 illustrates an embodiment of a method 1900 performed at the receiver side by an AR device of a user participating to the shared experience. At 1910, a scene description file is received by the AR device. In the scene description file, one anchor is attached to a set of trackables, eventually sorted according to a given criteria.

[0145] At 1920, the AR device enumerates its capabilities, that is it checks locally what is supported by the device. For example, if the set of trackables associated to an anchor is the one as illustrated on FIG. 16, the AR device determines whether or not it supports the feature-based trackable signaled in the set, whether or not it supports the image trackable and whether or not it supports the plane trackable.

[0146] At 1930, the AR device selects the first supported trackable from the set. For instance, if the AR device supports the feature-based trackable, then this trackable is selected for setting the anchor. If the feature-based trackable is not supported, then the AR device selects the image trackable if the AR device supports the image trackable, otherwise the plane trackable is selected if supported by the AR device. If none of the trackables is supported by the AR device, then the process ends for the AR device or another anchor is taken from the scene description file.

[0147] At 1940, the AR device then locate the selected trackable in the real world. In some variant, detecting the trackable can be done in parallel of selecting the trackable can conditioned the selection of the trackable. For instance, at 1930 the first supported trackable that is also detected by the device is selected.

[0148] At 1950, once the selected trackable is detected, the transform associated to the selected trackable is applied to retrieve the anchor position. At 1960, the anchor is set in the AR scene at the position determined at 1950, any action associated with the anchor is triggered, for example displaying a virtual asset in the AR scene.

[0149] In a variant, the AR scene and its description can evolve at runtime and be updated. More particularly, if the AR application in which an AR device has an uplink capability, the trackable part and the anchor part of the AR scene can be enriched.

[0150] As described above, the trackable selected by the AR device depends on its type and on the capability of the AR device.

[0151] For example, in the example explained with FIG. 15-17, the image trackable is selected by the AR device, because the feature-based trackable provided in the scene description file is not supported by the AR device. Then, a fallback trackable has been selected by the AR device. In the variant provided herein, one or more new trackables are determined by the AR device to set the anchor and provided to the server for sharing with other devices involved in the AR scene. For example, the AR device used for the example of FIG. 15-17 does not support the feature-based trackable provided in the scene description file because it was defined by a different vendor of the AR device, but the AR device supports feature-based trackable from its own vendor. The AR device can thus determine a new feature-based trackable that can be added to the list of trackables in the scene description file.

[0152] In another example, the AR device may have selected a feature-based trackable, but an image trackable is not associated to the anchor in the scene description file. The AR device can thus determine a new image trackable that can be added to the list of trackables in the scene description file.

[0153] In some variants, whatever the AR device has selected to set the anchor, the AR device can determine a new trackable to enrich the other user’s experience when using the scene description file for the AR scene.

[0154] When adding a new trackable to the scene description file, the transform’s array and parameter trackables and transforms of anchor must be updated too.

[0155] In these variants, the growing set of trackables may leads to a universal system allowing shared experience with many devices.

[0156] FIG. 20 illustrates an example of enriching the set of trackables for a given anchor from FIG. 18. In the example of FIG. 17, the AR device selected the image trackable associated with the given anchor and transform T 1 . The AR device then determines a new feature-based trackable “Spatial (S2)” on FIG. 20 associated with the given anchor and transform Ti. Note that here for spatial anchor, the transform Ti is the same than the transform TO (identity transform). Transforms associated to feature based trackable are necessary identical.

[0157] Feature-based trackables are associated with vendor extension. If a new feature-based trackable is added to the scene description file, a parameter to identify the vendor is useful.

[0158] In the extension MPEG_anchor, a parameter “trackablejd” of type string is already defined in TRACKABLE APPLICATION object. The “trackablejd” can be an URN that uniquely identifies the trackable. An example of URN is given as below:

[0159] “urn:mpeg:sd:2023:anchor:vendor_name:identifier / xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”.

[0160] Other parameters can be added to TRACKABLE APPLICATION object to define a vendor ID. The AR device can check if the trackable identified by the “trackablejd” is supported.

[0161] FIG. 21 illustrates a method 2100 for adding a new trackable associated with a given anchor in the AR scene. At 21 10, the AR device parses the set of trackables associated with a given anchor. In a variant, the set of trackables can comprises only a single trackable. If the first trackable of the set is supported, then the AR device processes to step 2150 and 2160. Otherwise, at 2120, the AR device checks if the next trackable in the set is supported. If yes, the AR device processes to step 2150 and 2160, otherwise it is checked whether it is the end of the trackable list. If this is the case, the anchor cannot be placed and the error is managed by the application (2140). Otherwise, the process loops on the set of trackables (2120). At 2150, the AR device sets the anchor using the selected trackable. At 2160, the AR device checks whether feature-based trackable is supported by the AR device. If not, the AR experience continues (2180). Otherwise, the AR device determines whether a type of featurebased trackable is present in the received scene description file. In a variant, it is checked whether feature-based trackable defined by the same vendor as the AR device is present in the received scene description file. If yes, the AR experience continues (2180). Otherwise, the AR device determines a new feature-based trackable at 2190. For that, one or more parameters are determined for locating the new trackable object in the real world as well as a transform for locating the given anchor in the local reference space of the new trackable. The pose TAWCSof the anchor in the world coordinate space is known by the application. The pose TnewTwCsof the new trackable in the world coordinate space is known by the application. The transform TnewTIRsbetween the new trackable and the local reference space of the new trackable is known. The final transform TAI RSfor the anchor in the local reference space of the new trackable is:

[0162] TAWCStransform: Anchor / world coordinate space.

[0163] T . IRS_ T JRs * T „ newT * T .wCs I A — I newT I wCs I A

[0164] The AR device saves the data relating to the new trackable: the anchor to which it is associated, spatial features of the trackable, specific vendor ID associated to the feature-based trackable and the transform Tx to place the anchor in the new trackable coordinates system. At 2192, the data is uploaded to the server so that the server updates the scene description file by adding the new trackable in the set of trackables associated to the given anchor.

[0165] In FIG. 21 , a feature-based trackable is added to the set of trackable attached to a given anchor. In a variant, the new trackable is an image trackable. For instance, at 2160 and 2170, it is checked whether or not image trackable is supported by the AR device and whether or not an image trackable is already associated to the given anchor. If the image trackable is supported by the AR device and no image trackable is associated with the given anchor, the AR device creates the new trackable at 2190 as a 2D marker in a similar manner as described above for 2D marker.

[0166] The addition of new trackables can be proposed to a user of an AR device participating in a shared AR scene via a user interface. FIG. 22 illustrates a block diagram of a system within which aspects of the present embodiments may be implemented, according to another embodiment. FIG. 22 shows one embodiment of an apparatus 2200 for generating a scene description file or for receiving a scene description file according to any one of the embodiments described herein. The apparatus comprises Processor 2210 and can be interconnected to a memory 2220 through at least one port. Both Processor 2210 and memory 2220 can also have one or more additional interconnections to external connections.

[0167] Processor 2210 is also configured to receive, by a user’s device, a scene description file describing the augmented reality scene, for at least one anchor object in the scene description file, select a trackable object in a set of at least two trackable objects associated with the at least one anchor object in the scene description file, wherein the selecting is based a given criteria, and set the at least one anchor using the selected trackable object, using any one of the embodiments described herein. For instance, the processor 2210 uses a computer program product comprising code instructions that implements any one of embodiments described herein.

[0168] In another embodiment, processor 2210 is configured to signal in the scene description file, in associated with at least one anchor object, a set of at least two trackable objects, wherein the at least two trackable objects are signaled in the set according to a given order, and transmit the scene description file to one or more devices, using any one of the embodiments described herein. For instance, the processor 2210 uses a computer program product comprising code instructions that implements any one of embodiments described herein.

[0169] In another embodiment, processor 2210 is configured for adding at least one new trackable object to a scene description file in association with at least one anchor object. For example the processor 2210 is configured to receive, by a user’s device, a scene description file describing an augmented reality scene, obtain at least one anchor object from the scene description file, the at least one anchor object being associated with at least one trackable object in the scene description file, responsive to a determination that a type of the at least one new trackable object is not present in the scene description file in association with at least one anchor object, create the at least one new trackable object in association with the at least one anchor, and provide the at least one new trackable object to one or mode devices, using any one of the embodiments described herein. For instance, the processor 2210 uses a computer program product comprising code instructions that implements any one of embodiments described herein.

[0170] In an embodiment, illustrated in FIG. 23, in a transmission context between two remote devices A and B over a communication network NET, the device A comprises a processor in relation with memory RAM and ROM which are configured to implement a method for generating or updating a scene description file, as described with FIG. 1 -21 and the device B comprises a processor in relation with memory RAM and ROM which are configured to implement a method for receiving a scene description file and setting an AR scene or adding a new trackable to an AR scene as described in relation with FIG 1 -21 . In accordance with an example, the network is a broadcast network, adapted to broadcast / transmit data relating to AR scene from device A to AR devices including the device B and to transmit data relating to AR scene from AR device B to device A.

[0171] FIG. 24 shows an example of the syntax of a signal transmitted over a packet-based transmission protocol. Each transmitted packet P comprises a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may comprise AR scene data generated according to any one of the embodiments described above. In a variant, the payload can comprise any signaling relating to a new trackable added to an AR scene as described above.

[0172] Various implementations involve decoding. “Decoding”, as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this application, for example, entropy decoding a sequence of binary symbols to reconstruct image or video data.

[0173] As further examples, in one embodiment “decoding” refers only to entropy decoding, in another embodiment “decoding” refers only to differential decoding, and in another embodiment “decoding” refers to a combination of entropy decoding and differential decoding, and in another embodiment “decoding” refers to the whole reconstructing picture process including entropy decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0174] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application can encompass all or part of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this application, for example, determining re-sampling filter coefficients, resampling a decoded picture.

[0175] As further examples, in one embodiment “encoding” refers only to entropy encoding, in another embodiment “encoding” refers only to differential encoding, and in another embodiment “encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase “encoding process” is intended to refer specifically to a subset of operations or generally to the broader encoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0176] Note that the syntax elements as used herein, are descriptive terms. As such, they do not preclude the use of other syntax element names.

[0177] This disclosure has described various pieces of information, such as for example syntax, that can be transmitted or stored, for example. This information can be packaged or arranged in a variety of manners, including for example manners common in video standards such as putting the information into an SPS, a PPS, a NAL unit, a header (for example, a NAL unit header, picture header or a slice header), or an SEI message. Other manners are also available, including for example manners common for system level or application level standards such as putting the information into one or more of the following: a. SDP (session description protocol), a format for describing multimedia communication sessions for the purposes of session announcement and session invitation, for example as described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transmission. b. DASH MPD (Media Presentation Description) Descriptors, for example as used in DASH and transmitted over HTTP, a Descriptor is associated to a Representation or collection of Representations to provide additional characteristic to the content Representation. c. RTP header extensions, for example as used during RTP streaming. d. ISO Base Media File Format, for example as used in OMAF and using boxes which are object-oriented building blocks defined by a unique type identifier and length also known as 'atoms' in some specifications. e. HLS (HTTP live Streaming) manifest transmitted over HTTP. A manifest can be associated, for example, to a version or collection of versions of a content to provide characteristics of the version or collection of versions.

[0178] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0179] Some embodiments refer to rate distortion optimization. In particular, during the encoding process, the balance or trade-off between the rate and distortion is usually considered, often given the constraints of computational complexity. The rate distortion optimization is usually formulated as minimizing a rate distortion function, which is a weighted sum of the rate and of the distortion. There are different approaches to solve the rate distortion optimization problem. For example, the approaches may be based on an extensive testing of all encoding options, including all considered modes or coding parameters values, with a complete evaluation of their coding cost and related distortion of the reconstructed signal after coding and decoding. Faster approaches may also be used, to save encoding complexity, in particular with computation of an approximated distortion based on the prediction or the prediction residual signal, not the reconstructed one. Mix of these two approaches can also be used, such as by using an approximated distortion for only some of the possible encoding options, and a complete distortion for other encoding options. Other approaches only evaluate a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and related distortion.

[0180] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0181] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0182] Additionally, this application may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory. Further, this application may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0183] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0184] It is to be appreciated that the use of any of the following 7”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0185] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0186] As will be evident to one of ordinary skill in the art, implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor- readable medium.

[0187] A number of embodiments has been described above. Features of these embodiments can be provided alone or in any combination, across various claim categories and types.

Claims

CLAIMS1 . A method for generating a scene description file describing an augmented reality scene, comprising: signaling in the scene description file, in association with at least one anchor object, a set of at least two trackable objects, wherein the at least two trackable objects are signaled in the set according to a given order that depends on one of an accuracy associated with a type of trackable objects, a tracking or detection complexity of trackable objects, a memory usage for tracking or detecting trackable objects, transmitting the scene description file to one or more devices.

2. The method of claim 1 , wherein each one of the at least two trackable objects is signaled using an indicator identifying a trackable object in an array of trackable objects.

3. The method of claim 1 or 2, further comprising signaling for at least one of said at least two trackable objects of the set, a transform associated to said one of said at least two trackable objects of the set, the transform indicating a pose of the at least one anchor object in a local coordinate system of said one of said at least two trackable objects of the set.

4. The method of claim 3, wherein the transform is signaled using a transform index indicating one transform in an array of transforms signaled in the scene description file.

5. A method for setting at least one anchor object in an augmented reality scene comprising:Receiving, by a user’s device, a scene description file describing the augmented reality scene, for at least one anchor object in the scene description file, selecting a trackable object in a set of at least two trackable objects associated with the at least one anchor object in the scene description file, wherein the selecting is based on a given criteria, the given criteria including at least one of: one or more capabilities of the user’s device, an accuracy associated with a type of the trackable objects, a tracking or detection complexity of trackable objects, a memory usage for tracking or detecting trackable objects, a given order according to which the at least two trackable objects are signaled in the set, setting the at least one anchor using the selected trackable object.

6. The method of claim 5, wherein selecting the trackable object in the set comprises:Parsing the set of trackable objects in an order provided by the given criteria, Selecting a first parsed trackable object that is supported by the user’s device.

7. The method of any of claims 5-6, further comprising: creating at least one new trackable object in association with the at least one anchor in the augmented reality scene, updating the scene description file to add the at least one new trackable object .

8. The method of claim 7, wherein creating the at least one new trackable object is responsive to a determination that the at least one new trackable object is not present in the set of at least two trackable objects associated with the at least one anchor in the scene description file.

9. A method for adding at least one new trackable object to a scene description file in association with at least one anchor object, the method comprising: receiving, by a user’s device, a scene description file describing an augmented reality scene, obtaining at least one anchor object from the scene description file, the at least one anchor object being associated with at least one trackable object in the scene description file, responsive to a determination that a type of the at least one new trackable object is not present in the scene description file in association with the at least one anchor object, creating the at least one new trackable object in association with the at least one anchor, updating the scene description file by adding the at least one new trackable object.

10. The method of any of claims 7-9, wherein the at least one new trackable object is one of a feature-based trackable object, or an image.1 1 . The method of any one of claims 7-10, wherein creating the at least one new trackable object in association with the at least one anchor in the augmented reality scene comprises: obtaining one or more parameters for locating the at least one new trackable object, obtaining a transform for locating the at least one anchor object in a local reference system of the at least one new trackable object.

12. The method of any one of claims 7-11 , wherein if the at least one new trackable object is a feature-based trackable object, creating the at least one new trackable object comprises associating a vendor identifier to the at least one new trackable object.

13. A method for updating a scene description file describing an augmented reality scene, the method comprising: receiving, from a user’s device, data relating to at least one first trackable object associated with at least one anchor object, the at least one anchor object being present in the scene description file in association with at least one second trackable object, signaling, in the scene description file, the at least one first trackable object in association with the at least one anchor object using the received data, providing the updated scene description file to one or more users’ device.

14. The method of claim 13, wherein the at least one first trackable object and the at least one second trackable object are signaled in association with the at least one anchor object, using an indicator identifying a trackable object in an array of trackable objects.

15. The method of any one of claims 1 -4, further comprising updating the scene description file according to claim 13.

16. An apparatus comprising one or more processors configured to implement the method of any one of claims 1 -15.

17. The apparatus of claim 16 wherein the apparatus is an augmented reality device.

18. A computer program comprising program code instructions for implementing the method according to any one of claims 1 -15, when executed by a processor.

19. A non-transitory computer readable medium comprising program code instructions for implementing the steps of the method according to any one of claims 1 -15, when executed by a processor.

Images