Management of real-world data in extended reality applications

Abstract

A mechanism is proposed to process scan data (3D model) information from a user's real environment for augmented or enhanced reality applications. The mechanism relies on a configuration step. Then, the transmission of scan data or raw data from a user equipment to a server over a network is described in detail. The architecture of the user equipment can be a lightweight extended reality (XR) UE that needs to delegate the extraction of a 3D model of a real scene and the segmentation of this model to a server, an advanced user equipment that is able to extract a 3D model but requires to delegate the segmentation task to a server, or a high-end user equipment that is able to perform the extraction of a 3D model locally for independent use.

Description

1. TECHNICAL FIELD The present principles generally relate to the field of managing scan data representative of a real environment of an extended reality scene. The transmission of scan data or raw data from a user equipment (UE) to a server over a network is detailed. In particular, the present principles relate to UE configuration and architecture. 2. BACKGROUND This section is intended to introduce the reader to various aspects of art that can be related to various aspects of the present principles described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] In the field of extended reality (XR) applications, knowledge of the real world can be needed for different purposes. For example, positioning of the XR rendering device, or for seamlessly inserting virtual content into the user's real environment. Such knowledge about the real world can include the position of trackable items to place virtual content correctly according to the user(s), and a 3D representation of the surrounding environment (point cloud, mesh, semantics) to ensure proper interaction (occlusion, collision, physics) between virtual and real content. This knowledge of the real world, also called scene understanding, can be acquired in real-time by an XR user equipment (UE), for example, through an XR runtime module that collects user environment data through sensors and processes them to build a representation of the real world (e.g. a 3D scene graph). A typical UE (e.g. XR UE) can be a head-mounted display (HMD), optical see-through glasses and video see-through HMDs for augmented reality (AR) and mixed reality (MR), mobile devices with position tracking and cameras, wearable devices, etc. Moreover, several different types of XR UEs can be envisaged based on the XR device functionalities to be provided by one or more devices, wearable devices, actuators, controllers and / or accessories, such as displays, cameras, sensors, sensor processing, wireless connectivity, XR / media processing and power supply. One or more devices / nodes / UEs can be grouped into a cooperative XR group to support any of the XR applications / experiences / services in the XR applications / experiences / services.

[0004] Depending on the device capabilities, the application can need to delegate the computation of scan data to a server. Then, the scan data has to be transmitted in uplink (UL) between the UE and the server or in downlink (DL) between the server and the UE. In case of a shared experience (e.g. multiple users), a shared model can be useful, which requires transmission in UL. A standalone mode is proposed, but there is a lack of a mechanism to configure, capture, format and transmit scan data in the network. 3. SUMMARY The following is a simplified overview of this principle to provide a basic understanding of some aspects of it. This overview is not a comprehensive summary of this principle. It is not intended to identify the key or decisive elements of this principle. The overview below merely presents some aspects of this principle in a simplified form as an introduction to the more detailed description provided below.

[0006] A mechanism is proposed to process scanned data (3D model) information from a user's real-world environment for extended or augmented reality applications. This mechanism relies on a configuration step. The transmission of scanned data or raw data over a network from the user device to the server is then described in detail. The user device architecture can be a lightweight extended reality (XR) UE that needs to delegate the extraction and segmentation of the 3D model of the real-world scene to the server; an advanced user device capable of extracting the 3D model but requiring the segmentation task to be delegated to the server; or a high-end user device capable of performing 3D model extraction locally for independent use.

[0007] This principle relates to a method for managing scan data of an extended or augmented scene. The method includes initializing a user device configured to acquire data about the real-world environment relative to the extended or augmented scene. The user device's configuration is determined to define a pattern for transmitting scan data to a server. The user device scans the real-world environment and transmits the scan data to the server according to the configuration. In one embodiment, the user device may delegate the construction of a 3D model to the server. In this case, the server may send the 3D model back to the user device, which then segments it. In a variant, the user device also delegates the segmentation of the 3D model to the server. The initialization of the user device may be performed after the exchange with the server.

[0008] This principle also relates to a device including a processor configured to implement the methods described above. This device may be the user equipment itself. 4. Description of the attached drawings This disclosure will be better understood upon reading the following description, which refers to the accompanying drawings, in which: - Figure 1 The diagram illustrates a common module used to manage the representation of the reality portion of an XR scene; - Figure 2 Will Figure 1 The instantiation of the general module is described as a scene understanding module that maintains a representation of the user environment; - Figure 3 An example architecture of a device 30 that can be configured to implement the methods of embodiments according to these principles is shown; - Figure 4 The diagram illustrates the main steps of a method for configuring, capturing, formatting, and transmitting scan data in a network based on this principle; - Figure 5 The illustration shows that Figure 4 The configuration process is executed at step 42; - Figure 6 and Figure 7 The diagram illustrates the principle. Figure 4 Two embodiments of runtime step 43. 5. Detailed Implementation The principle will be described more fully below with reference to the accompanying drawings, in which examples of the principle are shown. However, the principle can be implemented in many alternative forms and should not be construed as limited to the examples set forth herein. Thus, while the principle is readily adaptable to various modifications and alternatives, specific examples are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the principle to the specific forms disclosed, but rather, this disclosure will cover all modifications, equivalents, and alternatives falling within the spirit and scope of the principle as defined by the claims.

[0011] The terminology used herein is for the purpose of describing particular examples only and is not intended to be a limitation of the principles herein. The singular forms “a,” “an,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that when the terms “comprising,” “including,” “containing,” and / or “includes” are used in this specification, the presence of the stated feature, integer, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof is not excluded. Furthermore, when an element is referred to as “in response to” or “connected to” another element, it may be directly responsive to or connected to another element, or there may be intermediate elements. In contrast, when an element is referred to as “directly responsive to” or “directly connected to” another element, there are no intermediate elements. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items and may be abbreviated to “ / ”.

[0012] It will be understood that although the terms first, second, etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the teachings of this principle, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0013] Although some of the diagrams include arrows along the communication path to indicate the main direction of communication, it should be understood that communication can occur in the opposite direction to the arrows depicted.

[0014] The block diagrams and operation flowcharts describe some examples where each block represents a circuit element, module, or section of code that includes one or more executable instructions for implementing a specified logical function(s). It should also be noted that in other implementations, the functions(s) indicated in a block may not appear in the indicated order. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved.

[0015] The use of "according to an example" or "in an example" in this document means that a particular feature, structure, or characteristic described in connection with the example may be included in at least one implementation of this principle. The phrases "according to an example" or "in an example" appearing throughout the specification do not necessarily refer to the same example, nor are they necessarily separate or alternative examples that are mutually exclusive with other examples.

[0016] Reference numerals appearing in the claims are for illustrative purposes only and should not be construed as limiting the scope of the claims. Although not explicitly described, these examples and variations may be used in any combination or subcombination.

[0017] In extended reality (XR) or augmented reality (AR) applications, a good understanding of the real-world environment is particularly important when virtual objects should be seamlessly integrated into the real world. A 3D model of the real-world environment is used to check for collisions, check for occlusion, correctly locate shadows, and correctly position virtual objects. Several parameters can be associated with the scan's output, such as a mesh, texture, point cloud, or Gaussian splatting. Geometry (e.g., a mesh) is mandatory for checking for collisions, occlusion, and locating shadows and virtual objects. Textures can be used to calculate the relighting of the real-world scene. When a scene scan is performed, the output can be a point cloud. Each point is associated with a color. From this point cloud, it is possible to generate a mesh. When depth data is available, it is possible to combine them with an image to generate a textured mesh. Meshes and textures can be used independently. A 3D model of the real-world scene corresponds to, for example, a point cloud, a mesh, or a textured mesh. Labeling and segmentation are performed in the next step. The input is a textured mesh. In this case, the term segmented 3D model is used. The 3D model can be segmented and labeled. When segmented, each element / object of the 3D model can be accessed individually. Tagging allows semantic access. For example, all chairs in a scene can be selected. Segmentation and tagging require significant processing power. This is the case when using artificial intelligence (AI) technologies, which have enabled significant progress in the field of segmentation. Therefore, lightweight UEs often need to delegate this processing to external entities with greater capabilities, such as remote edge servers. The 3D model of the real-world scene can be extracted from images of the environment captured by RGB cameras from different viewpoints (from moving structures). When available, a depth map of the scene can be constructed using a depth camera or LiDAR sensor. The first level of reconstruction is typically a point cloud. A mesh is then generated.

[0018] Scene understanding is the understanding of the real-world environment. It provides a 3D model of the real-world scene. Segmentation and tagging allow for fine-grained and precise interactions between elements of the real-world environment and virtual elements of the XR or AR scene. In XR applications, scene description is used to combine explicit and easily parsed descriptions of the scene structure with some binary representation of the media content. Typical scene description languages, such as glTF, allow for the description of XR scenes. Scene descriptions can be used to transmit 3D models of the real-world environment. The pose of the 3D model is relative to XR space. This XR space can be associated with a traceable space. It depends on anchors, traceable objects, and XR space. An anchor is a virtual element whose position, orientation, scale, and other attributes are expressed in the traceable space defined by traceable objects. The position, orientation, scale, and other attributes of a virtual asset are expressed relative to the anchor. A traceable object is a real-world object that can be traced by the XR runtime. Each traceable object provides a local reference space, also called the traceable space, in which the anchor can be expressed. XR space is a frame of reference where 3D coordinates are expressed.

[0019] Proprietary and standardized solutions exist for extracting scan data. For example, Khronos OpenXR provides cross-platform direct access to different XR device runtimes across multiple platforms. OpenXR enables applications and engines to run on any system that exposes the OpenXR API. Several extensions of OpenXR address real-world scene capture (MSFT, FB). The OpenXR API specifies several extensions related to scene understanding, which, if supported by the XR UE, allow initiating a scan; if supported by the XR UE, acquire the first level of the 3D Model (without segmentation) associated with the XR Space; request the 3D Model or elements of the 3D model from the server; and set the refresh rate.

[0020] Based on this principle, a solution is possible for defining modules specifically for scene understanding in XR device environments, using a new generic API to call these modules. Semantics are proposed to specify how reality / virtual integration should be performed in the scene description file.

[0021] Figure 1 The diagram illustrates a generic module 10 for managing the representation of the reality portion of an XR scene. This module and its associated API 11 process an external scene graph 12 synchronized with the virtual scene managed by the rendering engine 13. Semantics are provided to include scene-understanding-related information and describe the representation of the reality scene in the virtual scene description file. The generic API 11 exposes a set of functions to the rendering engine 13, including initializing and configuring the external scene manager module (i.e., creating a session between the two modules and setting the parameters required for the external module to process its scene); starting or stopping external module processing; notifying the external module that an update has occurred in the scene managed by the rendering engine; and registering callbacks for the external module to request status from the PE module or notify the external scene that an update has occurred.

[0022] Figure 2 Will Figure 1The instantiation 21 of the general module 10 is depicted as a scene understanding module that maintains a representation 22 of the user environment. In this example embodiment, this module is the sole entry point for the rendering engine module 23 to acquire an understanding of the real-world scene surrounding the user and integrate the virtual scene into that real-world scene. The scene understanding (SU) module 21 creates and maintains a representation 22 of the real-world environment surrounding the user (e.g., a scene graph of the real-world scene). It obtains updates to the scene graph by scanning the user environment in real time (using the XR runtime API) or by retrieving pre-scanned data from the world storage module. The rendering engine (PE) 23 can control the configuration and initiation of the creation and updating of the scene graph 22. A general API is proposed herein that provides methods for the PE. The scanning of the user environment is associated with anchors and / or traceables that allow links between the virtual scene and the locations of elements in the real-world environment to be integrated into the virtual scene or the virtual scene. Those anchors and traceables can be described in a virtual scene description file (e.g., a glTF MPEG_anchor extension). The PE can request the detection and tracking of traceables from the XR runtime module. The detection and tracking of trackable objects leads to the definition of an XR space, in which the pose of one or more anchor points is expressed. Another XR space is defined for each anchor point, where the pose of virtual elements can be expressed. Both the PE and SU modules have access to a list of existing XR spaces and use them to spatially synchronize their scene graphs. The Id (xrSpaceId) of the trackable XR space is given as input to the API methods to process the scanning of real-world objects around each trackable. Each xrSpaceId gives a mapping between the trackable described in the XR scene and a set of elements in the world scene. Each object in the world scene returned by the SU module includes a set of properties, including a referenceId to uniquely identify it as an object in the world scene and an xrSpaceId to identify the XR space it resides in. It may also include visual properties such as pose, mesh, material, and texture descriptions; lighting properties for objects representing real-world light sources; and physical properties for handling interactions with other real or virtual objects (occlusion, collision, etc.). Additional parameters may include semantic information specifying the properties of the object (e.g., "wall," "table," "chair," "light," "sound," etc.).

[0023] The following table provides possible APIs based on this principle: In the `configure()` method, the `scanOccurence` parameter specifies when the scan of the real-world object must be updated after the first call to the `start()` function. In variants, the scanning process can start automatically after a call to the `configure()` method. An additional `autostart` parameter can be added to this method to explicitly specify this automatic start mode. The `scanOccurence` parameter can have different values: `ONCE`: Perform only one scan, such as in a static real-world scene; `N_FRAME`: Perform scans periodically, every N render frames, where N depends on the dynamism of the real-world scene; `AUTO`: Scanning occurs and is managed by the SU module. The SU module can perform a lightweight pre-analysis to detect significant movement in the real world and initiate a new scan from the start. This pre-analysis can be performed from raw image data, such as RGB images from a camera or depth images from a depth sensor. Several of these possible values ​​can be combined to handle different scenarios (e.g., `ONCE` and `AUTO`). Object semantics can be any label representing the type of an object: “wall,” “floor,” “table,” “chair,” “light,” “sound,” “free space,” etc. Computation options and scan volumes can be similar to, for example, those specified in the Microsoft OpenXR Scene Understanding Extension.

[0024] Based on this principle, semantics are defined in the scene description file. This semantics specifies whether and how the user's real-world environment should be integrated into the scene. Semantics already exist in scene description languages, such as the MPEG-SD MPEG_anchor extension, which specifies the real-world elements in which virtual elements should be located.

[0025] The semantics proposed in this paper can be added as new parameters to the MPEG_anchor extension, or a new MPEG_node_real (or any other name) extension can be specified outside of the MPEG_anchor extension. These extensions are related to the glTF format, but we can add these semantics to extensions of any other description format (USD, FBX, etc.). In the table below, the values ​​in the usage column indicate whether the parameter is optional (O) or mandatory (M) according to this principle.

[0026] The possible values ​​for the scanOptions item are: The semantics of the scanDetail object can be: The possible values ​​for primitiveType are: The `scanOccurence` parameter specifies when the scan of the real-world object must be updated. It can have the same format as in the API's `configure()` method. The semantics of the `scanVolumes` object can be defined as follows: 3D coordinates are expressed in the XR space associated with the trackable object linked to the anchor point.

[0027] A scene description format for scanning real-world scenes provided by the Scene Understanding module is proposed. This format relates to real-world objects returned by the getSceneComponents() method and the API's UPDATE callback. It can be a description of the entire space surrounding the user, or it can be an update with only differences from what was previously provided. The format defined below is an example based on the Khronos glTF language, extended with MPEG-SD extensions, but similar features can be adapted to other languages, such as those used for internal representations of scenes (PE scenes). Scans of one or more real-world objects can appear as a scene graph (e.g., a glTF scene graph). Each root node of this graph is associated / mapped to a virtual node (i.e., a node sharing the same XR space) of the glTF scene from the PE module, characterized by the MPEG_anchor extension. The transformation matrix of the "real" node should be expressed in a traceable XR space (xrSpaceId). The "real" node can have meshes and textures generated and provided by the SU module. Additional parameters can be added to the "real" node within a new MPEG_node_real extension (or any other name).

[0028] Figure 4 The diagram illustrates the main steps of a method for configuring, capturing, formatting, and transmitting scan data in a network based on this principle.

[0029] The method begins with session initialization step 41, which demonstrates the UE's capabilities. The UE establishes an XR session locally based on device capabilities or a request from the server.

[0030] In a first embodiment, the UE sends a message to the server requesting a scan of the virtual scene, and the UE demonstrates its capabilities. The UE sets the mode required for the scan. This initialization step can be completed during an uplink delivery session. In another embodiment, the server sends a message requesting a scan of the real scene, and the UE responds by demonstrating its capabilities. This initialization step can be completed during a downlink delivery session. The initialization message exchange includes at least information about the process of initiating a scan of the real scene. In a second step, a segmentation process can be initiated. For a shared AR experience, a parameter indicating that the 3D model is transferred to the server (shared data) is optional. The transfer of shared data assumes that extraction is performed locally.

[0031] The parameters are: ScanRequired: If true, the real-world environment must be scanned to build a 3D model (from the server or from the UE's request); SegmentationRequired: If true, the 3D model must be segmented (from the server or from the UE's request); SharedDatas: If true, the 3D model data or the segmented 3D model must be transferred to other UEs sharing the same XR experience.

[0032] Capability messages are the UE's response to an initialization message from the server. They are uplink (UL) messages. The parameters are LocalScanning: if true, the UE can extract scan data; and LocalSegmentation: if true, the UE can segment the 3D model.

[0033] Session initialization information can be formatted, for example, in JSON format.

[0034] Figure 5 The illustration shows that Figure 4 The configuration process is performed at step 42. According to this principle, during the configuration process, messages are exchanged between the UE and the server to configure the transmission of scan data. The exchanged information may relate to the type of data to be transmitted, the segmentation process, the specific request for the object to be scanned, the frequency of light to be evaluated, and the XR space. Depending on its capabilities and the initialization session, the UE extracts the scan data and transmits the 3D model to the server. In one embodiment, the server initiates configuration and sends a configuration message (e.g., when scanning and segmentation are completed on the server side). In another embodiment, the UE initiates configuration and sends a configuration message. Depending on this, configuration data exchange is adapted. When the server receives the 3D model, if a segmentation operation has been requested, the server has that information to perform the segmentation operation.

[0035] The parameters are scanOccurrence; XRSpace; and data: the type of data will vary depending on the UE's capabilities. Three different types of data are transmitted: raw data: transmits pose, image, and (optionally) depth maps to build a 3D model; 3D model: transmits the 3D model for segmentation and labeling; and segmented 3D model: transmits the segmented and labeled 3D model for shared applications. In one embodiment, each XRSpace considers one configuration message. In another embodiment, configuration messages are grouped into an array (if there is more than one traceable object / XRSpace in the scene).

[0036] Configuration parameters are negotiated between the UE and the server. This includes defining a shared XR space and defining the data transmitted to the server for 3D model construction. The format is extensible to support the exchange of additional / future configuration information. Configuration is performed during the XRSpatial downlink and uplink delivery session. The configuration format describes the data format. Session configuration information can be formatted, for example, in JSON format.

[0037] RawData is defined according to the following syntax: In one embodiment, the pose can be extracted from a set of images. In another embodiment, a depth map texture can be transmitted in a RawData object. This depth map corresponds to depth data associated with the view pose. In this case, the parameter "Depth" is not transmitted. The intrinsic parameters (camera) are known by the server (other parts of the initialization). A refresh can occur during the configuration process or be set by the UE (if the UE can extract the 3D model). In one embodiment, refresh occurrences can be logically combined. In another embodiment, data is transmitted only when a value changes.

[0038] Configuration parameters are negotiated between the UE and the server. This includes defining a shared XR space and defining the data transmitted to the server for 3D model segmentation. The format is extensible to support the exchange of additional / future configuration information. Configuration is performed during the XRSpatial downlink and uplink delivery sessions. A complete or partial 3D model can be transmitted for the segmentation process. When segmentation is not required, the UE can receive the 3D model.

[0039] 3DModel is defined according to the following syntax: A refresh can occur during the configuration process or be set by the UE (if the UE can extract the 3D model). The 3D model is transmitted when an update occurs. This occurrence value is considered in all scenarios, including when no data will be transmitted. Configuration parameters are negotiated between the UE and the server. This includes defining a shared XR space and defining the data transmitted to the server. The format is extensible to support the exchange of additional / future configuration information. Configuration is performed during XRSpatial downlink and uplink delivery sessions. The segmented 3D model can be transmitted from the UE to the server or from the server to the UE.

[0040] Segmented3DModel is defined according to the following syntax: A refresh can occur during the configuration process or be set by the UE (if the UE can extract the 3D model). The 3D model is transmitted when an update occurs. This occurrence value is considered in all scenarios, including when data is transmitted. In one embodiment, a graph of 3D objects is transmitted instead of an array of 3D objects. In this case, the 3D objects are nodes. In another embodiment, only a texture (atlas) is transmitted.

[0041] Figure 6 and Figure 7 The diagram illustrates the principle. Figure 4 Two embodiments of runtime step 43. Depending on the configuration at step 4, in Figure 6 The calling process or Figure 7 The call flow is used between the UE and the server at runtime.

[0042] The parameters are: Scan: If true, scene scanning is performed by the UE; if false, scanning is performed by the server; Seg: If true, 3D model segmentation is performed by the UE; if false, segmentation is performed by the server; Shared: If true, the 3D model is transferred from the UE to the server; if false, no data is transferred. The consistency between the transferred data and the scan volume defined during initialization is guaranteed by the server. In one embodiment, the application guides the user by displaying the scan volume at runtime. The cross "X" means "don't care," T represents true, and F represents false. Figure 6 The call process requires scanning and segmentation. The UE's capabilities correspond to the Scan and Seg items. Figure 7 In the call flow, only scanning is required. The UE's capabilities correspond to the Scan item. The Seg item is ignored.

[0043] Depending on the configuration process, the client on the XR device (UE) or server periodically transmits a set of raw data. When transmitting raw data, parameter values ​​are transmitted during runtime. Messages can be formatted, for example, in JSON format. The message payload is set according to the following example syntax: It can also transmit depth data.

[0044] When transmitting a 3D model, the client on the XR device periodically transmits the original 3D model to the server for segmentation. During runtime, parameter values ​​are transmitted. Messages can be formatted, for example, in JSON format. The message payload is set according to the following example syntax: When transmitting segmented 3D models, the client on the XR device periodically transmits the segmented objects to the server. The message payload is set according to the following example syntax: Figure 3 An example architecture of a device 30 is shown that can be configured to implement methods according to embodiments of this principle. The device is linked to other devices via its bus 31 and / or via I / O interfaces 36.

[0045] Device 30 includes the following components linked together via data and address bus 31: D - Processor 32 (or CPU), which is, for example, a DSP (or digital signal processor); -ROM (or read-only memory) 33; -RAM (or random access memory) 34; - Storage interface 35; -I / O interface 36, used to receive data to be transmitted from the application; and -power supply( Figure 2 (not shown in the text), such as batteries.

[0046] According to the example, the power supply is external to the device. In each of the mentioned memories, the term "register" used in the specification can correspond to a small area (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). ROM 33 includes at least the program and parameters. ROM 33 can store algorithms and instructions to execute the technology according to these principles. When powered on, CPU 32 uploads the program to RAM and executes the corresponding instructions.

[0047] RAM 34 includes in registers the program executed by CPU 32 and uploaded after device 30 is turned on, input data in registers, intermediate data in registers for different states of the method, and other variables in registers used for executing the method.

[0048] The implementations described herein can be implemented, for example, in methods or processes, apparatuses, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., only as a method or apparatus), the implementations of the features in question can be implemented in other forms (e.g., programs). Apparatuses can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in apparatuses (such as, for example, processors, which generally refer to processing devices, including, for example, computers, microprocessors, integrated circuits, or programmable logic devices). Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0049] Device 30 is connected, for example, via bus 31 to a set of sensors 37 and a set of rendering devices 38. Sensors 37 may be, for example, cameras, microphones, temperature sensors, inertial measurement units, GPS, humidity sensors, IR or UV light sensors, or wind sensors. Rendering devices 38 may be, for example, displays, speakers, vibrators, heaters, fans, etc.

[0050] According to the example, device 30 is configured to implement the methods according to this principle and belongs to a set that includes the following: -mobile device; - Communication equipment; -Gaming devices; - Tablet (or tablet computer); - Laptop computers; -Still image camera; -Camera.

[0051] The implementations described herein can be implemented, for example, in methods or processes, apparatuses, computer program products, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., only as a method or apparatus), the implementations of the features in question can be implemented in other forms (e.g., programs). Apparatuses can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in apparatuses (such as, for example, processors, which generally refer to processing devices, including, for example, computers, microprocessors, integrated circuits, or programmable logic devices). Processors also include communication devices, such as, for example, smartphones, tablets, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0052] The various processes and features described herein can be implemented in a wide variety of equipment or applications, particularly those associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture and / or depth information. Examples of such equipment include encoders, decoders, post-processors that process the output from decoders, pre-processors that provide input to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, cellular phones, PDAs, and other communication devices. It should be clear that the equipment can be mobile and even mounted in mobile vehicles.

[0053] Furthermore, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a processor-readable medium, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, a compact disc (“CD”), an optical disc (such as, for example, a DVD, commonly referred to as a digital multifunction disc or digital video disc), random access memory (“RAM”), or read-only memory (“ROM”). The instructions can form an application program tangibly implemented on the processor-readable medium. The instructions can be, for example, in hardware, firmware, software, or a combination thereof. The instructions can be found, for example, in an operating system, a separate application, or a combination of both. Thus, a processor can be characterized as, for example, a device configured to execute a process and a device comprising a processor-readable medium (such as a storage device) having instructions for executing the process. Furthermore, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0054] As will be apparent to those skilled in the art, implementations can generate various signals that are formatted to carry, for example, information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry as data rules for writing or reading the syntax of the described embodiments, or as data actual syntax values ​​written by the described embodiments. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0055] Many implementations have been described. However, it will be understood that various modifications can be made. For example, elements of different implementations can be combined, supplemented, modified, or removed to produce other implementations. Furthermore, those skilled in the art will understand that other structures and processes can replace those disclosed, and the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, these and other implementations are considered in this application.

Claims

1. A method for managing data in an extended or enhanced scenario, the method comprising: - Initialize the user equipment, which is configured to acquire data about the real-world environment relative to the extended or augmented scenario; - Determine the configuration of the user equipment regarding how data is transmitted to the server; - Data is obtained by capturing the real-world environment using the user device; as well as - Data is transmitted to the server according to the configuration.

2. The method of claim 1, wherein determining the configuration includes determining whether to delegate the construction of a 3D model of the real-world environment to a server, the construction being data-based.

3. The method of claim 2, wherein the server transmits the 3D model to the user equipment, and wherein the user equipment segments the 3D model.

4. The method of claim 1, wherein the user equipment constructs a 3D model of the real environment based on data, transmits the 3D model to the server, and wherein the configuration includes determining whether to delegate the segmentation of the 3D model to the server.

5. The method according to any one of claims 1 to 4, wherein initializing the user equipment includes sending a capture request and the user equipment's capabilities to the server.

6. The method according to any one of claims 1 to 4, wherein initializing the user equipment includes receiving a capture request from the real environment, and wherein the user equipment transmits its capabilities in accordance with the capture request.

7. An apparatus comprising a processor configured to: - Initialize the user equipment, which is configured to acquire data about the real-world environment relative to the extended or augmented scenario; - Determine the configuration of the user equipment for transmitting data to the server; - Data is obtained by capturing the real-world environment using the user device; and - Data is transmitted to the server according to the configuration.

8. The device of claim 7, wherein determining the configuration includes determining whether to delegate the construction of a 3D model of the real-world environment to a server, the construction being data-based.

9. The device of claim 8, wherein the server transmits the 3D model to the user equipment, and wherein the user equipment segments the 3D model.

10. The device of claim 7, wherein the user equipment constructs a 3D model of the real environment based on data, transmits the 3D model to the server, and wherein the configuration includes determining whether to delegate the segmentation of the 3D model to the server.

11. The device according to any one of claims 7 to 10, wherein initializing the user equipment includes sending a capture request to the server and the user equipment's capability.

12. The device according to any one of claims 7 to 10, wherein initializing the user equipment includes receiving a capture request from the real environment, and wherein the user equipment transmits its capabilities in accordance with the capture request.

Images