Policy-based volumetric point cloud data streaming

Abstract

A method comprising: receiving data from one or more point cloud sources; determining one or more structural elements within the received data, said structural elements having properties suitable for5 one or more predetermined transmission policies; encoding the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; encapsulating the one or more bitstreams into a streaming format; generating signalling comprising information about one or0 more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; transmitting the encapsulated one or more5 bitstreams over the one or more streaming format sessions to a client; and signalling said information to the client.

Description

POLICY-BASED VOLUMETRIC POINT CLOUD DATA STREAMINGTECHNICAL FIELD

[0001] The present invention relates to an apparatus, a method and a computer program for streaming of point cloud data.BACKGROUND

[0002] Point cloud is a set of data points in a coordinate system, for example in a three- dimensional coordinate system. The points may represent an external surface of an object in the scene space, e.g. in a three-dimensional space, where each point of each 3D surface is described as a 3D point with color and / or other attribute information such as surface normal or material reflectance. Point clouds typically contain a large amount of data, bringing challenges to data storage and transmission.

[0003] LiDAR (Light Detection and Ranging) devices are commonly used in industrial context for 3D environment sensing, modelling and navigation by generating dynamic point clouds about the 3D scene. Traditionally, LiDAR data is processed on the capturing device / platform (vehicle, drone, etc.). As LiDAR data grows bigger and more complex, more storage and processing resources are required. Thus, in order to optimize the storage and processing cost, LiDAR data processing may be pushed to edge or cloud computational resources. Accordingly, LiDAR data needs to be streamed from the capture device / platform to another network-connected entity.

[0004] However, streaming LiDAR or any other point cloud data involves various problems, for example, relating to high bitrates on bandwidth-limited transmission channels, different priority or quality requirements for different types or parts of the point cloud and lossy transport protocols.

[0005] Thus, there is a need for a more flexible approach for streaming point cloud data.SUMMARY

[0006] Now, an improved method and technical equipment implementing the method has been invented, by which the above problems are alleviated. Various aspects include amethod, an apparatus and a computer readable medium comprising a computer program, or a signal stored therein, which are characterized by what is stated in the independent claims. Various details of the embodiments are disclosed in the dependent claims and in the corresponding images and description.

[0007] The scope of protection sought for various embodiments of the invention is set out by the independent claims. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

[0008] According to a first aspect, there is provided an apparatus comprising means for receiving data from one or more point cloud sources; means for determining one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; means for encoding the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; means for encapsulating the one or more bitstreams into a streaming format; means for generating signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; means for transmitting the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and means for signalling said information to the client.

[0009] According to an embodiment, the apparatus comprises means for identifying a given structural element to belong to one or more predetermined classes; and means for assigning a value of relevance for each of said one or more predetermined classes of the given structural element.

[0010] According to an embodiment, the apparatus comprises means for encoding the structural elements as one or more separate point clouds.

[0011] According to an embodiment, the apparatus comprises means for encoding the structural elements as one or more identifiable parts of a point cloud.

[0012] According to an embodiment, the apparatus comprises means for associating the structural elements with one or more attributes; and means for generating a signallingabout the attributes to be provided in or along the encoded one or more bitstreams comprising the structural elements.

[0013] According to an embodiment, the apparatus comprises means for packetizing the encoded one or more bitstreams comprising the structural elements based on the associated signalling and / or the value of relevance of the structural elements.

[0014] According to an embodiment, the apparatus comprises means for indicating reliability requirements per a data packet in payload-specific header fields of the packetized bitstream.

[0015] According to an embodiment, the apparatus comprises means for updating the signalling based on capturing conditions of the one or more point cloud sources.

[0016] According to an embodiment, said signalling is carried out according to Session Description Protocol.

[0017] According to an embodiment, the signalling comprises a syntax element for a grouping parameter, which comprises constituent information about the one or more bitstreams.

[0018] According to an embodiment, the signalling comprises a syntax element for a G- PCC mapping attribute, which comprises indication of the one or more bitstreams and related parameters.

[0019] According to an embodiment, said streaming format is Real Time Transfer Protocol.

[0020] An apparatus according to a second aspect comprises at least one processor and at least one memory, said at least one memory stored with computer program code thereon, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: receive data from one or more point cloud sources; determine one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; encode the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; encapsulate the one or more bitstreams into a streaming format; generate signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one ormore streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; transmit the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and signal said information to the client.

[0021] A method according to a third aspect comprises receiving data from one or more point cloud sources; determining one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; encoding the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; encapsulating the one or more bitstreams into a streaming format; generating signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; transmitting the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and signalling said information to the client.

[0022] According to a fourth aspect, there is provided an apparatus comprising means for receiving signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, wherein said structural elements have properties suitable for one or more predetermined transmission policies; means for receiving the one or more bitstreams as encapsulated over the one or more streaming format sessions; means for decapsulating the one or more bitstreams based on said signalling; means for decoding the one or more bitstreams into point cloud data; and means for composing a composition of the point cloud data from the point cloud data of the one or more bitstreams.

[0023] An apparatus according to a fifth aspect comprises at least one processor and at least one memory, said at least one memory stored with computer program code thereon, the at least one memory and the computer program code configured to, with the at least oneprocessor, cause the apparatus at least to perform: receive signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, wherein said structural elements have properties suitable for one or more predetermined transmission policies; receive the one or more bitstreams as encapsulated over the one or more streaming format sessions; decapsulate the one or more bitstreams based on said signalling; decode the one or more bitstreams into point cloud data; and compose a composition of the point cloud data from the point cloud data of the one or more bitstreams.

[0024] A method according to a sixth aspect comprises receiving signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, wherein said structural elements have properties suitable for one or more predetermined transmission policies; receiving the one or more bitstreams as encapsulated over the one or more streaming format sessions; decapsulating the one or more bitstreams based on said signalling; decoding the one or more bitstreams into point cloud data; and composing a composition of the point cloud data from the point cloud data of the one or more bitstreams

[0025] Computer readable storage media according to further aspects comprise code for use by an apparatus, which when executed by a processor, causes the apparatus to perform the above methods.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] For a more complete understanding of the example embodiments, reference is now made to the following descriptions taken in connection with the accompanying drawings in which:

[0027] Fig. 1 shows a flow chart for an encoding or a stream creation method according to an embodiment;

[0028] Figs. 2a - 2c show implementation examples of the architecture of functionalities according to various embodiments; and

[0029] Fig. 3 shows a flow chart for a decoding or a stream composing method according to an embodiment.DETAILED DESCRIPTON OF SOME EXAMPLE EMBODIMENTS

[0030] MPEG-I (ISO / IEC 23090) standard has been developed for encoding, encapsulation, and delivery of immersive media. Volumetric video may be captured using one or more three-dimensional (3D) cameras. When multiple cameras are in use, the captured footage is synchronized so that the cameras provide different viewpoints to the same world. In contrast to traditional 2D / 3D video, volumetric video describes an immersive 3D model of the world where the viewer is free to move and observe different parts of the world.

[0031] Volumetric video enables the viewer to move in six degrees of freedom (6DOF): in contrast to common 360° video, where the user has from 2 to 3 degrees of freedom (yaw, pitch, and possibly roll), a volumetric video represents a 3D volume of space rather than a flat image plane. Volumetric video frames contain a large amount of data because they model the contents of a 3D volume instead of just a two-dimensional (2D) plane. However, only a relatively small part of the volume changes over time. Therefore, it may be possible to reduce the total amount of data by only coding information about an initial state and changes which may occur between frames. Volumetric video can be rendered from synthetic 3D animations, reconstructed from multi-view video using 3D reconstruction techniques such as structure from motion, or captured with a combination of cameras and depth sensors such as LiDAR (Light Detection and Ranging), for example.

[0032] Volumetric video data represents a three-dimensional scene or object, and thus such data can be viewed from any viewpoint. Volumetric video data can be used as an input for augmented reality (AR), virtual reality (VR) and mixed reality (MR) applications. Such data describes geometry (shape, size, position in 3D-space) and respective attributes (e.g. color, opacity, reflectance, ...), together with any possible temporal changes of thegeometry and attributes at given time instances (e.g. frames in 2D video). Volumetric video is either generated from 3D models, i.e. computer-generated imagery (CGI), or captured from real-world scenes using a variety of capture solutions, e.g. a multi-camera, a laser scan, a combination of video and dedicated depths sensors, etc. Also, a combination of CGI and real- wo rid data is possible. Examples of representation formats for such volumetric data are triangle meshes, point clouds, or voxel. Temporal information about the scene can be included in the form of individual capture instances, i.e. “frames” in 2D video, or other means, e.g. position of an object as a function of time.

[0033] Increasing computational resources and advances in 3D data acquisition devices has enabled reconstruction of highly detailed volumetric video representations of natural scenes. Infrared, lasers, time-of-flight and structured light are all examples of devices that can be used to construct 3D video data. Representation of the 3D data depends on how the 3D data is used. Dense voxel arrays have been used to represent volumetric medical data. In 3D graphics, polygonal meshes are extensively used. Point clouds on the other hand are well suited for applications, such as capturing real world 3D scenes where the topology is not necessarily a 2D manifold. Another way to represent 3D data is coding this 3D data as a set of texture and depth map as is the case in the multi-view plus depth. Closely related to the techniques used in multi-view plus depth is the use of elevation maps, and multi-level surface maps.

[0034] In 3D point clouds, each point of each 3D surface is described as a 3D point with color and / or other attribute information such as surface normal or material reflectance. Point cloud is a set of data points in a coordinate system, for example in a three- dimensional coordinate system being defined by X, Y, and Z coordinates. The points may represent an external surface of an object in the scene space, e.g. in a three-dimensional space.

[0035] To precisely describe 3D objects or real-world scenes, point clouds produce a large set of points in 3D space with geometry information and attribute information. The geometry information represents the 3D coordinates of each point in the point cloud, while the attribute information describes the characteristics (e.g., color and reflectance) of each point. Point cloud requires a large amount of data, bringing huge challenges to data storage and transmission.

[0036] MPEG 3D Graphics Coding group (3DG) announced the Call for Proposal on point cloud compression already in 2017 aiming to develop efficient compression technologies for different types of point clouds that are acquired with different methods. Due to the different compression requirements for different types of point clouds, the standardization of point cloud compression in MPEG-I (ISO / IEC 23090) has been divided into two distinct technical solutions: MPEG Video based Point Cloud Coding (V-PCC; ISO / IEC 23090 Part 5) which is suitable for coding dense content, and MPEG Geometry based Point Cloud Coding (G-PCC; ISO / IEC 23090 Part 9) which is efficient for relatively sparse data.

[0037] In V-PCC, a projection-based coding principle, where the point coordinate is encoded as the distance with respect to a particular plane, is used. The attribute associated to a 3D point is projected to a 2D map. 2D video codecs can be used to code both the geometry and attribute information.

[0038] G-PCC, in turn, is a 3D oriented compression scheme in which the point cloud in 3D space is directly encoded. Data structures, such as octree and kd-tree, are used to represent the point cloud. G-PCC encodes the geometry information and attribute information of point clouds separately.

[0039] In terms of compression performances, G-PCC provides state-of-the-art results for a majority of point cloud types. In contrast, V-PCC outperforms G-PCC for the particular case of dense points clouds with a low level of noise and motion.

[0040] LiDAR devices are commonly used in industrial context for 3D environment sensing, modelling and navigation by generating dynamic point clouds about the 3D scene. Traditionally, LiDAR data is processed on the capturing device / platform (vehicle, drone, etc.). As LiDAR data grows bigger and more complex, more storage and processing resources are required. Furthermore, recording the scene may involve more than one capture device. G-PCC is aimed to provide high coding performance for such dynamic point clouds with relatively sparse data content.

[0041] It is noted that LiDAR is mentioned here only as an example of technologies that produce point cloud data via electromagnetic wave sensing or by means of content generation. Other sources of point cloud data, applicable to the embodiments describedfurther below, include, for example, RADAR systems, mmWave -based sensing systems, and 3D scene rendering systems, among others.

[0042] G-PCC encodes the geometry and attribute information of a point cloud separately. The geometry information containing point positions is coded first while the attribute compression depends on the reconstructed geometry. The attributes may contain any properties associated with individual points such as color, reflection, etc.

[0043] G-PCC makes no assumption about the input point cloud coordinate representation. The points have an internal integer-based value, converted from a floatingpoint value representation. This conversion is conceptually similar to voxelization of the input point cloud, and can be achieved by scaling, translation, and rounding.

[0044] A key concept in G-PCC is that point clouds can be divided in tiles and slices to support partial decoding of selected regions. Thus, G-PCC supports parallel processing both in encoder and decoder with a slice coding. A single point cloud frame can be divided into multiple data unit as a slice to allow parallel processing within a single frame. A slice is a set of points comprising one geometry data unit and zero or more attribute data units that can be independently encoded and decoded. A group of slices may be identified by a common tile identifier. A tile inventory contains metadata that defines zero or more spatial regions. Each spatial region is identified by either an implicit or explicit tile ID. Each slice that belongs to the same tile may be allowed to have dependencies while the slices with different indices of the tile should be independent. A tile may overlap with another tile and the decoder can decode a partial area of the point cloud by accessing specific slices.

[0045] Bitstream is structured in syntax structures of (10) different types:Sequence / Geometry / Attribute Parameter Sets o Coding configuration parameters Geometry / Attribute Data units o encoded geometry and attribute dataTile Inventory, Frame Boundary Marker, Defaulted Attributes, Frame-specific attribute properties, User Data o Additional meta data

[0046] Syntax structures can be encapsulated in type-length-value encoding.

[0047] For the more detailed implementation of G-PCC encoding, a reference is made to MPEG G-PCC (ISO / IEC 23090-9).

[0048] It is noted that G-PCC is mentioned here only as an example of point cloud encoding that preserves the structural information of a point cloud when encoding it. Other schemes that exhibit similar properties that allow identifying geometric regions within a point cloud (e.g., 3-dimensional matrix representation) may be equally applicable to the embodiments described further below.

[0049] A Real Time Transfer Protocol (RTP) is intended for an end-to-end, real-time transfer or streaming media and provides facilities for jitter compensation and detection of packet loss and out-of-order delivery. RTP allows data transfer to multiple destinations through IP multicast or to a specific destination through IP unicast. The majority of the RTP implementations are built on the User Datagram Protocol (UDP). Other transport protocols may also be utilized. RTP can be used together with other protocols such as H.323 and Real Time Streaming Protocol RTSP.

[0050] The RTP specification describes two protocols: RTP and RTCP. RTP is used for the transfer of multimedia data, and the RTCP is used to periodically send control information and QoS parameters.

[0051] RTP sessions may be initiated between client and server using a signalling protocol, such as H.323, the Session Initiation Protocol (SIP), or RTSP. These protocols may use the Session Description Protocol (RFC 8866) to specify the parameters for the sessions.

[0052] RTP is designed to carry a multitude of multimedia formats, which permits the development of new formats without revising the RTP standard. To this end, the information required by a specific application of the protocol is not included in the generic RTP header. For a class of applications (e.g., audio, video), an RTP profile may be defined. For a media format (e.g., a specific video coding format), an associated RTP payload format may be defined. Every instantiation of RTP in a particular application may require a profile and payload format specifications.

[0053] The profile defines the codecs used to encode the payload data and their mapping to payload format codecs in the protocol field Payload Type (PT) of the RTP header.

[0054] An RTP session is established for each multimedia stream. Audio and video streams may use separate RTP sessions, enabling a receiver to selectively receive components of a particular stream. The RTP specification recommends even port number for RTP, and the use of the next odd port number of the associated RTCP session. A single port can be used for RTP and RTCP in applications that multiplex the protocols.

[0055] Each RTP stream consists of RTP packets, which in turn consist of RTP header and payload parts.

[0056] RTP packets are created at the application layer and handed to the transport layer for delivery. Each unit of RTP media data created by an application begins with the RTP packet header.

[0057] The RTP header has a minimum size of 12 bytes. After the header, optional header extensions may be present. This is followed by the RTP payload, the format of which is determined by the particular class of application. The fields in the header are as follows:- Version: (2 bits) Indicates the version of the protocol.- P (Padding): (1 bit) Used to indicate if there are extra padding bytes at the end of the RTP packet.- X (Extension): (1 bit) Indicates the presence of an extension header between the header and payload data. The extension header is application or profile specific.- CC (CSRC count): (4 bits) Contains the number of CSRC identifiers that follow the SSRC- M (Marker): (1 bit) Signalling used at the application level in a profile-specific manner. If it is set, it means that the current data has some special relevance for the application.- PT (Payload type): (7 bits) Indicates the format of the payload and thus determines its interpretation by the application.- Sequence number: (16 bits) The sequence number is incremented for each RTP data packet sent and is to be used by the receiver to detect packet loss and to accommodate out-of-order delivery.- Timestamp: (32 bits) Used by the receiver to play back the received samples at appropriate time and interval. When several media streams are present, the timestamps may be independent in each stream. The granularity of the timing is application specific. For example, video stream may use a 90 kHz clock. The clock granularity is one of the details that is specified in the RTP profile for an application.- SSRC: (32 bits) Synchronization source identifier uniquely identifies the source of the stream. The synchronization sources within the same RTP session will be unique.- CSRC: (32 bits each) Contributing source IDs enumerate contributing sources to a stream which has been generated from multiple sources.- Header extension: (optional, presence indicated by Extension field) The first 32- bit word contains a profile-specific identifier (16 bits) and a length specifier (16 bits) that indicates the length of the extension in 32-bit units, excluding the 32 bits of the extension header.

[0058] In this disclosure, the Session Description Protocol (SDP) is used as an example of a session specific file format. SDP is a format for describing multimedia communication sessions for the purposes of announcement and invitation. Its predominant use is in support of conversational and streaming media applications. SDP does not deliver any media streams itself, but it is used between endpoints for negotiation of network metrics, media types, and other associated properties. The set of properties and parameters is called a session profile. SDP is extensible for the support of new media types and formats.

[0059] The Session Description Protocol describes a session as a group of fields in a text-based format, one field per line. The form of each field is as follows:<character>=<value><CRXLF> where <character> is a single case-sensitive character and <value> is structured text in a format that depends on the character. Values may be UTF-8 encoded. Whitespace is not allowed immediately to either side of the equal sign.

[0060] Session descriptions consist of three sections: session, timing, and media descriptions. Each description may contain multiple timing and media descriptions. Names are only unique within the associated syntactic construct.

[0061] As discussed above, LiDAR data has typically been processed on the capturing device / platform (vehicle, drone, etc.). However, as LiDAR data easily grows bigger and more complex, more storage and processing resources are required. Thus, in order to optimize the storage and processing cost, LiDAR data processing may be pushed to edge or cloud computational resources. Accordingly, LiDAR data needs to be streamed from the capture device / platform to another network-connected entity.

[0062] Another reason for streaming LiDAR data is centralized processing of distributed capture of LiDAR information. One example is digital twinning of mining tunnels, where multiple mining vehicles capture LiDAR based point clouds of the environment and these captured 3D point clouds are then required to be sent to a central processing unit / entity for generation of a combined and consolidated 3D point cloud, which is then treated as the digital twin of the mine.

[0063] However, streaming LiDAR data involves various problems, including at least the following:LiDAR devices generate high volumes of data which may consume high bitrates on bandwidth-limited transmission channels.Not all information in the 3D scene has the same priority or importance. For example, dynamically moving entities on the road may have higher priority compared to stationary structures such as buildings (e.g. due to obstacle avoidance). On the other hand, stationary structures may have higher priority in a different use case (e.g. mining tunnel wall inspection). Sometimes, only the field-of-view of the operator is important (e.g. teleoperation). Hence, streaming the whole of 3D sensed LiDAR data with the same priorities and quality results in non-optimal bandwidth utilization in most of the cases.Real-time transport protocol is usually lossy in nature. Loss of a portion of LiDAR data may cause to discard a bigger chunk of LiDAR data due to coarse granularity of LiDAR data.

[0064] Accordingly, there is a need for a more flexible approach for streaming different portions of point cloud data having different quality levels.

[0065] In the following, an enhanced method for streaming point cloud data will be described in more detail, in accordance with various embodiments.

[0066] The method, which is disclosed in Figure 1 and illustrates the operation of a sender, comprises receiving (100) data from one or more point cloud sources; determining (102) one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; encoding (104) the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; encapsulating (106) the one or more bitstreams into a streaming format; generating (108) signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; transmitting (110) the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and signalling (112) said information to the client.

[0067] Thus, one or more structural elements within one or more point clouds received from one or more sensing devices are identified. The structural elements are parts or subgroups of points in said one or more point clouds, which have certain properties suitable for at least one transmission policy. The properties may relate to, for example, urgency, priority, encoding, precision, scheduling, reliability mechanisms, QoS tagging of a given structural element.

[0068] Once the structural elements have been identified, the received point cloud data including the structural elements is encoded into one or more bitstreams, wherein the encoding is controlled based on the properties of said one or more structural elements at least to the extent whether to encode one or more bitstreams to be encapsulated in a corresponding number of streaming format bitstreams, such as RTP streams. Using one bitstream simplifies the transmission implementation, whereas using multiple bitstreams enables organising the structural elements with versatile transmission properties into bitstreams having different transmission policies.

[0069] One or more RTP sessions may be initiated between the sender and the client, wherein Session Description Protocol (SDP) may be used on a signalling protocol to specify the parameters for the sessions. Thus, signalling information is generated, which allows the client to map a session, e.g., a RTP session, to one or more appropriatebitstreams, and allows the client to determine the relations between sessions, e.g., RTP sessions, that contain different structural elements. Then the one or more bitstream are transmitted over the one or more RTP sessions clients, as well as the signalling information.

[0070] Accordingly, the method enables to create and map multiple point cloud originated bitstreams, such as G-PCC bitstreams, to SDP that allows to signal to a receiver all information required to receive separate RTP sessions and reconstruct point cloud based on the provided information. The properties of the structural elements may be used to control whether to encode one or more bitstreams.

[0071] Figure 2a illustrates a simplified architecture of functionalities for implementing the embodiments. Therein, data is received from one or more point cloud sources (200), such as a LiDAR camera. A detector of structural elements 202 determines structural elements within the received point cloud data. A point cloud encoder 204, such as a G-PCC encoder, encodes the received point cloud data, taking into account the structural elements within the received data, to one or more bitstreams. A packetizer 206 encapsulates the one or more bitstreams, for example, into a RTP payload format. A session description generator 208 generates signalling information comprising at least mapping information between one or more RTP sessions and the underlying one or more RTP bitstreams and relations of the structural elements in the one or more RTP sessions. The packetizer 206 sends the encapsulated one or more bitstreams over the one or more RTP sessions to the receiver via the network 210. The session description generator 208, in turn, sends the signalling to the receiver via the network 210.

[0072] The receiver, which may comprise e.g. a network server or cloud computing node, comprises a depacketizer 212, which receives the signalling from the session description generator 208. The depacketizer 212 also receives the one or more RTP streams over one or more RTP sessions. Based on the signalling comprising information about the RTP sessions, and relations thereof, the depacketizer 212 decapsulates the one or more bitstreams, for example, from the RTP payload format. A decoder 214 decodes the one or more bitstreams representing the point cloud data. A composer 216 composes the point cloud data from the received structural elements.

[0073] Figure 2b illustrates an implementation of the architecture of functionalities according to another embodiment. Therein, the detector of structural elements 202 determines the structural elements within the received point cloud data, and at least partly based on their properties, divides the point cloud data to a plurality of bitstreams 1-n to be separately processed. Thus, the processing chain comprises a corresponding number of point cloud encoders 204.1 - 2O4.n, such as G-PCC encoders, or a corresponding number of encoder threads 204.1 - 204. n parallelly executed by a single encoder. The processing chain comprises a corresponding number of packetizers 206.1 - 206. n, each encapsulating the bitstream, for example, into a RTP payload format. The packetizers 206.1 - 2O6.n provide the session description generator 208 information about the encapsulated bitstreams, and the session description generator 208 generates signalling information comprising at least mapping information between one or more RTP sessions and the underlying one or more RTP bitstreams and relations of the structural elements in the one or more RTP sessions. The packetizers 206.1 - 206. n send their corresponding encapsulated bitstream over an RTP session to the receiver via the network 210. The session description generator 208, in turn, sends the signalling to the receiver via the network 210.

[0074] The receiver, in turn, comprises a corresponding number of depacketizers 212.1 - 212. n, which receive the signalling from the session description generator 208. Each of the depacketizers 212.1 - 212.n also receive an RTP stream of an RTP session. Based on the signalling comprising information about the RTP sessions, and relations thereof, the depacketizers 212.1 - 212.n decapsulate the bitstream, for example, from the RTP payload format. A corresponding number of point cloud decoders 214.1 - 214.n, such as G-PCC decoders, or a corresponding number of decoder threads 214.1 - 214.n parallelly executed by a single decoder decodes the bitstream representing the point cloud data. Based on the signalling comprising information about the RTP sessions, and relations thereof, the composer 216 composes the point cloud data from the received structural elements.

[0075] Figure 2c illustrates an implementation of the architecture of functionalities according to yet another embodiment. Therein, a plurality of point cloud sources 200.1 - 200.n provide point cloud data for generation of a combined and consolidated 3D point cloud. The bitstream from each of the point cloud sources 200.1 - 200.n may be processedseparately as a default, and thus the detector of structural elements 202 may be omitted in this implementation. Otherwise, the processing chain of the bitstreams corresponds to that depicted in Figure 2b.

[0076] As becomes evident from Figures 2a - 2c, another aspect relates to the operation of the receiver, such as a network server or cloud computing node. The operation is illustrated in Figure 3, showing a method comprising receiving (300) signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, wherein said structural elements have properties suitable for one or more predetermined transmission policies; receiving (302) the one or more bitstreams as encapsulated over the one or more streaming format sessions; decapsulating (304) the one or more bitstreams based on said signalling; decoding (306) the one or more bitstreams into point cloud data; and composing (308) a composition of the point cloud data from the point cloud data of the one or more bitstreams.

[0077] Thus, the above implementations and the embodiments related thereto, as disclosed further below, enable real-time packetization and streaming of LiDAR data from source to another network entity. Moreover, LiDAR data packetization may be carried out based on several quality / viewing metrics and characteristics, specific to the use case. The streaming of a mix of different quality levels for different portions of LiDAR data for the bandwidth adaptation is also enabled. Several portions of the LiDAR data may also be prioritized from latency point of view, wherein e.g., very important portions of LiDAR data may be transmitted with lower latency compared to less important portions.

[0078] According to an embodiment, the signalling comprises a syntax element for a grouping parameter, which comprises constituent information about the one or more bitstreams.

[0079] According to an embodiment, the signalling comprises a syntax element for a G- PCC mapping attribute, which comprises indication of the one or more bitstreams and related parameters.

[0080] Thus, new syntax elements may be included in the signalling for handling composition of the plurality of bitstreams, which may be created based on the different properties of the structural elements of the point cloud data. The signalling may be carried out, for example, via SDP without limiting to that protocol only. The embodiments relating to the new syntax elements and the related signalling are illustrated further below.

[0081] A part of the implementation comprises identifying one or more structural elements within the one or more point clouds received from the one or more sensing devices for the purpose determining and executing transmission policies for said structural elements for a given point cloud encoding scheme. The receiver may also provide feedback information to influence the transmission behavior. Moreover, a further part of the implementation provides the necessary signaling to synchronize sender and receiver of the encoded point cloud media stream. In the following, these four aspects - identifying structural elements, defining a transmission policy, specifying feedback, and signaling -are discussed more in detail.Structural Elements

[0082] According to an embodiment, the method comprises identifying a given structural element to belong to one or more predetermined classes; and assigning a value of relevance for each of said one or more predetermined classes of the given structural element.

[0083] Thus, determining structural elements for consideration during transmission seeks to identify one or more classes of such elements and assign a score to each of these classes for subsequent processing. Such a score may reflect relevance for a scene represented by a point cloud and be used later to associate the said structural element with one or more properties for transmission (e.g., urgency, priority, encoding, precision, scheduling, reliability mechanisms, QoS tagging), retention (e.g., how long to store such element for retransmission or reference), or further purposes.

[0084] A variety of means may be applied for structuring the point cloud into structural elements, including, but not limited to:Statistical sampling;Clustering methods;Inference using one or more object detection schemes;Leveraging field-of-view (FoV), possibly also considering areas adjacent to the FoV with different treatment depending on the distance or direction from the FoV; Structural elements determined by movement, tilting, panning, or zooming direction of a data source;Structural elements informed by sensors (e.g., camera, radar, microphone, accelerometers);Structural elements informed by information received from third parties (e.g., other devices).

[0085] The structuring operation may be carried out on an individual point cloud or a time sequence of point clouds. The structuring operation may be carried out repeatedly, iteratively, or recursively.

[0086] The resulting structural elements may be disjoint or overlapping. The resulting structural elements may or may not form a hierarchy, for example, one structural element may be a refinement of another structural element (e.g., provide a broader view or more detail). The resulting structural elements may be used to adjust scanning and / or representation precision.

[0087] According to an embodiment, the method comprises encoding the structural elements as one or more separate point cloud.

[0088] According to an embodiment, the method comprises encoding the structural elements as one or more identifiable parts of a point cloud.

[0089] Referring to Figure 2b, structural elements detected by the detector 202 in the source point cloud are encoded as separate point clouds or as identifiable parts of a point cloud, possibly at different resolutions. This may be implemented by the discernible encoding units 204.1 - 2O4.n, where the encoding of individual units may be independent or exhibit cross-unit dependencies or spatial / temporal ones. The encoded separate or identifiable parts of a point cloud are fed into the packetizers 206.1 - 206. n for packetization and transmission.Transmission

[0090] According to an embodiment, the method comprises associating the structural elements with one or more attributes; and generating a signalling about the attributes to be provided in or along the encoded one or more bitstreams comprising the structural elements.

[0091] The structural elements may be associated with attributes, which may be signalled along with the encoding of the structural elements. The associated attributes may include priority, precision, etc. The corresponding signalling may be, for example, explicitly signalled by the packetizers 206.1 - 2O6.n or implicitly inferred from the transmission order. The session description generator 208 composes the signalling from each packetizer 206.1 - 2O6.n and decides, which aspects are to be signalled via SDP.

[0092] According to an embodiment, the method comprises packetizing the encoded one or more bitstreams comprising the structural elements based on the associated signalling and / or the value of relevance of the structural elements.

[0093] The structural elements may be used in the media packetization scheme, i.e., in the corresponding RTP payload(s) generated by packetizers 206.1 - 206. n. Specifically, the structural elements, their associated signaling, and / or their score values may be leveraged for the following purposes, among others:They may inform how to group structural elements into individual packets. The structural elements may be used to determine transmission order.They may inform the error protection properties assigned to different packets or parts of their payload, e.g., which packets are transmitted reliably, to which packets to apply FEC, which parts of payloads to protect via redundancy, FEC, retransmission, or other mechanisms.They may be used for state or to infer delivery deadlines.They may inform other optimizations when packetizing.

[0094] Transmission of different structural elements within RTP may follow, in principle, two different approaches:1) as a single RTP stream; or2) as several RTP streams.

[0095] An RTP stream may be characterized as having all packets transmitted within the same RTP session or the same transport connection. Alternatively, an RTP stream may also use other means for differentiation, such as distinct payload formats, identifiers (e.g., SSRC identifiers), flow identifiers, or QoS markers (such as diffserv code points).Approach 1 - Transmission as a single RTP stream

[0096] According to the approach 1 , all structural elements are encoded and transmitted as RTP payloads within the same RTP stream.

[0097] According to an embodiment, the method comprises indicating reliability requirements per a data packet in payload-specific header fields of the packetized bitstream.

[0098] RTP payload-specific headers are defined to indicate the reliability requirements per packet. Possibly, the reliability requirements may be repeated or otherwise encoded also in other packets so that a receiver can tell that a packet requiring reliability was lost. Thus, a retransmission may be initiated e.g., by sending a NACK or a selective ACK.

[0099] According to an embodiment, the payload-specific header field may comprise a 32-bit field that describes as 1- or 2-bit combination the reliability needs of the 32 or 16 packet window, from -16 to +15 packets relative to its own sequence number. According to an embodiment, the payload-specific header field may be used to indicate -15+N to N, for a 16 packets window, where N is a parameter for session negotiation / description. Different sequence number spaces per reliability level than described above may also be used.

[0100] Reliability may also extend beyond an individual packet and indicate e.g., crosspayload and / or cross-packet dependencies. Dependencies may be used, besides, to indicate reliability demands, but also to indicate that a packet has less value (and may not need reliability), if another packet that it depends on could not be delivered.

[0101] The repair mechanisms employed to cope with lost packets (such as NACKs, FEC, etc.) may be negotiated orthogonally to the media type as part of the RTP session.

[0102] Reliability mechanisms are not necessarily limited to time -bounded repair to meet real-time demands of a media stream. In addition, a reliability mechanism may ensure reliable delivery of payload at timeline independent of the real-time stream, referred to aslong-term reliability. In practice, priority for transmission would be given to the real-time stream over such long-term reliability that allow less than best effort support for long-term repair for archival purposes. The allocation of the transmission capacity may be carried out as LEDBAT-style operation, as disclosed in RFC 6817, to use transmission capacity only if not fully utilized by the media stream. Media repair may also be performed using out-of- band mechanisms by storing the payloads persistently at the sender and allowing for later retrieval of selected packets by means of other protocols, possibly reliable ones such as TCP, QUIC (disclosed in RFC9000), HTTP, or FTP. In another embodiment, special type of packets that can be called probing packets could be sent for added reliability.

[0103] Prior to transmission, individual packets may be explicitly marked with respect to their value of relevance (i.e. score) and / or reliability and / or real-time needs. The marking may be performed, for example, as disclosed in the extension header draft for framemarking [draft-ietf-avtext-framemarking-15] to determine if a packet can be dropped. The packet marking may also involve L2 or L3 mechanisms such a VLAN tags, TSN traffic types, or differentiated services code points in the IP QoS field.

[0104] When using QUIC as the underlying transport protocol, reliable QUIC streams and unreliable QUIC datagrams may be used in a meaningful combination to achieve the desired reliability level. This may include transmitting a given packet as a datagram for the first k attempts (k > 1) and then reverting to a reliable stream, possibly at low priority for long-term reliability.

[0105] This dynamic adjustment of the score (and thus reliability, priority, etc.) of a given packet and its respective payload may be applied irrespective of the underlying transport and be mapped to appropriate transmission behaviour.

[0106] Encoding and transmitting all structural elements within the same RTP stream advantageously requires only a single port number (pair).Approach 2) Transmission as multiple RTP streams

[0107] According to the approach 2, the structural elements are carried in packets in multiple independent RTP streams. The relationship between the multiple RTP streams is defined via session metadata (e.g., SDP). This may include, for example, the transmissionpriority, reliability, capacity sharing, possibly congestion control algorithm in use, among other properties. Specifically:

[0108] Each RTP stream may define its own time bounds for media transmission and delivery, including dejitter and repair buffer. Each RTP stream may define its own reliability demands. According to an embodiment, a first stream may be dedicated to realtime transmission and a second stream may be deemed particularly reliable. Packets will first be transmitted over the first stream meet the real-time demands of the application. Packets not received within the targeted time bounds of the first stream are considered lost for immediate, i.e., real-time, use. Nevertheless, they may be retransmitted via the second (reliable) stream for archival purposes, possibly at lower bit rate.

[0109] According to another embodiment, each class of structural elements is carried in its own RTP stream. The mapping of structural elements to a given streams may be based on stream-specific reliability properties, transmission priorities, and time bounds, among others.

[0110] With structural elements spread across different streams, the multiple streams may feature independent RTP sequence number spaces for proper protocol operation per stream. To ensure proper ordering and correct interpretation of the structural elements at the receiver side, further correlation information may need to be signalled. Such signalling may be embedded in the encoded information of the structural elements. This may include assigning sequence numbers of structural elements for recomposition that span streams.

[0111] The approach 2, where the structural elements are carried in packets in multiple independent RTP streams, is advantageous in that different streams may provide clear QoS treatment indications, which may be easily signalled using different port numbers, e.g., for integrated services. Alternatively, or in addition, the treatment mechanisms defined in the approach 1 may be applied here as well.

[0011] It is also advantageous that all packets within a stream are treated coherently, which may simplify implementation and processing inside the network.

[0113] Explicit signaling may be used to correlate the different streams and error handling mechanisms may be defined to address cases, where one or more streams fails.

[0114] Depending on the stream multiplexing in use, several port numbers may be needed.Feed forward and Feedback

[0115] The sender may provide information (up front and updated later) about the media streams to be transmitted to the receiver and the receiver may use this information to make choices on how to process media and how to inform the sender in case of transmission errors (e.g., lost packets). This allows constructing a dynamic feedback loop, including both control and data packets.[01 .16] Multiple control loops may operate at the same time and possibly at different time scales. One example is a control loop using the signaling channel (e.g., SDP) to adjust the negotiated stream composition at coarse granularity. Another example is using RTP and RTCP packets to convey information about transmission and reception status and inform or ask sender and / or receiver about specific action or to carry out specific actions.Signaling

[0117] Regardless of using either of the two transport approaches 1 or 2 above, signalling may be defined for the sender and the receiver to agree on the transmission parameters to be used, or for the sender to declare such parameters so that the receiver knows what to expect. In the following, Session Description Protocol (SDP) [RFC8866] is used as sample notation to represent such signalling information. It is, however, noted that different representation formats may be used instead, including but not limited to JSON, CBOR, XML, manifest files, and others.

[0118] A minimal description of a media stream carrying point cloud data (e.g. compressed using G-PCC) could be: m=application 54321 RTP / AVPF 95 c=IN IP4 98.76.54.32 a=rtpmap: 95 gpcc / 90000

[0119] This indicates the use of the top-level MIME type “application”, the choice of the RTP profile, e.g. AVPF [RFC4585] in this case, and indicates the transport layer parameters such as IP address and port number. It also indicates that RTP payload number 95 shall be used to refer to a G-PCC encoded media stream (sampled at a 90 kHZ clock inthis case). It is understood that other transports may require further or different parameters, e.g., an SNI value indicated in a secure connection establishment.

[0120] The minimal description would need to exist for every RTP session that is to carry point cloud data (e.g. LiDAR data) according to the embodiments. If multiple RTP sessions are used according to the approach 2, each of the sessions needs its own session description (an m= line plus attributes in SDP).

[0121] Moreover, the multiple sessions may need to be put into relation to each other. This may involve indicating that the multiple sessions form a joint media transport, on one hand, and expressing the dependencies across the RTP sessions, on the other hand.

[0122] SDP supports a grouping mechanism via the a=group attribute at the conference level. In the following example, a new attribute value “MS” (multiple streams) is used as the SDP group attribute. One specific feature for grouping frames, decoding dependency (DDP), may optionally be used to express if structural elements carried in one session depend on structural elements carried in another session. The a=mid attributes are used to identify the RTP sessions and tie them together via the grouping. The remaining attributes are just placeholders to indicate that the different RTP sessions exhibit different transmission and processing properties.A=group:MS 1 2 3 / / newly defined as no currently existing grouping framework appears a good match a=group:DDP....’ / / decoding dependencies could be used to express semantic relations m=application 54321 RTP / AVPF 95 a=rtpmap: 95 gpcc / 90000 a=mid: 1 a=pcc-reliability: ... prob delay 100 rtx 1 a=pcc-adapt: ... none m=application 54322 RTP / AVPF 96 a=rtpmap: 96 gpcc / 90000 a=mid:2 a=pcc-reliability: ... prob delay 100 rtx 1 a=pcc-adapt: ... spatialm=application 54323 RTP / AVPF 97 a=rtpmap: 97 gpcc / 90000 a=mid:3 a=pcc-reliability: ... full a=pcc-adapt... none

[0123] The above signalling describes the minimum set of attributes to indicate whether the structural elements are carried in a single or multiple RTP stream(s). However, the above signalling may also be supplemented with further attributes to further describe aspects or properties of the respective streams. Examples of further attributes are discussed in the following.

[0124] For each stream, it may be useful to describe the resolution of the structural elements of the point cloud carried in that stream (line 1 below) and / or the coverage of the point cloud (line 2). It may also be useful to provide a distance coverage (not shown in line 2), indicating how far e.g. a LiDAR can scan, e.g., as a measure of accuracy as a function of distance.1: a=pcc-resolution:#lines #points-per-line #depth-resolution - point cloud resolution2: a=pcc-coverage:h-angle v-angle

[0125] It may also be useful to provide an anchor reference for the point cloud, relative to which coordinates are given. This may be useful, for example, if an absolute reference, e.g., with respect to the environment (room, building, GPS position) shall be provided for the point cloud or if multiple point cloud sources are available that may be located relatively to one another (e.g., for later point cloud fusion). If the anchor is empty, specific semantics may be inferred, e.g., that all other coordinates are provided only as relative coordinates. Line 3 shows a possible representation with a type indicating that multiple representation of coordinates are conceivable. Two examples for such representations are given: "metric” to provide coordinates in metrics units (other unit systems are equally possible), “gps” offers coordinates using the Global Positioning System. Obviously, alternative choices for coordinates are possible as long as they fulfil the accuracy needs of the given application. Besides the position, it is useful to also provide the orientation ofthe respective point cloud sources respectively its coordinate system origin (line 4). Those may be specified, for example, in decimal degrees or radiant.3: a=pcc-anchor:<type> <type-specific format>Type "metric" <x> <y> <z> — spatial coordinates x, y, z, e.g., measured in millimetersType "gps" <lat> <lon> <alt> — global positioning coordinates, standard representation4: a=pcc-orientation: <tilt> <pan> — e.g., in decimal degrees

[0126] Furthermore, when multiple LiDARs are in operation, or a single LiDAR generates multiple point clouds the resulting structural elements need to be put in relation to one another. To this end the position of a point cloud (e.g., a LiDAR) may be specified as shown in line 5, relative to the anchor or a common coordinate system or another LiDAR, which then also includes the relative orientation (if any).5: a=pcc-position:<delta-x>, <delta-y>, <delta-z> <delta-tilt> <delta-pan>

[0127] This information may be valid within the set of structural elements (e.g., one group of m= lines in SDP) or across sets of structural elements.

[0128] A LiDAR may provide a variety of attributes for each point in a point cloud or a group thereof. For a given application, a sender and a receiver may negotiate which of these attributes shall actually be transmitted. For example, it may be advantageous to limit the number of attributes to reduce the necessary transmission capacity. For example, color or reflectivity of a point may not be needed if obstacle detection is the primary goal. There are multiple ways how to represent such choices. One option is listing the attributes one by one, as shown in line 6. Another option is defining well-known profiles that determine which attributes are understood, as shown in line 7. These properties may be declared by the sender or negotiated between sender and receiver, e.g., using the offer / answer mechanism or another means of capability negotiation. Multiple options to choose from may be provided.6: a=pcc-point-attr:"attr" <attrl>, <attr2>, ...7: a=pcc-point-attr: "profile" <profile-name>

[0129] Similar consideration may apply to other metadata per point, as shown in lines 8 and 9, again for individual metadata and for profile, respectively:8: a=pcc-point -metadata: "meta" <metal>, <meta2>, ...9: a=pcc-point -metadata: "profile" <profile>

[0130] Structural elements are grouped into classes that exhibit the same (transmission) properties: CSE:= Class of Structural Elements. For these classes, it can be defined how transmission of the structural elements is organized. If, similar to video coding, a differentiating is made between intra-coded and inter-coded frames, the corresponding frame structure could be communicated.10: a=pcc-frame-structure: “- "I frames" vs. "P frames"

[0131] An overall transmission may also indicate how many focus areas (e.g., FoV vs the rest) are available (see line 11):11 : a=pcc-focus:<k> — number of levels of focus area

[0132] To allow for dynamic rate adjustment, a sender may indicate which precision levels (line 12) available for rate reduction, possibly defined per CSE, and then provide again attribute-specific or profile-style information about each level (not shown). Precision levels could use floating point or integer resolution indicated on such level, or specify them by the number of bits or other feasible metrics. The sender may also indicate that a full precision level is available (line 13), which could also be combined with the others (not shown).12: a=pcc-precision: levels <pl>, <p2>, ...13: a=pcc-precision: exact

[0133] If several classes of structural elements are available, a sender may wish to describe them and their relationship (e.g., dependencies)14: a=pcc-structure: ...

[0134] It may be advantageous to support a number of different priorities for transmission and specify them (line 15) and then define for each element or stream which priority shall be used (not shown).15: a=pcc-priorities:<k>

[0135] Similarly, different reliability levels may be supported and those may be indicated per class of structural elements or per stream, as shown in line 16. The example levels listed below indicate full reliability (“full”), probabilistic reliability (“prob”). For either, an optional maximum delay bound and optionally associated reliability mechanismssuch a retransmission (“rtx”) or forward error correction (“fee”) with associated FEC algorithm and parameters, “none” may be chosen to not select any reliability mechanism. Further levels are conceivable.16: a=pcc-re liability :<cse>: levelWith level “= "full"“| "prob“ ["delay" <max>]“[("rtx" <k>“| "fee" <desc>)]“| "none"

[0136] The sender may also indicate which adaptivity support it can provide to reduce capacity. This may include the aforementioned precision but also more general spatial or temporal resolution (as is also done for video codecs). Further options are possible, e.g., limited distance range or depth, reducing the number of sources, among others.17: a=pcc-adapt: flexibilityWith flexibility “= "spatial" <param-range>“| "temporal" | ...

[0137] The reliability mechanisms may be used to inform the receiver which kind of feedback it can provide to the sender. This may include generic feedback, such as (negative or positive) acknowledgements (of selected packets or structural elements), reference point cloud selection, using spatio-temporal tradeoffs, etc. They may be carried in RTCP feedback packets, as described above. Moreover, codec-specific feedback mechanisms are available. All these may be negotiated via signaling so that sender and receiver can act accordingly. a. 18: a=pcc-feedback:<fbl>, <fb2>, ...

[0138] According to an embodiment, the method comprises updating the signalling based on capturing conditions of one or more point cloud sources.

[0139] While the above information may be indicated during stream setup, it is appreciated that this information may change dynamically at different time scales and thus may require updating. While this may be achievable using out-of-band signalling updates such as in SDP, it is also understood that time scales may exist that require fast updates, including updates tightly coupled to the media transmission. Hence, any of this information may also be carried within media packets (header, media payload) or control packets. The underlying reasons for the updates may be manifold: for example, in case of moving vehicles, the anchor information may change frequently, whereas in case of a robot with a LiDAR sensor mounted on a moving part, its position will change during operation.

[0140] It is also noted that the signalling and the feedback are not limited to be carried for use within / among the structural elements point cloud media streams and their associated RTP sessions only, but they can also refer to other suitable media. Point clouds may obviously have a variety of different sources, including 2D or 3D video with possible (Al-based) inference to construct point clouds.

[0141] The embodiments relating to the encoding aspects may be implemented in an apparatus comprising: means for receiving data from one or more point cloud sources; means for determining one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; means for encoding the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; means for encapsulating the one or more bitstreams into a streaming format; means for generating signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; means for transmitting the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and means for signalling said information to the client.

[0142] According to an embodiment, the apparatus comprises means for identifying a given structural element to belong to one or more predetermined classes; and means for assigning a value of relevance for each of said one or more predetermined classes of the given structural element.

[0143] According to an embodiment, the apparatus comprises means for encoding the structural elements as one or more separate point clouds.

[0144] According to an embodiment, the apparatus comprises means for encoding the structural elements as one or more identifiable parts of a point cloud.

[0145] According to an embodiment, the apparatus comprises means for associating the structural elements with one or more attributes; and means for generating a signalling about the attributes to be provided in or along the encoded one or more bitstreams comprising the structural elements.

[0146] According to an embodiment, the apparatus comprises means for packetizing the encoded one or more bitstreams comprising the structural elements based on the associated signalling and / or the value of relevance of the structural elements.

[0147] According to an embodiment, the apparatus comprises means for indicating reliability requirements per a data packet in payload-specific header fields of the packetized bitstream.

[0148] According to an embodiment, the apparatus comprises means for updating the signalling based on capturing conditions of the one or more point cloud sources.

[0149] According to an embodiment, said signalling is carried out according to Session Description Protocol.

[0150] According to an embodiment, the signalling comprises a syntax element for a grouping parameter, which comprises constituent information about the one or more bitstreams.

[0151] According to an embodiment, the signalling comprises a syntax element for a G- PCC mapping attribute, which comprises indication of the one or more bitstreams and related parameters.

[0152] According to an embodiment, said streaming format is Real Time Transfer Protocol.

[0153] The embodiments relating to the encoding aspects may likewise be implemented in an apparatus comprising at least one processor and at least one memory, said at least one memory stored with computer program code thereon, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: receive data from one or more point cloud sources; determine one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; encode the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; encapsulate the one or more bitstreams into a streaming format; generate signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structuralelements in said one or more streaming format sessions; transmit the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and signal said information to the client.

[0154] According to an embodiment, the apparatus comprises computer program code configured to cause the apparatus to identify a given structural element to belong to one or more predetermined classes; and assign a value of relevance for each of said one or more predetermined classes of the given structural element.

[0155] According to an embodiment, the apparatus comprises computer program code configured to cause the apparatus to encode the structural elements as one or more separate point clouds.

[0156] According to an embodiment, the apparatus comprises computer program code configured to cause the apparatus to encode the structural elements as one or more identifiable parts of a point cloud.

[0157] According to an embodiment, the apparatus comprises computer program code configured to cause the apparatus to associate the structural elements with one or more attributes; and generate a signalling about the attributes to be provided in or along the encoded one or more bitstreams comprising the structural elements.

[0158] According to an embodiment, the apparatus comprises computer program code configured to cause the apparatus to packetize the encoded one or more bitstreams comprising the structural elements based on the associated signalling and / or the value of relevance of the structural elements.

[0159] According to an embodiment, the apparatus comprises computer program code configured to cause the apparatus to indicate reliability requirements per a data packet in payload-specific header fields of the packetized bitstream.

[0160] According to an embodiment, the apparatus comprises computer program code configured to cause the apparatus to update the signalling based on capturing conditions of the one or more point cloud sources.

[0161] According to an embodiment, said signalling is carried out according to Session Description Protocol.

[0162] According to an embodiment, the signalling comprises a syntax element for a grouping parameter, which comprises constituent information about the one or more bitstreams.

[0163] According to an embodiment, the signalling comprises a syntax element for a G- PCC mapping attribute, which comprises indication of the one or more bitstreams and related parameters.

[0164] According to an embodiment, said streaming format is Real Time Transfer Protocol.

[0165] The embodiments relating to the decoding aspects may be implemented in an apparatus comprising means for receiving signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, wherein said structural elements have properties suitable for one or more predetermined transmission policies; means for receiving the one or more bitstreams as encapsulated over the one or more streaming format sessions; means for decapsulating the one or more bitstreams based on said signalling; means for decoding the one or more bitstreams into point cloud data; and means for composing a composition of the point cloud data from the point cloud data of the one or more bitstreams.

[0166] The embodiments relating to the decoding aspects may likewise be implemented in an apparatus comprising at least one processor and at least one memory, said at least one memory stored with computer program code thereon, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: receive signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, wherein said structural elements have properties suitable for one or more predetermined transmission policies; receive the one or more bitstreams as encapsulated over the one or morestreaming format sessions; decapsulate the one or more bitstreams based on said signalling; decode the one or more bitstreams into point cloud data; and compose a composition of the point cloud data from the point cloud data of the one or more bitstreams.

[0167] Such apparatuses may comprise e.g. the functional units disclosed in any of the Figures 2a - 2c for implementing the embodiments.

[0168] In the above, some embodiments have been described with reference to encoding. It needs to be understood that said encoding may comprise one or more of the following: encoding source image data into a bitstream, encapsulating the encoded bitstream in a container file and / or in packet(s) or stream(s) of a communication protocol, and announcing or describing the bitstream in a content description, such as the Media Presentation Description (MPD) of ISO / IEC 23009-1 (known as MPEG-DASH) or the IETF Session Description Protocol (SDP). Similarly, some embodiments have been described with reference to decoding. It needs to be understood that said decoding may comprise one or more of the following: decoding image data from a bitstream, decapsulating the bitstream from a container file and / or from packet(s) or stream(s) of a communication protocol, and parsing a content description of the bitstream.

[0169] In the above, where the example embodiments have been described with reference to an encoder or an encoding method, it needs to be understood that the resulting bitstream and the decoder or the decoding method may have corresponding elements in them. Likewise, where the example embodiments have been described with reference to a decoder, it needs to be understood that the encoder may have structure and / or computer program for generating the bitstream to be decoded by the decoder.

[0170] In general, the various embodiments of the invention may be implemented in hardware or special purpose circuits or any combination thereof. While various aspects of the invention may be illustrated and described as block diagrams or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0171] Embodiments of the inventions may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highlyautomated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0172] Programs, such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre stored design modules. Once the design for a semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.

[0173] The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of the exemplary embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended examples. However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention.

Claims

CLAIMS1. An apparatus comprising: means for receiving data from one or more point cloud sources; means for determining one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; means for encoding the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; means for encapsulating the one or more bitstreams into a streaming format; means for generating signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; means for transmitting the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and means for signalling said information to the client.

2. The apparatus according to claim 1, comprising means for identifying a given structural element to belong to one or more predetermined classes; and means for assigning a value of relevance for each of said one or more predetermined classes of the given structural element.

3. The apparatus according to claim 1 or 2, comprising means for encoding the structural elements as one or more separate point clouds.

4. The apparatus according to claim 1 or 2, comprisingmeans for encoding the structural elements as one or more identifiable parts of a point cloud.

5. The apparatus according to any preceding claim, comprising means for associating the structural elements with one or more attributes; and means for generating a signalling about the attributes to be provided in or along the encoded one or more bitstreams comprising the structural elements.

6. The apparatus according to any preceding claim, comprising means for packetizing the encoded one or more bitstreams comprising the structural elements based on the associated signalling and / or the value of relevance of the structural elements.

7. The apparatus according to any preceding claim, comprising means for indicating reliability requirements per a data packet in payloadspecific header fields of the packetized bitstream.

8. The apparatus according to any preceding claim, comprising means for updating the signalling based on capturing conditions of the one or more point cloud sources.

9. The apparatus according to any preceding claim, wherein said signalling is carried out according to Session Description Protocol.

10. The apparatus according to any preceding claim, wherein the signalling comprises a syntax element for a grouping parameter, which comprises constituent information about the one or more bitstreams.

11. The apparatus according to any preceding claim, wherein the signalling comprises a syntax element for a G-PCC mapping attribute, which comprises indication of the one or more bitstreams and related parameters.

12. The apparatus according to any preceding claim, wherein said streaming format is Real Time Transfer Protocol.

13. A method comprising: receiving data from one or more point cloud sources; determining one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; encoding the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements; encapsulating the one or more bitstreams into a streaming format; generating signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; transmitting the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and signalling said information to the client.

14. An apparatus comprising at least one processor and at least one memory, said at least one memory stored with computer program code thereon, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: receive data from one or more point cloud sources; determine one or more structural elements within the received data, said structural elements having properties suitable for one or more predetermined transmission policies; encode the received data to one or more bitstreams, wherein the encoding is at least partly controlled based on the properties of said one or more structural elements;encapsulate the one or more bitstreams into a streaming format; generate signalling comprising information about one or more streaming format sessions for said one or more bitstreams, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of said one or more structural elements in said one or more streaming format sessions; transmit the encapsulated one or more bitstreams over the one or more streaming format sessions to a client; and signal said information to the client.

15. The apparatus according to claim 14, comprising computer program code configured to cause the apparatus to identify a given structural element to belong to one or more predetermined classes; and assign a value of relevance for each of said one or more predetermined classes of the given structural element.

16. The apparatus according to claim 14 or 15, comprising computer program code configured to cause the apparatus to encode the structural elements as one or more separate point clouds.

17. The apparatus according to claim 14 or 15, comprising computer program code configured to cause the apparatus to encode the structural elements as one or more identifiable parts of a point cloud.

18. The apparatus according to any of claims 14 - 17, comprising computer program code configured to cause the apparatus to associate the structural elements with one or more attributes; and generate a signalling about the attributes to be provided in or along the encoded one or more bitstreams comprising the structural elements.

19. The apparatus according to any of claims 14 - 18, comprising computer program code configured to cause the apparatus to packetize the encoded one or morebitstreams comprising the structural elements based on the associated signalling and / or the value of relevance of the structural elements.

20. The apparatus according to any of claims 14 - 19, comprising computer program code configured to cause the apparatus to indicate reliability requirements per a data packet in payload-specific header fields of the packetized bitstream.

21. The apparatus according to any of claims 14 - 20, comprising computer program code configured to cause the apparatus to update the signalling based on capturing conditions of the one or more point cloud sources.

22. The apparatus according to any of claims 14 - 21, wherein said signalling is carried out according to Session Description Protocol.

23. The apparatus according to any of claims 14 - 22, wherein the signalling comprises a syntax element for a grouping parameter, which comprises constituent information about the one or more bitstreams.

24. The apparatus according to any of claims 14 - 23, wherein the signalling comprises a syntax element for a G-PCC mapping attribute, which comprises indication of the one or more bitstreams and related parameters.

25. The apparatus according to any of claims 14 - 24, wherein said streaming format is Real Time Transfer Protocol.

26. An apparatus comprising: means for receiving signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, whereinsaid structural elements have properties suitable for one or more predetermined transmission policies; means for receiving the one or more bitstreams as encapsulated over the one or more streaming format sessions; means for decapsulating the one or more bitstreams based on said signalling; means for decoding the one or more bitstreams into point cloud data; and means for composing a composition of the point cloud data from the point cloud data of the one or more bitstreams.

27. A method comprising: receiving signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming format sessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, wherein said structural elements have properties suitable for one or more predetermined transmission policies; receiving the one or more bitstreams as encapsulated over the one or more streaming format sessions; decapsulating the one or more bitstreams based on said signalling; decoding the one or more bitstreams into point cloud data; and composing a composition of the point cloud data from the point cloud data of the one or more bitstreams.

28. An apparatus comprising at least one processor and at least one memory, said at least one memory stored with computer program code thereon, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: receive signalling comprising information about one or more streaming format sessions for one or more bitstreams representing point cloud data, said information comprising at least mapping information between said one or more streaming formatsessions and said one or more bitstreams and relations of one or more structural elements of the point cloud data in said one or more streaming format sessions, wherein said structural elements have properties suitable for one or more predetermined transmission policies; receive the one or more bitstreams as encapsulated over the one or more streaming format sessions; decapsulate the one or more bitstreams based on said signalling; decode the one or more bitstreams into point cloud data; and compose a composition of the point cloud data from the point cloud data of the one or more bitstreams.

Images