Point cloud compression using non-orthogonal projection

Abstract

A method of operating a decoding device for point cloud decoding includes receiving a compressed bitstream. The method also includes decoding the compressed bitstream into a two-dimensional (2-D) frame representing a three-dimensional (3-D) point cloud. Each of the 2-D frames includes a set of patches, and each patch includes a cluster of points in the 3-D point cloud. The cluster of points corresponds to attributes associated with the 3-D point cloud. A patch in the patch set, the patch set, and the 2-D frame correspond to respective access levels representing the 3-D point cloud. The method also includes identifying a first flag and a second flag. In response to identifying the first flag and the second flag, the method includes reading metadata from the bitstream. The method also includes generating a 3-D point cloud based on the metadata and using the set of 2-D frames.

Description

Technical Field

[0001] The present disclosure generally relates to compression and encoding of multimedia data. More specifically, the present disclosure relates to an apparatus and method for compressing and decompressing point clouds. Background Art

[0002] With the ready availability of powerful handheld devices such as smartphones, three hundred and sixty degree (360°) video is becoming a new way to experience immersive video. 360° video brings consumers an immersive, "real life", "immersive" experience by capturing a 360° view of the world. Users can interactively change their viewpoint and dynamically view any part of the captured scene or object they desire. Display and navigation sensors track head movements in real time to determine the area of ​​the 360° video that the user wants to view. 360° video provides a three degrees of freedom (3DoF) immersive experience. Six degrees of freedom (6DoF) is the next level of immersive experience, where users can turn their heads and move around in a virtual / augmented environment. Providing a 6DoF experience requires multimedia data (such as point clouds) that is three-dimensional (3-D) in nature.

[0003] Point clouds and meshes are collections of 3-D points that represent a surface model of an object or scene. Point clouds are commonly found in a variety of applications such as gaming, 3-D mapping, visualization, medical applications, augmented reality, virtual reality, autonomous driving, multi-view playback, 6DoF immersive media, to name a few. Point clouds, if not compressed, generally require a large amount of bandwidth for transmission. Due to the large bit rate requirements, point clouds are usually compressed before transmission. The compression hardware and process for point clouds is different from the traditional compression hardware and process used for traditional two-dimensional (2-D) multimedia. Summary of the invention

[0004] Technical Solution

[0005] The present disclosure provides point cloud compression using non-orthogonal projection.

[0006] In a first embodiment, a decoding device for point cloud decoding is provided. The decoding device includes a communication interface and a processor operably coupled to the communication interface. The communication interface is configured to receive a compressed bitstream. The processor is configured to decode the compressed bitstream into a two-dimensional (2-D) frame representing a three-dimensional (3-D) point cloud. Each of the 2-D frames includes a patch set, and each patch in the patch set includes a cluster of points in the 3-D point cloud. The cluster of points corresponds to an attribute associated with the 3-D point cloud. A patch in the patch set, the patch set, and the 2-D frame correspond to various access levels representing the 3-D point cloud. The processor is also configured to identify whether a first flag is included in the compressed bitstream and is set to true (true), and the first flag signaling notification is applied to When the modification of metadata of the first access level in each access level is enabled. The processor is also configured to identify whether a second flag is included in the compressed bitstream and is set to true, and the second flag indicates the presence of metadata associated with the modification of the first access level in each access level. In response to identifying that the first flag and the second flag are included in the compressed bitstream and are set to true, the processor is configured to read metadata from the compressed bitstream. In response to identifying that at least one of the first flag or the second flag is either not included in the compressed bitstream or is set to false, the processor is configured to assign predetermined parameters to the metadata. The processor is also configured to generate a 3-D point cloud based on the metadata and using the 2-D frame set and the metadata.

[0007] In another embodiment, a coding device for point cloud coding is provided. The coding device includes a processor and a communication interface operably coupled to the processor. The processor is configured to generate a 2-D frame representing a 3-D point cloud. Each of the 2-D frames includes a patch set, and each patch in the patch set includes a cluster of points in the 3-D point cloud. The cluster of points corresponds to an attribute associated with the 3-D point cloud. A patch in the patch set, the patch set, and the 2-D frame correspond to various access levels representing the 3-D point cloud. The processor is also configured to generate metadata indicating a modification of one of the various access levels. The processor is also configured to generate a first flag and generate a second flag, the first flag signaling when the modification of the metadata applied to the first access level in each access level is enabled, and the second flag indicates the presence of metadata associated with the modification of the first access level in each access level. The processor is also configured to encode the 2-D frame, the first flag, the second flag, and the metadata to generate a compressed bit stream. The communication interface is configured to transmit the compressed bit stream.

[0008] In yet another embodiment, a method for decoding is provided. The method includes receiving a compressed bitstream. The method also includes decoding the compressed bitstream into a 2-D frame representing a 3-D point cloud. Each of the 2-D frames includes a patch set, and each patch in the patch set includes a cluster of points in the 3-D point cloud. The cluster of points corresponds to an attribute associated with the 3-D point cloud. A patch in the patch set, the patch set, and the 2-D frame correspond to respective access levels representing the 3-D point cloud. The method includes identifying whether a first flag is included in the compressed bitstream and is set to true, the first flag signaling notification when the modification of metadata applied to the first access level in each access level is enabled. The method also includes identifying whether a second flag is included in the compressed bitstream and is set to true, the second flag indicating the presence of metadata associated with the modification of the first access level in each access level. In response to identifying that the first flag and the second flag are included in the compressed bitstream and are set to true, the method includes reading metadata from the compressed bitstream. In response to identifying that at least one of the first flag or the second flag is either not included in the compressed bitstream or is set to false, the method includes assigning predetermined parameters to the metadata. The method also includes generating a 3-D point cloud based on the metadata and using the set of 2-D frames and the metadata.

[0009] For those skilled in the art, other technical features will be clear from the following drawings, description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0011] Figure 1 An example communication system according to an embodiment of the present disclosure is shown;

[0012] Figure 2 An example electronic device according to an embodiment of the present disclosure is shown;

[0013] Figure 3 An example electronic device according to an embodiment of the present disclosure is shown;

[0014] Figure 4 shows a point cloud and a mesh according to an embodiment of the present disclosure;

[0015] Figure 5A An example block diagram of an encoder according to an embodiment of the present disclosure is shown;

[0016] Figure 5B An example block diagram of a decoder according to an embodiment of the present disclosure is shown;

[0017] Fig. 6A , Figure 6B and Figure 6C shows an example 3-D point cloud and a 2-D frame representing the 3-D point cloud according to an embodiment of the present disclosure;

[0018] Fig. 7A An example flow chart illustrating receiving a patch at a scaled resolution according to an embodiment of the present disclosure;

[0019] Figure 7B and Figure 7C The process of creating a bitstream including a patch and receiving the bitstream according to an embodiment of the present disclosure is shown;

[0020] Fig. 8A The process of generating a progressive non-orthogonal projection of a point cloud according to an embodiment of the present disclosure is shown;

[0021] Figure 8B An example block diagram of a decoder according to an embodiment of the present disclosure is shown;

[0022] Fig. 9 An adaptive region of point cloud density according to an embodiment of the present disclosure is shown;

[0023] Fig.10 shows adaptive point size and shape of a point cloud according to an embodiment of the present disclosure;

[0024] Fig.11 shows an example flow chart for decoding a point cloud according to an embodiment of the present disclosure; and

[0025] Fig.12 An example flowchart for encoding a point cloud according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0026] Before the following detailed description, it may be advantageous to set forth the definitions of specific words and phrases used throughout this patent document. The term "coupling" and its derivatives refer to any direct or indirect communication between two or more elements, whether or not these elements are in physical contact with each other. The terms "send", "receive" and "communication" and their derivatives include direct and indirect communication. The terms "include" and "comprise" and their derivatives mean including but not limited to. The term "or" is inclusive, meaning and / or. The phrase "associated with..." and its derivatives refer to including, included in, interconnected with, included in, connected to or connected with, coupled to or coupled with, communicable with, collaborative with, interlaced, juxtaposed, close to, coupled to or combined with, having, having... attributes, having a relationship with, etc. The term "controller" refers to any device, system or part thereof that controls at least one operation. Such a controller can be implemented with hardware or a combination of hardware and software and / or firmware. The function associated with any particular controller can be centralized or distributed, whether local or remote. When used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items may be used, and only one of the items in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.

[0027] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or parts thereof suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disc (CD), a digital video disc (DVD) or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical discs or erasable memory devices.

[0028] Definitions for other specific words and phrases are provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most, instances, such definitions apply to prior and future uses of such defined words and phrases.

[0029] Discussed below Figures 1 to 12 The various embodiments used to describe the principles of the present disclosure in this patent document are merely exemplary and should not be interpreted in any way as limiting the scope of the present disclosure. Those skilled in the art will appreciate that the principles of the present disclosure can be implemented in any appropriately arranged system or device.

[0030] Virtual Reality (VR) is a rendered version of a visual scene. In a specific embodiment, VR is a rendered version of both visual and audio scenes. When an individual moves within a range defined by an application or VR scene, the rendering is designed to simulate the visual stimulation of the real world to the observer or user as naturally as possible, and simulates audio sensory stimulation if available. For example, VR places users in an immersive world that interacts with their head movements. At the video level, VR is achieved by providing a video experience that covers as much field of view (FOV) as possible and synchronizing the perspective of the rendered video with the head movement. Although many different types of devices can provide such an experience, a head-mounted display (HMD) is a popular device that enables users to watch VR scenes and adjust the displayed content based on the movement of the user's head. Typically, HMDs either rely on a dedicated screen that is integrated into the device (tethered) and runs with an external computer, or rely on a device (untethered) inserted into the HMD, such as a smart phone. The first method takes advantage of lightweight screens and benefits from high computing power. In contrast, smart phone-based systems take advantage of higher mobility and can be produced at lower costs. In both cases, the generated video experience is the same. Note that as used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0031] A point cloud is a 3-D representation of an object similar to an object in a VR environment. Similarly, a point mesh is a 3-D representation of an object similar to an object in a VR environment. A point cloud or point mesh may include a scene, such as the surface of an object in the environment.

[0032] Generally, a point cloud is a collection of data points defined by a coordinate system. For example, in a 3-D Cartesian coordinate system, each point in a point cloud is identified by three coordinates, namely X, Y, and Z. When each point is identified by three coordinates, the exact location relative to the origin in a 3-D environment or space is identified. The origin is where the X-axis, Y-axis, and Z-axis intersect. The points in a point cloud typically represent the outer surface of an object. Each point in a point cloud is defined by properties such as the geometric location of each point within the three coordinates and texture (such as color, intensity, normal, reflectivity, etc.).

[0033] Similarly, a 3-D mesh is a 3-D representation of an object similar to a point cloud. A 3-D mesh shows the external structure of an object made up of polygons. For example, a 3-D mesh is a collection of vertices, edges, and faces that define the shape of an object. As another example, a mesh (or point cloud) can be rendered on a spherical coordinate system. In a particular embodiment, each point can be located in an X, Y, Z coordinate within a sphere. Similarly, texture coordinates U and V indicate the location of an image texture. When the object is rendered, the vertices of the mesh, the corresponding texture coordinates, and the texture image are input to a graphics processing unit that maps the mesh onto a 3-D geometric shape. A user can have a FOV located at the center of a virtual sphere and see a portion of the 360° scene corresponding to that viewport. In a particular embodiment, alternative shapes can be used instead of a sphere, such as a cube, an icosahedron, an octahedron, etc. Point clouds and meshes will be referenced below. Figure 4 For more detailed description and discussion.

[0034] Point clouds and meshes are commonly used in a variety of applications, including gaming, 3-D mapping, visualization, medicine, augmented reality, VR, autonomous driving, multi-view playback, 6-DOF immersive media, to name a few. As used hereinafter, the term "point cloud" also refers to "3-D point cloud" and "3-D mesh".

[0035] Due to the size and complexity of the data associated with a single point cloud, transmitting a point cloud from one electronic device to another typically requires a large amount of bandwidth. The transmission of point clouds typically requires specific compression techniques to reduce the size of the data prior to transmission. For example, compressing a point cloud may require specialized hardware or a specific compression algorithm or a combination thereof. In general, compression algorithms for point clouds are different from compression algorithms for other multimedia formats (such as images and video, VR, etc.).

[0036] As described above, a point cloud is a collection of points with attributes, and each attribute requires a certain number of bits. For example, for a point cloud with ten bits of geometry data, thirty bits per point can be used for geometric coordinates (such as X, Y, and Z coordinates), making each coordinate ten bits. Similarly, for a point cloud with eight bits of color data, twenty-four bits per point can be used for color coordinates (such as red, green, and blue (RGB)), making each coordinate an eight-bit color coordinate. Therefore, a single point with ten bits of geometry and eight bits of color attributes uses fifty-four bits. If a frame includes one million points, the number of bits per frame is fifty-four million bits (fifty-four bits per point multiplied by one million points per frame). For each object, there are thirty frames per second. Without compression, 1.62 gigabytes per second (fifty-four million bits per frame multiplied by 30 frames per second) are used to transmit a point cloud from one electronic device to another. Therefore, a lot of compression is required to transmit a point cloud from one electronic device to another.

[0037] Embodiments of the present disclosure take into account that due to bandwidth size constraints, compressing point clouds is necessary to accelerate and improve the transmission of point clouds from a source device to another device (such as a user device). That is, uncompressed point clouds use a large amount of bandwidth for transmission and cause the rendering of point clouds to be delayed. Specific dedicated hardware components can be used to meet real-time requirements or reduce delays or lags in point cloud rendering. Such hardware components are typically expensive. According to embodiments of the present disclosure, using a codec to compress point clouds reduces transmission bandwidth and specific types of hardware. For example, when manipulating a point cloud to fit a 2-D frame, video codecs such as HEVC, AVC, VP9, ​​VP8, JVNET, etc. can be used to compress the point cloud. For example, a point cloud is manipulated from its original 3-D shape into multiple patches that represent the point cloud in 2-D.

[0038] Thus, embodiments of the present disclosure provide systems and methods for manipulating 3-D objects such as point clouds. Manipulating the 3-D object includes projecting the 3-D object onto a 2-D frame so that the frame can be compressed, transmitted to a display device, and the content within the 2-D frame can be reconstructed into the 3-D object and ultimately rendered so that the 3-D object can be viewed by a user. After the 3-D point cloud is projected onto multiple 2-D video frames, various 2-D compression hardware and software components can compress the point cloud. When the point cloud is deconstructed to fit into multiple 2-D frames, the frames can be transmitted using less bandwidth than transmitting the original point cloud.

[0039] 3-D objects can be manipulated by the encoder and transmitted to the decoder, which reconstructs the 3-D object for viewing by the user. The encoder projects the 3-D object onto one or more 2-D frames, compresses the frames, and generates and transmits a compressed bitstream including the compressed frames. The decoder receives the bitstream, decompresses the frames, reconstructs and renders the 3-D object so that the user can view the displayed object.

[0040] According to an embodiment of the present disclosure, an architecture for performing point cloud compression using a video codec is provided. An embodiment of the present disclosure provides an architecture for point cloud compression using a video codec. A point cloud can be deconstructed and multiple 2-D frames can be generated, which include regular patches of the geometry of each point in the point cloud, as well as various attributes or textures of the point cloud. For example, a point cloud can be deconstructed and mapped onto a 2-D frame. The 2-D frame can be compressed using various compressions of video or images or both.

[0041] Embodiments of the present disclosure include methods and systems for manipulating point clouds. For example, embodiments of the present disclosure include manipulating sparse point clouds, scaling point clouds, identifying regions of interest (ROI) of point clouds, and adaptive point size rendering of point clouds. Embodiments of the present disclosure include methods and systems for point cloud compression using non-orthogonal projections. In addition, embodiments of the present disclosure include metadata that supports functions including scaling, offset, rotation, point size, and point shape.

[0042] Figure 1 An example computing system 100 is shown in accordance with the present disclosure. Figure 1 The embodiment of the system 100 shown is for illustration only. Other embodiments of the system 100 may be used without departing from the scope of the present disclosure.

[0043] The system 100 includes a network 102 that facilitates communication between various components in the system 100. For example, the network 102 can communicate Internet Protocol (IP) packets, frame relay frames, asynchronous transfer mode (ATM) cells, or other information between network addresses. The network 102 includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or part of a global network such as the Internet, or any other communication system(s) at one or more locations.

[0044] The network 102 facilitates communication between the server 104 and various client devices 106-116. The client devices 106-116 may be, for example, smart phones, tablet computers, laptop computers, personal computers, wearable devices, or head-mounted displays (HMD). The server 104 may represent one or more servers. Each server 104 includes any suitable computing or processing device capable of providing computing services to one or more client devices. Each server 104 may, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication through the network 102. As described in more detail below, the server 104 may transmit the point cloud to one or more display devices, such as the client devices 106-116. In a particular embodiment, each server 104 may include an encoder.

[0045] Each client device 106-116 represents any suitable computing or processing device that interacts with at least one server or other computing device(s) via the network 102. In this example, the client devices 106-116 include a desktop computer 106, a mobile phone or mobile device 108 (such as a smart phone), a personal digital assistant (PDA) 110, a laptop computer 112, a tablet computer 114, and an HMD 116. However, any other or additional client devices may be used in the system 100. In certain embodiments, each client device 106-116 may include a decoder.

[0046] In this example, some client devices 108-116 communicate indirectly with the network 102. For example, client devices 108 and 110 (mobile device 108 and PDA 110, respectively) communicate via one or more base stations 118, such as cellular network base stations or eNodeBs (eNBs). Mobile device 108 includes a smart phone. In addition, client devices 112, 114, and 116 (laptops, tablet computers, and HMDs, respectively) communicate via one or more wireless access points 120, such as IEEE 802.11 wireless access points. As described in more detail below, HMD 116 can display a 360° scene including one or more point clouds. Note that these are for illustration only, and each client device 106-116 can communicate directly with the network 102, or indirectly with the network 102 via any suitable intermediate device(s) or network(s). In particular embodiments, server 104 or any client device 106 - 114 may be used to compress the point cloud and transmit the data to another client device, such as any client device 106 - 116 .

[0047] In certain embodiments, the mobile device 108 (or any other client device 106-116) can securely and efficiently transmit information to another device, such as, for example, the server 104. When the mobile device 108 (or any other client device 106-116) is attached to a headset via a bracket, the mobile device 108 can be used as a VR display and operate similarly to the HMD 116. The mobile device 108 (or any other client device 106-116) can trigger the transmission of information between itself and the server 104.

[0048] although Figure 1 One example of system 100 is shown, but may be used for Figure 1 Various changes may be made. For example, system 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems come in a variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0049] The processes and systems provided in the present disclosure allow a client device 106-116 or a server 104 to compress, send, receive, render a point cloud, or a combination thereof. For example, the server 104 can then compress the point cloud data and transmit it to the client devices 106-116. For another example, any client device 106-116 can compress the point cloud data and transmit it to any client device 106-116 or the server 104.

[0050] Figure 2 and Figure 3 An example device in a computing system according to an embodiment of the present disclosure is shown. Specifically, Figure 2 An example server 200 is shown, and Figure 3 An example electronic device 300 is shown. The server 200 may represent Figure 1 The server 104, and the electronic device 300 can represent Figure 1 One or more of the client devices 106-116.

[0051] Server 200 may represent one or more local servers, one or more compression servers, or one or more encoding servers. Figure 2 As shown, server 200 includes a bus system 205 that supports communication between at least one processor(s) 210, at least one storage device(s) 215, at least one communication interface 220, and at least one input / output (I / O) unit 225. In certain embodiments, server 200 is an encoder.

[0052] Processor 210 executes instructions that may be stored in memory 230. The instructions stored in memory 230 may include instructions for decomposing a point cloud, compressing a point cloud. The instructions stored in memory 230 may also include instructions for encoding a point cloud to generate a bitstream. The instructions stored in memory 230 may also include instructions for rendering a point cloud on an omnidirectional 360° scene (e.g., via a VR head device (such as Figure 1 The processor 210 may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processor(s) 210 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuits.

[0053] Memory 230 and permanent storage 235 are examples of storage devices 215, which represent any (multiple) structures capable of storing and facilitating retrieval of information (such as data, program code, or other suitable information, whether temporary or permanent). Memory 230 may represent random access memory or (multiple) any other suitable volatile or non-volatile storage device. Permanent storage 235 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, flash memory, or an optical disk.

[0054] Communication interface 220 supports communication with other systems or devices. For example, communication interface 220 may include interfaces that facilitate communication via Figure 1 Communication interface 220 may include a network interface card or wireless transceiver for communicating with network 102. Communication interface 220 may support communication over any suitable physical or wireless communication link(s).

[0055] I / O unit 225 allows input and output of data. For example, I / O unit 225 can provide a connection for user input through a keyboard, mouse, keypad, touch screen, motion sensor, or any other suitable input device. I / O unit 225 can also send output to a display, printer, or any other suitable output device.

[0056] In certain embodiments, server 200 implements compression of the point cloud, which will be discussed in more detail below. In certain embodiments, server 200 generates a plurality of 2-D frames corresponding to the three dimensions of the point cloud. In certain embodiments, server 200 maps the three dimensions of the point cloud into 2-D. In certain embodiments, server 200 generates a compressed bitstream by encoding compressed two-dimensional frames representing the point cloud.

[0057] Note that although Figure 2 Described as indicating Figure 1104, but the same or similar structure can be used for one or more of the various client devices 106-116. For example, the desktop computer 106 or laptop computer 112 can have a Figure 2 The same or similar structure as shown.

[0058] Figure 3 An electronic device 300 according to an embodiment of the present disclosure is shown. Figure 3 The embodiment of the electronic device 300 shown is for illustration only, and other embodiments may be used without departing from the scope of the present disclosure. The electronic device 300 may have a variety of configurations, and Figure 3 The scope of the present disclosure is not limited to any particular implementation of the electronic device. In certain embodiments, Figure 1 One or more of the client devices 106-116 may include the same or similar configuration as the electronic device 300. In certain embodiments, the electronic device 300 is an encoder, a decoder, or both.

[0059] In certain embodiments, the electronic device 300 may be used for data transmission, image or video compression, image or video decompression, encoding, decoding, and media rendering applications. The electronic device 300 may be a mobile communication device, such as, for example, a wireless terminal, a desktop computer (similar to a Figure 1 desktop computer 106), mobile devices (similar to Figure 1 Mobile devices 108), PDAs (similar to Figure 1 PDA110), laptop computer (similar to Figure 1 laptop computer 112), tablet computer (similar to Figure 1 Tablet computer 114), head mounted display (similar to Figure 1 HMD 116) etc.

[0060] like Figure 3 As shown, the electronic device 300 includes an antenna 305, a radio frequency (RF) transceiver 310, a transmission (TX) processing circuit 315, a microphone 320, and a reception (RX) processing circuit 325. The electronic device 300 also includes a speaker 330, one or more processors 340, an input / output (I / O) interface (IF) 345, an input 350, a display 355, and a memory 360. The memory 360 includes an operating system (OS) 361, one or more applications 362, and a point cloud 363.

[0061] The RF transceiver 310 receives an input RF signal sent by another component on the system from the antenna 305. For example, the RF transceiver 310 receives an RF signal sent via a Bluetooth or WI-FI signal from an access point (such as a base station, a WI-FI router, a Bluetooth device) of a network 102 (such as WI-FI, Bluetooth, a cellular network, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network). The RF transceiver 310 can down-convert the input RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuit 325, which generates a processed baseband signal by filtering, decoding, or digitizing the baseband or intermediate frequency signal, or a combination thereof. The RX processing circuit 325 sends the processed baseband signal to the speaker 330 (such as for voice data) or the processor 340 for further processing (such as for web browsing data).

[0062] The TX processing circuit 315 receives analog or digital voice data from the microphone 320, or receives other output baseband data from the processor 340. The output baseband data may include network data, email, or interactive video game data. The TX processing circuit 315 encodes, multiplexes, digitizes, or a combination thereof, the output baseband data to generate a processed baseband or intermediate frequency signal. The RF transceiver 310 receives the output processed baseband or intermediate frequency signal from the TX processing circuit 315, and up-converts the baseband or intermediate frequency signal into an RF signal that is transmitted via the antenna 305.

[0063] The processor 340 may include one or more processors or other processing devices and executes an OS 361 stored in the memory 360 to control the overall operation of the electronic device 300. For example, the processor 340 may control the reception of forward channel signals and the transmission of backward channel signals through the RF transceiver 310, the RX processing circuit 325, and the TX processing circuit 315 according to well-known principles. The processor 340 is also capable of executing other applications 362 residing in the memory 360, such as decompressing and generating received point clouds.

[0064] Processor 340 may execute instructions stored in memory 360. Processor 340 may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. For example, in some embodiments, processor 340 includes at least one microprocessor or microcontroller. Example types of processor 340 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuits.

[0065] Processor 340 is also capable of executing other processes and programs resident in memory 360, such as receiving, storing, and timely indicating operations by providing image capture and processing. Processor 340 can move data into or out of memory 360 as required by the executed process. In some embodiments, processor 340 is configured to receive, store, and process data based on OS 361 or in response to a request from an eNB (similar to Figure 1 The processor 340 is also coupled to an I / O IF 345, which provides the electronic device 300 with the ability to connect to other devices (such as client devices 106-116). The I / O IF 345 is the communication path between these accessories and the processor 340.

[0066] Processor 340 is also coupled to input 350. An operator of electronic device 300 can use input 350 to input data or input to electronic device 300. Input 350 can be a keyboard, touch screen, mouse, trackball, voice input or any other device that can act as a user interface to allow a user to interact with electronic device 300. For example, input 350 can include voice recognition processing, thereby allowing a user to input voice commands via microphone 320. For another example, input 350 can include a touch panel, a (digital) pen sensor, a key or an ultrasonic input device. The touch panel can recognize touch input of at least one of a capacitive scheme, a pressure-sensitive scheme, an infrared scheme or an ultrasonic scheme. For example, in a capacitive scheme, input 350 can recognize touch or proximity. Input 350 can also include a control circuit. By providing additional inputs to processor 340, input 350 can be associated with (multiple) sensors 365 and / or cameras. As discussed in more detail below, sensor 365 includes inertial sensors (such as accelerometers, gyroscopes and magnetometers), optical sensors, motion sensors, cameras, pressure sensors, heart rate sensors, altimeters, etc. For example, input 350 may utilize motion detected by a motion sensor associated with sensor 365 as input.

[0067] Processor 340 is also coupled to display 355. Display 355 may be a Liquid Crystal Display (LCD), a Light-Emitting Diode (LED) display, an Organic LED (OLED), an Active Matrix OLED (AMOLED), or other display capable of rendering text and / or graphics (such as from a website, video, game, image, etc.). Display 355 may be sized to fit into an HMD. Display 355 may be a single display screen or multiple display screens capable of creating a stereoscopic display. In a particular embodiment, display 355 is a Heads-Up Display (HUD).

[0068] Memory 360 is coupled to processor 340. A portion of memory 360 may include random access memory (RAM), and another portion of memory 360 may include flash memory or other read-only memory (ROM).

[0069] Memory 360 may include a permanent storage device (not shown), which represents any (multiple) structures capable of storing and facilitating retrieval of information (such as data, program code, and / or other suitable information, whether temporary or permanent). Memory 360 may include one or more components or devices that support longer-term storage of data, such as a read-only memory, a hard drive, a flash memory, or an optical disk. Point cloud 363 may include various 3-D point clouds, 3-D meshes, 3-D objects, etc. In a particular embodiment, point cloud 363 may include a projection of a patch of a 3-D object on one or more 2-D frames. Point cloud 363 may also include one or more 3-D point clouds that can be rendered on display 355.

[0070] The electronic device 300 may further include one or more sensors 365 that measure physical quantities or detect the activation state of the electronic device 300 and convert the measured or detected information into electrical signals. For example, the sensor(s) 365 may include one or more buttons for touch input (located on the head device or the electronic device 300), one or more cameras, a posture sensor, an eye tracking sensor, a gyroscope or a gyroscope sensor, an air pressure sensor, a magnetic sensor or a magnetometer, an acceleration sensor or an accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red, green, and blue (RGB) sensor), a biophysical sensor, a temperature / humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasonic sensor, an iris sensor, a fingerprint sensor, etc. The sensor(s) 365 may further include a control circuit for controlling at least one of the sensors included therein. As will be discussed in more detail below, one or more of these sensor(s) 365 may be used to control a user interface (UI), detect UI input, determine orientation and direction toward the user for three-dimensional content display recognition, etc. Any of these sensor(s) 365 may be located within the electronic device 300, may be located within an auxiliary device operably connected to the electronic device 300, may be located within a head device configured to hold the electronic device 300, or may be located in a single device in which the electronic device 300 includes the head device.

[0071] As will be discussed in more detail below, in this illustrative embodiment, the electronic device 300 receives an encoded and compressed bitstream. The electronic device 300 decodes the received bitstream into a plurality of 2-D frames. In a particular embodiment, the decoded bitstream also includes an occupancy map. The decoded bitstream may also include a flag indicating whether specific metadata is applied to modify the point cloud. The electronic device 300 decompresses the plurality of 2-D frames. The plurality of 2-D frames may include a set of frames indicating coordinates, such as the geographic location of each point in the point cloud. For example, a frame may include a graphical depiction, such as one or more patches of each geometric point of a point cloud represented in 2-D. Another set of frames may include a texture associated with each point, such as the color of each point. The electronic device 300 may then reconstruct and render the point cloud in three dimensions.

[0072] As will be discussed in more detail below, in this illustrative embodiment, the electronic device 300 can be similar to the server 200 and encode the point cloud. The electronic device 300 can generate multiple 2-D frames representing the geometry and texture of the point cloud. The point cloud can be mapped to one or more 2-D frames. For example, one frame set can include the geometric points depicted in the patch. In another example, another frame set can include the texture or color or both of the point cloud.

[0073] 2-D frames represent various hierarchical levels representing 3-D point clouds. For example, the lowest level may be a single patch located on a frame. A single patch is a single cluster of points representing the attributes of a 3-D point cloud. For example, a cluster of points of a patch may represent the geometry of the points, such as the geographic location of each point within the point cloud. In another example, a cluster of points of a patch may represent a color value associated with each point in the cluster. The next level may be a single 2-D frame. A single 2-D frame includes one or more patches representing a 3-D point cloud. The next level is a 2-D group of frames (GOF). The next level is a sequence. A sequence includes two or more GOFs. The maximum level is a video including two or more sequences. In a particular embodiment, more or fewer levels may represent a 3-D point cloud. For example, levels may be represented by patches, frames, and sequences. In another example, there may be an intermediate level between patches and frames, referred to as a patch group.

[0074] The electronic device 300 can identify metadata within a frame. Metadata can be signaled at different hierarchical levels. Different levels are used to assign one or more flags to indicate whether specific metadata is applied to each corresponding level. For example, metadata applied to a frame modifies each patch in the frame. In another example, metadata applied to a GOF is then applied to each patch within each frame of the GOF. In another example, metadata applied to a sequence is then applied to each patch within each frame of the GOF within a specific sequence. In another example, metadata applied at the patch level can be applied to a specific patch of a specific frame.

[0075] although Figure 2 and Figure 3 An example of a device in a computing system is shown, but the Figure 2 and Figure 3 Make various changes. For example, Figure 2 and Figure 3The various components in the embodiment may be combined, further subdivided, or omitted, and additional components may be added as required. As a specific example, processor 340 may be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, like computing and communication networks, electronic devices and servers may have a variety of configurations, and Figure 2 and Figure 3 The present disclosure is not limited to any particular electronic device or server.

[0076] Figure 4 Point cloud 405 and example grid 410 are shown in accordance with an embodiment of the present disclosure. Point cloud 405 depicts an illustration of a point cloud. A point cloud is digitized data that visually defines an object in 3-D space. As depicted, point cloud 405 includes a plurality of points, each point representing an external coordinate of an object, similar to a topographic map. For example, each point may include one or more attributes. Attributes may include geometry, such as geographic location. Attributes of each point may also include color, intensity, texture, motion, material properties, and the like. Attributes other than geometry may be referred to as textures, such that textures represent various aspects and properties associated with each point in the point cloud.

[0077] Similarly, mesh 410 depicts a diagram of a 3-D mesh. Mesh 410 is digital data that visually defines an object in 3-D space. The graphical depiction of mesh 410 is defined by many polygonal or triangular interconnections of information between individual points. Each polygon may include various information, such as attributes. Attributes may include geometry and texture. Textures include color reflectivity, motion, and the like. For example, topological data provides connection information between vertices, such as adjacency of vertices, edges, and faces. Geometric information provides the geometric position of each vertex in 3-D space. Attribute information provides normals, colors, and application-related information for each individual vertex. The vertices of each polygon are similar to points in point cloud 405. Each polygon of mesh 410 represents the outer surface of an object.

[0078] Point clouds (such as point cloud 405) and meshes (such as mesh 410) utilize considerable bandwidth to be transferred from one computing device to another. Compression is necessary to reduce storage and bandwidth requirements. For example, lossy compression can compress point clouds and meshes while keeping distortion within tolerable levels while reducing the size of the data.

[0079] Figure 5A An example block diagram of an encoder 500 according to an embodiment of the present disclosure is shown. Figure 5B An example block diagram of a decoder 550 according to an embodiment of the present disclosure is shown.

[0080] The encoder 500 may be similar to Figure 1 Server 104, Figure 1 Any of the client devices 106-116, and including similar Figure 2 Server 200 and Figure 3 The decoder 550 may be similar to the internal components of the electronic device 300. Figure 1 Any of the client devices 106-116, and including similar Figure 3 The encoder 500 can be connected to the internal components of the electronic device 300. Figure 1 The network 102 communicates with the decoder 550. The embodiments of the encoder 500 and the decoder 550 are for illustration only. Other embodiments may be used without departing from the scope of the present disclosure.

[0081] The encoder 500 can compress, encode and transmit the point cloud, mesh or both as a coded bitstream. In a particular embodiment, the encoder 500 generates a plurality of 2-D frames onto which the point cloud or mesh is mapped or projected. For example, the point cloud is unfolded and mapped onto a plurality of 2-D frames. For example, the point cloud can be unfolded along one axis (such as the Y axis) and the image is mapped along the remaining axes (such as the X axis and the Z axis). In another example, a cluster of points in a 3-D point cloud can be projected onto a 2-D frame. For example, the geometric coordinates Z of a point in a 3-D point cloud are projected onto the X and Y coordinates of a 2-D frame, wherein the X and Y coordinates of the 2-D frame correspond to the X and Y coordinates of a point in the 3-D point cloud. In a particular embodiment, when the point cloud is mapped onto a 2-D frame, the encoder 500 generates an occupancy map (such as an occupancy map frame 522) indicating the location of each pixel of the point cloud. For example, the occupancy map included in the occupancy map frame 522 indicates valid points and invalid points for each frame. Invalid points are locations of the frame that do not correspond to points in the point cloud, while valid points are locations of the frame that correspond to points in the 3-D point cloud.Occupancy map frames 522 are represented as one or more frames that include an occupancy map.

[0082] In certain embodiments, encoder 500 may represent 3-D point cloud 502 as an equation.

[0083] Equation 1:

[0084] p n = < x n ,y n , z n , r n , g n , b n , nx n ,ny n , nz n ,u n , Vn >, n = 1, ..., N

[0085] In equation 1, (x n ,y n , z n ) represents the X, Y, and Z coordinates of point "n" in the 3-D point cloud 502 in 3-D space. In addition, (r n , g n , b n ) represents the color of point “n” in the 3-D point cloud 502. Similarly, (nx n ,ny n , nz n ) represents the normal of point "n" in 3-D point cloud 502 in 3-D space. In addition, (u n , v n ) represents the texture of point "n" in 3-D point cloud 502. In certain embodiments, point "n" may have a subset of the properties given in Equation 1. In certain embodiments, point "n" may have additional properties, such as material properties, motion, etc. These properties may be associated with different frames, such as one or more geometry frames 514 and one or more texture frames 518. Additional sets of frames may be generated through frame packing 512 to represent additional properties, such as material properties, motion, etc.

[0086] The function g(y, z) may represent a geometry frame, such as one of geometry frames 514. For example, a point p n An example orthogonal projection of a point can be expressed as p n = < x n ,y n , z n For example, the orthogonal projection of point "n" on the (Y, Z) plane of the geometry frame is represented by g(y n , z n )=x n That is, at a specific frame position (y n , z n ), the value of the x-coordinate or a quantized version of the x-coordinate is stored. In certain embodiments, the actual position of the point in the 2-D frame is not (y n , z n ). Note that projections on other planes (such as (X, Z), (X, Y) or any arbitrary plane) can be defined similarly. In a specific embodiment, the actual value that is compressed is "x n ” where the offset is constant for the entire patch.

[0087] In certain embodiments, encoder 500 is a network server, a server computer such as a management server, or any other electronic computing system capable of mapping the three dimensions of a point cloud into two dimensions, compressing frames, and encoding images for transmission. In certain embodiments, encoder 500 may be a "cloud" of computers interconnected by one or more networks, where each computer is a computing system that is connected via Figure 1 The network 102 is accessed utilizing clustered computers and components to act as a single pool of seamless resources.

[0088] The decoder 550 can decode, decompress, and generate a received encoded bitstream that includes a representation of a point cloud, a mesh, or both. In a particular embodiment, the decoder 550 can generate multiple point clouds from a received bitstream that includes multiple 2-D frames. For example, each of the pixels of the point cloud can be mapped based on information received in the 2-D frame and a received occupancy map.

[0089] Encoder 500 illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that encodes and compresses a point cloud for transmission. In a particular embodiment, encoder 500 packages a representation of a point cloud for transmission to one or more decoders (such as decoder 550) via a bitstream. Encoder 500 includes a received 3-D point cloud 502, a patch generator 504, auxiliary information 506, a frame packing 512, various frames (such as one or more geometry frames 514, one or more texture frames 518, and one or more occupancy map frames 522), various encoders (such as encoding engine 516, encoding engine 520, and encoding engine 524), and a multiplexer 526.

[0090] 3-D point cloud 502 represents an input to encoder 500. 3-D point cloud 502 may be stored in a memory associated with encoder 500. 3-D point cloud 502 may be a single 3-D object or a video including movement of a 3-D object.

[0091] The patch generator 504 decomposes the 3-D point cloud 502 and creates multiple patches. The patches can be organized by attributes. For example, the attributes can include geometry, texture, reflectivity, material, etc. Additional frame sets (in addition to the geometry frame 514, texture frame 518, and occupancy map frame 522) can represent additional attributes, such as color, reflectivity, material, intensity, normal, etc. Geometry (such as geometry frame 514) is the geographic location of each point of the 3-D point cloud 502. Texture (such as texture frame 518) represents a single aspect of each point of the 3-D point cloud 502, such as color. As described above, additional frames representing other attributes can be created. The patch generator 504 can cluster geometry attributes and texture attributes onto different 2-D frames. Clustering can be based on criteria such as normal direction, distance to the projection frame, continuity, etc. Each generated cluster is represented as a patch, as shown below with respect to Fig. 6A , Figure 6B and Figure 6C As described.

[0092] In certain embodiments, the patch generator 504 projects and maps the points in the point cloud onto the 2-D frame via a one-to-one mapping. The patch generator 504 may project and map the points in the point cloud onto the 2-D frame by projecting the points onto different planes. In certain embodiments, the projection and mapping of the points in the point cloud onto the 2-D frame may be based on raster scan mapping, mapping based on X geometric coordinates, mapping based on Y geometric coordinates, mapping based on Z geometric coordinates, color-based mapping, normal direction-based mapping, etc.

[0093] The patch generator 504 also generates auxiliary information 506. The auxiliary information 506 is information associated with each patch, a single frame, multiple frames, etc. In a particular embodiment, the auxiliary information 506 is metadata that indicates to the decoder various modifications to be performed when reconstructing the point cloud based on the modifications performed by the encoder when generating the frame. For example, the patch and the frame may include auxiliary information such as (i) scaling, (ii) offsets, such as spatial position offsets and depth offsets, (iii) rotations, (iv) the size of each point, (v) the shape of each point, (vi) resolution, (vii) point density, etc.

[0094] In a particular embodiment, metadata for scaling may be expressed based on geometric coordinates "x", "y", and "z". For example, with respect to the x-coordinate, metadata may be expressed as "scale_metadata_on_axis_x". With respect to the y-coordinate, metadata may be expressed as "scale_metadata_on_axis_y". Similarly, with respect to the z-coordinate, metadata may be expressed as "scale_metadata_on_axis_z". In a particular embodiment, a point in a point cloud includes integer coordinates that are converted into actual physical coordinates using scaling.

[0095] In a particular embodiment, metadata for the offset may be expressed based on geometric coordinates "x", "y", and "z". For example, with respect to the x-coordinate, metadata may be expressed as "offset_metadata_on_axis_x". With respect to the y-coordinate, metadata may be expressed as "offset_metadata_on_axis_y". Similarly, with respect to the z-coordinate, metadata may be expressed as "offset_metadata_on_axis_z". In a particular embodiment, a point in a point cloud includes integer coordinates that are converted into actual physical coordinates using an offset.

[0096] In particular embodiments, metadata for rotation may be expressed based on geometric coordinates "x", "y", and "z". For example, with respect to the x-coordinate, metadata may be expressed as "rotation_metadata_on_axis_x". With respect to the y-coordinate, metadata may be expressed as "rotation_metadata_on_axis_y". Similarly, with respect to the z-coordinate, metadata may be expressed as "rotation_metadata_on_axis_z". In particular embodiments, points in a point cloud include integer coordinates that are converted into actual physical coordinates using rotation. Note that in particular embodiments, points in a point cloud include integer coordinates that are converted into actual physical coordinates using scaling, offset, and rotation.

[0097] In a particular embodiment, metadata for the size of points rendered in a decoded point cloud may be represented as "point_size_metadata". The point size metadata adjusts the size of points or pixels in the 3-D point cloud 502 when reconstructed by the decoder 550. If a small point size is used, the reconstructed point cloud may include holes. Conversely, if the point size is too large, the reconstructed point cloud may appear blurry. Therefore, the points included in the reconstructed point cloud (such as the 3-D point cloud 502 after reconstruction by the decoder 550) have an optimal size so that the point cloud does not include holes while not appearing blurry. The point size metadata may be predefined. For example, the creator of the 3-D point cloud 502 may signal the optimal point size in metadata for use by the decoder 550.

[0098] Similar to the point size, metadata for the shape of the points rendered in the decoded point cloud can be represented as "point_shape_metadata". The point shape controls the visual quality of the point cloud. Similar to the size of each point, the shape of each point can ensure that the reconstructed 3-D point cloud does not appear blurry or include holes. As shown below, Table 1 includes example metadata values ​​that can indicate different shapes for each point. Note that additional shapes can be used for points in the 3-D point cloud 502. Note that additional "Point_Shape" integer values ​​can indicate additional point shapes.

[0099] Table 1:

[0100]

[0101]

[0102] A decoder, such as decoder 550, uses the auxiliary information 506 to reconstruct the point cloud. If the encoder 500 modifies the 3-D point cloud while generating the 2-D frame, auxiliary information (such as metadata) is generated. For example, if the 3-D point cloud is scaled to a smaller size by sub-sampling a plurality of points or pixels included in the 3-D point cloud, the scaling parameters may be represented as metadata and included in the auxiliary information 506. Sub-sampling the 3-D point cloud to reduce the resolution reduces the number of points included in each 2-D frame, which may reduce the bit rate when the frames are transmitted as an encoded bitstream 528. The decoder may then reconstruct the point cloud and up-sample the frame based on the scaling parameters and reconstruct the 3-D point cloud at a higher resolution.

[0103] Additionally, auxiliary information 506 may indicate a ROI of 3-D point cloud 502. The ROI may indicate a portion of a point cloud that is modified differently than the rest of 3-D point cloud 502. The ROI may be scaled to a first parameter while the rest of the 3-D point cloud is scaled to another parameter. The creator of 3-D point cloud 502 may identify the ROI. For example, the creator of 3-D point cloud 502 may identify the face of a human figure as the ROI because the face is important and should be minimally compressed compared to the rest of 3-D point cloud 502 via the scaling parameter. Fig. 9 The ROI of the 3-D point cloud 502 is further described.

[0104] The auxiliary information 506 may also include a flag 508. The flag 508 indicates when the auxiliary information is applied to a specific patch, a set of patches, a single frame, a group of frames, a sequence, or a video, etc. The access level refers to the layer (tier) or level of various elements that make up the 2-D representation of the 3-D point cloud, such as patches, frames, groups of frames, sequences, videos, etc. The access level is one or more groups of patches or frames. For example, a single patch in a frame can be one access level. Another access level can be a single 2-D frame including many patches. Another access level can be a group of frames (Group Of Frames, GOF) including two or more frames, each frame including a patch. Another access level can be a sequence. A sequence represents a portion of a video in which a 3-D point cloud moves. For example, if the point cloud is walking on a road, lifting something from the ground, climbing stairs, etc. A sequence includes one or more GOFs. A video may include multiple sequences. Note that a single point cloud can be represented by more or fewer access levels because the access levels provide various hierarchical categories of the 3D point cloud 502. For example, access levels may be represented by patches, frames, and sequences. In another example, there may be intermediate levels between patches and frames, called patch groups. Additional or alternative access levels may be defined.

[0105] In addition to the metadata types described above, metadata can also be applied to specific access levels of a point cloud. For example, metadata for scaling can be applied to a specific patch at the patch level. Example metadata for adjusting the scaling of x, y, and z coordinates at an access level representing a patch can be represented as "patch_scale_metadata_on_axis_x", "patch_scale_metadata_on_axis_y", and "patch_scale_metadata_on_axis_z", respectively. Metadata for offset, rotation, etc. can be represented similarly. In addition, metadata representing different access levels can be represented in a similar manner for frames, GOFs, sequences, videos, etc.

[0106] Metadata such as scale, resolution, point density, offset, rotation, point size, point shape, etc. can be referred to as parameter sets and are applied to different access levels. For example, a Patch Parameter Set (PPS) includes types of metadata for a single patch in a single frame representing a portion of a 3-D point cloud. Note that a patch on a frame representing a geometric shape has a corresponding counterpart on a frame representing another attribute (such as texture, color, material, etc.). In another example, a Frame Parameter Set (FPS) includes types of metadata associated with each frame so that all patches within the frame have the same frame-level parameters. Similarly, a GOF Parameter Set (GofPS) defines GOF-level metadata that is the same for all frames included in the GOF. Similarly, a Sequence Parameter Set (SPS) includes sequence-level metadata that is the same for all 2-D frames and frames corresponding to patches included in the sequence. In addition, the Video Parameter Set (VPS) includes metadata that is the same for all point cloud sequences, such as when the bitstream includes multiple sequences of different resolutions. In a particular embodiment, the order of priority is (i) PPS, (ii) FPS, (iii) GofPS, (iv) SPS, and (v) VPS. Note that some parameter sets (such as PPS, FPS, GofPS, SPS, and VPS) or access levels may not necessarily be defined for point clouds. In addition, additional parameter sets or access levels may be defined for point clouds.

[0107] For example, if patch-level scaling metadata parameters are signaled for a patch, the metadata is considered to be the scaling value of the patch. Conversely, if patch-level scaling metadata parameters are not signaled for a patch, but frame-level scaling parameters are signaled for the frame containing the patch, the frame-level scaling metadata parameters are assigned to the scaling parameters of the patch.

[0108] The auxiliary information 506 may include non-orthogonal transformation parameters. For example, when the patch generator 504 generates the patch, the patch generator 504 may use non-orthogonal projection instead of orthogonal projection. When identified at a decoder (such as the decoder 550), the non-orthogonal transformation parameters are used to reconstruct the 3-D point cloud.

[0109] The flags 508 may include one or more control flags that indicate to the decoder when the auxiliary information 506 is applied to the 2-D frame to reconstruct the 3-D point cloud. The flags 508 may signal when information (such as metadata) included in the auxiliary information 506 is applied to each corresponding access level. In certain embodiments, the flags 508 are separated from the auxiliary information 506 and multiplexed into the encoded bitstream 528 via the multiplexer 526. In certain embodiments, the flags 508 are omitted.

[0110] Each access level may have one or more flags 508 that control whether a metadata parameter set is signaled in the bitstream. Various flags 508 may signal when specific metadata is applicable to a specific access level. As described above, the metadata may include metadata that controls the following: (i) scaling, (ii) offset, (iii) rotation, (iv) size of each point, (v) shape of each point, etc. In a particular embodiment, the flag 508 may be a single control binary flag that is (i) absolute signaling by reusing previously signaled metadata, (ii) conditional signaling by reusing previously signaled metadata, and (iii) using a reference identifier. Absolute signaling using two control binary flags is discussed below with reference to grammars 1, 2, 3, and 4. Absolute signaling using one control binary flag is discussed below with reference to grammars 6 and 7. Absolute signaling by reusing previously signaled metadata is discussed below with reference to grammar 8. Conditional signaling by reusing previously signaled metadata is discussed below with reference to grammar 9. Reference identifiers for signaling metadata are discussed below with reference to grammar 10.

[0111] In a particular embodiment, two control binary flags 508 may be used to signal metadata. The two flags 508 may include (i) an enable flag and (ii) a presence flag associated with each type of metadata at each specified access level. The access levels may include patch level, frame level, GOF level, sequence level, and video level, among others. The number or description of access levels associated with a point cloud (such as 3-D point cloud 502) may be predefined or predetermined. In a particular embodiment, two access levels are defined, such as patch and frame. In a particular embodiment, multiple access levels are defined in addition to access levels at patch level, frame level, GOF level, sequence level, and video level.

[0112] If the flags 508 include two control flags (such as an enable flag and a presence flag), the enable flag is signaled one access level higher than the presence flag. If the enable flag is true, the presence flag is signaled at the same access level.

[0113] Syntax 1:

[0114]

[0115]

[0116] Syntax 1 shows that if the frame level enable flag is true, the decoder 550 will read the frame level presence flag at the frame level. Thereafter, if the frame level presence flag is true, the decoder 550 will read the frame level metadata parameter set. Thereafter, if the patch level enable flag is true, the decoder 550 reads the enable flags for each of the various types of patch level metadata, such as scale, offset, rotation, dot size, and dot shape.

[0117] For example, if the syntax element "patch_metadata_enabled_flag" is signaled at the frame level (the access level associated with the frame), then the syntax element identifies whether there is at least one patch within the frame for which the patch parameter set is signaled. If the flag is true, the syntax element "patch_metadata_present_flag" is signaled at the patch level and identifies which patches the patch parameter set is signaled for. A sample syntax for scaling metadata is shown below. If the flag is not signaled, then the flag is inferred to be false by the decoder 550.

[0118] Syntax 2:

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125] In addition, regarding the two flags 508, if the enable flag is false or if the exist flag is false, the metadata of the highest access level is set to a default value. If not signaled, the metadata of an access level lower than the highest access level is set to the corresponding metadata of the higher access level. If the enable flag is false or if the exist flag is false, the metadata may not be signaled.

[0126] Syntax 3:

[0127]

[0128]

[0129] The above syntax 3 shows various metadata parameter sets (Parameters Set, PS) at different access levels, such as GOF parameter set (syntax element "GofPS"), frame parameter set (syntax element "FPS") and patch parameter set (syntax element "PPS"). Each parameter set includes a flag (such as flag 508) indicating whether the metadata parameter set is signaled in the bitstream. Within each parameter set, each metadata has a flag that controls whether the metadata is signaled in the parameter set. For example, the metadata parameters in GofPS are initialized to default values. The metadata parameters in FPS are initialized to their corresponding parameters in GofPS. Thereafter, the metadata parameters in PPS are initialized to their corresponding parameters in FPS. In addition, syntax 2 shows metadata for scaling, offset, rotation, point size, and point shape.

[0130] Syntax 3 shows that if the enable flag is present at the frame level, the decoder 550 performs one of two actions. If the frame level enable flag is present, the decoder 550 reads the scaling metadata and applies it to the x, y, and z coordinates, respectively. If the frame level enable flag is not present, the decoder 550 uses the metadata from the higher access level of the frame access level for the x, y, and z metadata. In a particular embodiment, the higher access level may be the GOF access level.

[0131] Syntax 4 below shows that two control flags, an enabled flag and a present flag (denoted as flag 508), are applied to signal whether metadata is applied at a particular access level.

[0132] Syntax 4:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143] Syntax 4 shows a flag 508 that signals when metadata is applicable to different access levels. A decoder 550 may receive a coded bitstream 528. The coded bitstream 528 may include auxiliary information 506, which includes both the flag 508 and the metadata. Lines 1 to 13 of Syntax 4 show an example bitstream header. Starting from line 14 of Syntax 4, a GOF header syntax is shown. The GOF syntax header corresponds to the access level of the GOF. Starting from line 47, a GOF metadata syntax is shown. The GOF metadata syntax is represented as a GOF parameter set. Starting from line 95, frame auxiliary information is shown. Starting from line 151, a frame level metadata syntax is shown. The frame metadata syntax is represented as a frame parameter set. The frame level corresponds to the access level of the frame. In addition, starting from line 199, a patch level metadata syntax is shown. The patch metadata syntax is represented as a patch parameter set. The patch level corresponds to the access level of the patch. When a default value is assigned to metadata for a particular access level, the default value may be predetermined, or the metadata may be assigned a value from the next highest access level. In a particular embodiment, predetermined values ​​are assigned only to the highest access level, while lower access levels are assigned values ​​from the next highest access level. Note that the flag may be interchangeably set to zero or false. It is also noted that the flag may be interchangeably set to one or true.

[0144] Referring to the above syntax 4, the syntax elements "gof_metadata_enabled_flag", "gof_scale_enabled_flag", "gof_offset_enabled_flag", "gof_rotation_enabled_flag", "gof_point_size_enabled_flag" and "gof_point_shape_enabled_flag" are included in the bitstream header semantics. The syntax element "gof_metadata_enabled_flag" of lines 5 and 6 of syntax 4 indicates whether the GOF level metadata parameter set is signaled in the bitstream. If "gof_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the GOF level metadata parameter set can be signaled for some GOFs. Alternatively, if "gof_metadata_enabled_flag" is equal to zero (or false), it indicates to the decoder 550 that the GOF level metadata parameter set is not signaled. When the GOF level metadata parameter set is not signaled, the metadata parameters in the GOF level are assigned default values. For example, the default value may be a value from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value.

[0145] The syntax element "gof_scale_metadata_enabled_flag" of lines 7 and 48 of Syntax 4 indicates whether the GOF level scaling metadata parameters are signaled in the bitstream. If "gof_scale_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the GOF level scaling metadata parameters can be signaled for some GOFs. Alternatively, if "gof_scale_metadata_enabled_flag" is equal to zero (or false), it indicates to the decoder 550 that the GOF level scaling metadata parameters are not signaled. When the GOF level scaling metadata parameters are not signaled, the scaling metadata parameters in the GOF level are assigned default values. For example, the default value may be a value from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value. Note that if "gof_scale_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0146] Similarly, the syntax element "gof_offset_metadata_enabled_flag" of lines 8 and 58 of Syntax 4 indicates whether the GOF level offset metadata parameter is signaled in the bitstream. If "gof_offset_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the GOF level offset metadata parameter can be signaled for some GOFs. Alternatively, if "gof_offset_metadata_enabled_flag" is equal to zero (or false), it indicates that the GOF level offset metadata parameter is not signaled. When the GOF level offset metadata parameter is not signaled, the offset metadata parameter in the GOF level is assigned a default value. For example, the default value may be a value from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value. Note that if "gof_offset_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0147] In addition, the syntax element "gof_rotation_metadata_enabled_flag" of lines 9 and 68 of Syntax 4 indicates whether the GOF level rotation metadata parameter is signaled in the bitstream. If "gof_rotation_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the GOF level rotation metadata parameter can be signaled for some GOFs. Alternatively, if "gof_rotation_metadata_enabled_flag" is equal to zero (or false), it indicates that the GOF level rotation metadata parameter is not signaled. When the GOF level rotation metadata parameter is not signaled, the rotation metadata parameter in the GOF level is assigned a default value. For example, the default value may be a value from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value. Note that if "gof_rotation_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0148] In addition, the syntax element "gof_point_size_metadata_enabled_flag" of lines 10 and 78 of Syntax 4 indicates whether the GOF level point size metadata parameter is signaled in the bitstream. If "gof_point_size_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the GOF level point size metadata parameter may be signaled for some GOFs. Alternatively, if "gof_point_size_metadata_enabled_flag" is equal to zero (or false), it indicates that the GOF level point size metadata parameter is not signaled. When the GOF level point size metadata parameter is not signaled, the point size metadata parameter in the GOF level is assigned to its default value. For example, the default value may be a value from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value. Note that if "gof_point_size_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0149] In addition, the syntax element "gof_point_shape_metadata_enabled_flag" of lines 11 and 86 of Syntax 4 indicates whether the GOF level point shape metadata parameters are signaled in the bitstream. If "gof_point_shape_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the GOF level point shape metadata parameters can be signaled for some GOFs. Alternatively, if "gof_point_shape__metadata_enabled_flag" is equal to zero (or false), it indicates that the GOF level point shape metadata parameters are not signaled. When the GOF level point shape metadata parameters are not signaled, the point shape metadata parameters in the GOF level are assigned to their default values. For example, the default value may be a value from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value. Note that if "gof_point_shape_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0150] Refer to the above syntax 4, the syntax elements "gof_metadata_present_flag", "gof_scale_metadata_present_flag", "gof_offset_metadata_enabled_flag", "gof_ rotation_metadata_enabled_flag", "gof_point_size_metadata_present_flag", "gof_point_shape_metadata_present_flag", "fram e_metadata_enabled_flag", "frame_scale_metadata_enabled_flag", "frame_offset_metadata_enabled_flag", "frame_rotation_metadata_enabled_flag", "frame_point_size_metadata_enabled_flag" and "frame_point_shape_metadata_enabled_flag" are included in the frame group header semantics.

[0151] The syntax element "gof_metadata_present_flag" of lines 31 and 33 of Syntax 4 indicates whether a gof level metadata parameter set is signaled for the current GOF. If "gof_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that no gof level metadata parameter set is signaled for the current GOF. Alternatively, if "gof_metadata_present_flag" is equal to zero (or false), it indicates to the decoder 550 that no gof level metadata parameter set is signaled for the current GOF. When no gof level metadata parameter set is signaled for the current GOF, the scaling metadata parameters in the current GOF are assigned default values. For example, the default value may be a value from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value. Note that if "gof_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0152] Similarly, the syntax element "gof_scale_metadata_present_flag" of lines 49 and 51 of Syntax 4 indicates whether GOF level scaling metadata parameters are signaled for the current GOF. If "gof_scale_metadata_present_flag" is equal to one (or true), it is indicated to the decoder 550 that GOF level scaling metadata parameters along the x, y and z axes are signaled for the current GOF. The syntax elements "gof_scale_metadata_on_axis_x", "gof_scale_metadata_on_axis_y" and "gof_scale_metadata_on_axis_z" indicate the values ​​of GOF level scaling along the x-axis, y-axis and z-axis for the current GOF, respectively. If "gof_scale_metadata_present_flag" is equal to zero (or false), it is indicated to the decoder 550 that GOF level scaling metadata parameters are not signaled for the current GOF. When GOF level scaling metadata parameters are not signaled for the current GOF, the syntax elements "gof_scale_metadata_on_axis_x", "gof_scale_metadata_on_axis_y" and "gof_scale_metadata_on_axis_z" are set to a default value, such as one. If "gof_scale_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0153] The syntax element "gof_offset_metadata_enabled_flag" of line 58 of Syntax 4 indicates whether the GOF level offset metadata parameters are signaled for the current GOF. If "gof_offset_metadata_present_flag" is equal to one (or true), the GOF level offset metadata parameters along the x, y and z axes are signaled for the current GOF to the decoder 550. For example, "gof_offset_metadata_on_axis_x", "gof_offset_metadata_on_axis_y" and "gof_offset_metadata_on_axis_z" indicate the values ​​of the GOF level offsets along the x-axis, y-axis and z-axis for the current GOF, respectively. Alternatively, if "gof_offset_metadata_present_flag" is equal to zero (or false), the GOF level offset metadata parameters are not signaled for the current GOF to the decoder 550. When no GOF level offset metadata parameters are signaled for the current GOF, "gof_offset_metadata_on_axis_x", "gof_offset_metadata_on_axis_y", and "gof_offset_metadata_on_axis_z" are set to default values, such as zero. For example, the default values ​​may be values ​​from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default values ​​may be predefined values. If "gof_offset_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0154] The syntax element "gof_rotation_metadata_enabled_flag" of line 68 of Syntax 4 indicates whether GOF level rotation metadata parameters are signaled for the current GOF. If "gof_rotation_metadata_present_fla" is equal to one (or true), the decoder 550 is indicated that the GOF level rotation metadata parameters along the x, y and z axes are signaled for the current GOF. Alternatively, if "gof_rotation_metadata_present_flag" is equal to zero (or false), it indicates that the GOF level rotation metadata parameters are not signaled for the current GOF. When the GOF level rotation metadata parameters are not signaled for the current GOF, the syntax elements "gof_rotation_metadata_on_axis_x", "gof_rotation_metadata_on_axis_y" and "gof_rotation_metadata_on_axis_z" are set to default values, such as zero. For example, the default values ​​may be values ​​from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value. Note that if "gof_rotation_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0155] Referring to the rotation metadata, the syntax element "gof_rotation_metadata_on_axis_x" indicates the value of the GOF level rotation angle along the x-axis for the current GOF, expressed in 2. -16 The unit is degree. The value range of "gof_rotation_metadata_on_axis_x" is -180*2 16 To 180*2 16 Similarly, the syntax element "gof_rotation_metadata_on_axis_y" indicates the value of the GOF level rotation angle along the y-axis for the current GOF, expressed in 2. -16 The unit is degree. The value range of "gof_rotation_metadata_on_axis_y" is -180*2 16 To 180*2 16 -1. In addition, the syntax element "gof_rotation_metadata_on_axis_z" indicates the value of the GOF level rotation angle along the z axis for the current GOF, expressed in 2 -16 The unit is degree. The value range of "gof_rotation_metadata_on_axis_z" is -180*216 To 180*2 16 -1. In certain embodiments, the rotation angle may be sent with lower precision, such as 2 -8 .

[0156] The syntax element "gof_point_size_metadata_present_flag" of lines 79 and 81 of Syntax 4 indicates whether the GOF level point size metadata parameter is signaled for the current GOF. If "gof_point_size_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that the GOF level point size metadata parameter is signaled for the current GOF. Alternatively, if "gof_point_size_metadata_present_flag" is equal to zero (or false), it indicates that the GOF level point size metadata parameter is not signaled for the current GOF. When the GOF level point size metadata parameter is not signaled, gof_point_size_metadata is set to a default value, such as one. Note that if "gof_point_size_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero. In addition, the item "gof_point_size_metadata" indicates the value of the GOF level point size for the current GOF.

[0157] Similarly, the syntax element "gof_point_shape_metadata_present_flag" of line 87 of Syntax 4 indicates whether the GOF level point shape metadata parameters are signaled for the current GOF. If "gof_point_shape_metadata_present_flag" is equal to one (or true), it indicates that the GOF level point shape metadata parameters are signaled for the current GOF. Alternatively, if "gof_point_shape_metadata_present_flag" is equal to zero (or false), it indicates that the GOF level point shape metadata parameters are not signaled for the current GOF. When the GOF level point shape metadata parameters are not signaled, "gof_point_shape_metadata" is set to a default value, such as zero. For example, the default value may be a value from the next highest access level. In another example, if the GOF access level is the highest defined access level, the default value may be a predefined value. Note that if "gof_point_snape_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero. In addition, the syntax element "gof_point_shape_metadata" indicates the value of the GOF level point shape for the current GOF. The value of the GOF level point shape for the current GOF may be identified similarly to the values ​​of Table 1 above.

[0158] The syntax element "frame_metadata_enabled_flag" of line 105 of Syntax 3 indicates whether the frame level metadata parameter set is signaled in the bitstream. If "frame_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the frame level metadata parameter set may be signaled for some frames. Alternatively, if "frame_metadata_enabled_flag" is equal to zero (or false), it indicates that the frame level metadata parameter set is not signaled, so the metadata parameters in the frame level are assigned corresponding values ​​from the GOF level.

[0159] The syntax element "frame_scale_metadata_enabled_flag" of lines 38 and 152 of Syntax 4 indicates whether the frame level scaling metadata parameters are signaled in the bitstream. If "frame_scale_metadata_enabled_flag" is equal to one (or true), it indicates that the frame level scaling metadata parameters can be signaled for some frames. Alternatively, if "frame_scale_metadata_enabled_flag" is equal to zero (or false), it indicates that the frame level scaling metadata parameters are not signaled. If the frame level scaling metadata parameters are not signaled, the scaling metadata parameters in the frame level are assigned to the corresponding values ​​from the GOF level. Note that if "frame_scale_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0160] Similarly, the syntax element "frame_offset_metadata_enabled_flag" of lines 39 and 162 of Syntax 4 indicates whether the frame level offset metadata parameter is signaled in the bitstream. If "frame_offset_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the frame level offset metadata parameter may be signaled for some frames. Alternatively, if "frame_offset_metadata_enabled_flag" is equal to zero (or false), it indicates that the frame level offset metadata parameter is not signaled. If the frame level offset metadata parameter is not signaled, the offset metadata parameter in the frame level is assigned the corresponding value from the GOF level. Note that if "frame_offset_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0161] Similarly, the syntax element "frame_rotation_metadata_enabled_flag" of lines 40 and 172 of Syntax 4 indicates whether the frame level rotation metadata parameters are signaled in the bitstream. If "frame_rotation_metadata_enabled_flag" is equal to one (or true), it indicates that the frame level rotation metadata parameters can be signaled for some frames. Alternatively, if "frame_rotation_metadata_enabled_flag" is equal to zero (or false), it indicates that the frame level rotation metadata parameters are not signaled. If the frame level rotation metadata parameters are not signaled, the rotation metadata parameters in the frame level are assigned the corresponding values ​​from the GOF level. Note that if "frame_rotation_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0162] In addition, the syntax element "frame_point_size_metadata_enabled_flag" of lines 41 and 182 of Syntax 3 indicates whether the frame level point size metadata parameter is signaled in the bitstream. If "frame_point_size_metadata_enabled_flag" is equal to one (or true), it indicates that the frame level point size metadata parameter can be signaled for some frames. Alternatively, if "frame_point_size_metadata_enabled_flag" is equal to zero (or false), it indicates that the frame level point size metadata parameter is not signaled. If the frame level point size metadata parameter is not signaled, the point size metadata parameter in the frame level is assigned the corresponding value from the GOF level. If "frame_point_size_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0163] In addition, the syntax element "frame_point_shape_metadata_enabled_flag" indicates whether the frame level point shape metadata parameters are signaled in the bitstream. If "frame_point_shape_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the frame level point shape metadata parameters can be signaled for some frames. Alternatively, if "frame_point_shape__metadata_enabled_flag" is equal to zero (or false), it indicates that the frame level point shape metadata parameters are not signaled. If the frame level point shape metadata parameters are not signaled, the point shape metadata parameters in the frame level are assigned to the corresponding values ​​in the GOF level. Note that if "frame_point_shape_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0164] Referring to Syntax 4 above, the syntax elements "frame_metadata_present_flag", "frame_scale_metadata_present_flag", "frame_offset_metadata_enabled_flag", "frame_rotation_metadata_enabled_flag", "frame_point_size_metadata_present_flag", "frame_point_shape_metadata_present_flag", "patch_metadata_enabled_flag", "patch_scale_metadata_enabled_flag", "patch_offset_metadata_enabled_flag", "patchrotation_metadata_enabled_flag", "patch_point_size_metadata_enabled_flag" and "patch_point_shape_metadata_enabled_flag" are included in the frame group auxiliary information semantics. These elements are specified one per frame group.

[0165] For example, the syntax element "frame_metadata_present_flag" of lines 106 and 108 of Syntax 4 indicates whether a frame-level metadata parameter set is signaled for the current frame. If "frame_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that a frame-level metadata parameter set is signaled for the current frame. Alternatively, if "frame_metadata_present_flag" is equal to zero (or false), it indicates that no frame-level metadata parameter set is signaled for the current frame. When no frame-level metadata parameter set is signaled for the current frame, the scaling metadata parameters in the current frame are assigned corresponding values ​​from the GOF level. If "frame_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0166] Similarly, the syntax element "frame_scale_metadata_present_flag" of lines 153 and 155 of Syntax 4 indicates whether frame-level scaling metadata parameters are signaled for the current frame. If "frame_scale_metadata_present_flag" is equal to one (or true), the decoder 550 is indicated that frame-level scaling metadata parameters along the x, y, and z axes are signaled for the current frame. For example, the syntax elements "frame_scale_metadata_on_axis_x", "frame_scale_metadata_on_axis_y", and "frame_scale_metadata_on_axis_z" indicate the values ​​of frame-level scaling of the current frame along the x-axis, y-axis, and z-axis, respectively. Alternatively, if "frame_scale_metadata_present_flag" is equal to zero (or false), it indicates that no frame level scaling metadata parameters are signaled for the current frame, so "frame_scale_metadata_on_axis_x", "frame_scale_metadata_on_axis_y", and "frame_scale_metadata_on_axis_z" are set to a default value, such as one. Note that if "frame_scale_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0167] The syntax element "frame_offset_metadata_enabled_flag" of lines 162 to 171 of Syntax 4 indicates whether the frame level offset metadata parameter is signaled for the current frame. The syntax elements "frame_offset_metadata_on_axis_x", "frame_offset_metadata_on_axis_y", and "frame_offset_metadata_on_axis_z" indicate the values ​​of the frame level offsets of the current frame along the x-axis, y-axis, and z-axis. If "frame_offset_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that the frame level offset metadata parameters along the x-axis, y-axis, and z-axis are signaled for the current frame. Alternatively, if "frame_offset_metadata_present_flag" is equal to zero (or false), it indicates that the frame level offset metadata parameters are not signaled for the current frame. Note that if the frame level offset metadata parameters are not signaled for the current frame, "frame_offset_metadata_on_axis_x", "frame_offset_metadata_on_axis_y", and "frame_offset_metadata_on_axis_z" are set to default values, such as zero. For example, the default values ​​may be values ​​from the next highest access level, such as the GOF access level. In another example, if the frame access level is the highest defined access level, the default values ​​may be predefined values. If "frame_offset_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0168] The syntax element "frame_rotation_metadata_enabled_flag" of line 172 of Syntax 4 indicates whether frame level rotation metadata parameters are signaled for the current frame. If "frame_rotation_metadata_present_flag" is equal to one (or true), it indicates that frame level rotation metadata parameters along the x, y and z axes are signaled for the current frame. Alternatively, if "frame_rotation_metadata_present_flag" is equal to zero (or false), it indicates that frame level rotation metadata parameters are not signaled for the current frame. When the frame level rotation metadata parameters are not signaled, "frame_rotation_metadata_on_axis_x", "frame_rotation_metadata_on_axis_y" and "frame_rotation_metadata_on_axis_z" are set to default values, such as zero. For example, the default value may be a value from the next highest access level (such as a GOF access level). In another example, if the frame access level is the highest defined access level, the default value may be a predefined value. Note that if "frame_rotation_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0169] Referring to the frame rotation metadata, the syntax element "frame_rotation_metadata_on_axis_x" indicates the value of the frame-level rotation angle along the x-axis for the current frame, expressed in 2. -16 Similarly, "frame_rotation_metadata_on_axis_y" indicates the value of the frame-level rotation angle along the y-axis for the current frame, expressed in degrees. -16 In addition, "frame_rotation_metadata_on_axis_z" indicates the value of the frame-level rotation angle along the z-axis for the current frame, expressed in 2 -16 The unit is degrees. The value range of "frame_rotation_metadata_on_axis_x", "frame_rotation_metadata_on_axis_y" and "frame_rotation_metadata_on_axis_z" is -180*2 16 To 180*2 16 -1.

[0170] The syntax element "frame_point_size_metadata_present_flag" of line 183 of Syntax 4 indicates whether the frame-level point size metadata parameter is signaled for the current frame. If "frame_point_size_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that the frame-level point size metadata parameter is signaled for the current frame. Alternatively, if "frame_point_size_metadata_present_flag" is equal to zero (or false), it indicates that the frame-level point size metadata parameter is not signaled for the current frame. When the frame-level point size metadata parameter is not signaled for the current frame, "frame_point_size_metadata" is set to a default value, such as one. Note that if "frame_point_size_metadata_present_flag" is not present in Syntax 4, the item is assumed to be zero. In addition, "frame_point_size_metadata" of line 186 of Syntax 4 indicates the value of the frame-level point size for the current frame.

[0171] In addition, the syntax element "frame_point_shape_metadata_present_flag" of line 191 of Syntax 4 indicates whether the frame level point shape metadata parameters are signaled for the current frame. If "frame_point_shape_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that the frame level point shape metadata parameters are signaled for the current frame. Alternatively, if "frame_point_shape_metadata_present_flag" is equal to zero (or false), it indicates that the frame level point shape metadata parameters are not signaled for the current frame. If the frame level point shape metadata parameters are not signaled for the current frame, the syntax element "frame_point_shape_metadata" is set to a default value, such as zero. For example, the default value may be a value from the next highest access level (such as the GOF access level). In another example, if the frame access level is the highest defined access level, the default value may be a predefined value. Note that if "frame_point_shape_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero. In addition, the syntax element "frame_point_shape_metadata" indicates the value of the frame level point shape for the current frame. The value of the frame level point shape for the current frame may be identified similarly to the values ​​of Table 1 above.

[0172] The syntax element "patch_metadata_enabled_flag" of lines 110 and 112 of Syntax 4 indicates whether the patch-level metadata parameter set is signaled in the bitstream. If "patch_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the patch-level metadata parameter set may be signaled for some patches. Alternatively, patch_metadata_enabled_flag is equal to zero (or false), indicating that the patch-level metadata parameter set is not signaled. When the patch-level metadata parameter set is not signaled, the metadata parameters in the patch level are assigned corresponding values ​​from the frame level.

[0173] In addition, the syntax element "patch_scale_metadata_enabled_flag" of lines 113 and 200 of Syntax 4 indicates whether the patch level scaling metadata parameters are signaled in the bitstream. If "patch_scale_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the patch level scaling metadata parameters may be signaled for some patches. Alternatively, if "patch_scale_metadata_enabled_flag" is equal to zero (or false), it indicates that the patch level scaling metadata parameters are not signaled. If the patch level scaling metadata parameters are not signaled, the scaling metadata parameters in the patch level are assigned corresponding values ​​from the GOF level. Note that if "patch_scale_metadata_enabled_flag" is not present, it is assumed to be zero.

[0174] The syntax element "patch_offset_metadata_enabled_flag" of lines 114 and 210 of Syntax 4 indicates whether the patch level offset metadata parameter is signaled in the bitstream. If "patch_offset_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the patch level offset metadata parameter may be signaled for some patches. Alternatively, if "patch_offset_metadata_enabled_flag" is equal to zero (or false), it indicates that the patch level offset metadata parameter is not signaled. If the patch level offset metadata parameter is not signaled, the offset metadata parameter in the patch level is assigned the corresponding value in the frame level. Note that if "patch_offset_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0175] Similarly, the syntax element "patch_rotation_metadata_enabled_flag" of lines 115 and 220 of Syntax 4 indicates whether the patch-level rotation metadata parameters are signaled in the bitstream. If "patch_rotation_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the patch-level rotation metadata parameters may be signaled for some patches. Alternatively, if "patch_rotation_metadata_enabled_flag" is equal to zero (or false), it indicates that the patch-level rotation metadata parameters are not signaled. If the patch-level rotation metadata parameters are not signaled, the rotation metadata parameters in the patch level are assigned the corresponding values ​​at the frame level. Note that if "patch_rotation_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0176] The syntax element "patch_point_size_metadata_enabled_flag" of lines 116 and 230 of Syntax 4 indicates whether the patch level point size metadata parameter is signaled in the bitstream. If "patch_point_size_metadata_enabled_flag" is equal to one (or true), it indicates that the patch level point size metadata parameter may be signaled for some patches. Alternatively, if "patch_point_size_metadata_enabled_flag" is equal to zero (or false), it indicates that the patch level point size metadata parameter is not signaled. If the patch level point size metadata parameter is not signaled, the point size metadata parameter in the patch level is assigned the corresponding value at the frame level. Note that if "patch_point_size_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0177] In addition, the syntax element "patch_point_shape_metadata_enabled_flag" of lines 117 and 238 of Syntax 4 indicates whether the patch level point shape metadata parameters are signaled in the bitstream. If "patch_point_shape_metadata_enabled_flag" is equal to one (or true), it indicates to the decoder 550 that the patch level point shape metadata parameters may be signaled for some patches. Alternatively, if "patch_point_shape_metadata_enabled_flag" is equal to zero (or false), it indicates that the patch level point shape metadata parameters are not signaled. If the patch level point shape metadata parameters are not signaled, the point shape metadata parameters in the patch level are assigned corresponding values ​​from the frame level. Note that if "patch_point_shape_metadata_enabled_flag" is not present in Syntax 4, it is assumed to be zero.

[0178] Referring to Syntax 4 above, the syntax elements "patch_metadata_present_flag", "patch_scale_metadata_present_flag", "patch_offset_metadata_enabled_flag", "patch_rotation_metadata_enabled_flag", "patch_point_size_metadata_present_flag", and "patch_point_shape_metadata_present_flag" are specified once per patch.

[0179] For example, the syntax element "patch_metadata_present_flag" of lines 132 and 134 of Syntax 4 indicates whether a patch-level metadata parameter set is signaled for the current patch. If 'patch_metadata_present_flag' is equal to one (or true), it indicates to the decoder 550 that a patch-level metadata parameter set is signaled for the current patch. Alternatively, if 'patch_metadata_present_flag' is equal to zero (or false), it indicates that no patch-level metadata parameter set is signaled for the current patch. If no patch-level metadata parameter set is signaled for the current patch, the scaling metadata parameters in the current patch are assigned corresponding values ​​from the frame level. Note that if 'patch_metadata_present_flag' is not present in Syntax 4, it is assumed to be zero.

[0180] The syntax element "patch_scale_metadata_present_flag" of the row of Syntax 4 indicates whether patch level scaling metadata parameters are signaled for the current patch. If "patch_scale_metadata_present_flag" is equal to one (or true), it indicates that patch level scaling metadata parameters along the x, y and z axes are signaled for the current patch. Alternatively, if "patch_scale_metadata_present_flag" is equal to zero (or false), it indicates that patch level scaling metadata parameters are not signaled for the current patch, so "patch_scale_metadata_on_axis_x", "patch_scale_metadata_on_axis_y" and "patch_scale_metadata_on_axis_z" are set to default values, such as one. Note that if "patch_scale_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero. Furthermore, the syntax elements “patch_scale_metadata_on_axis_x”, “patch_scale_metadata_on_axis_y”, and “patch_scale_metadata_on_axis_z” indicate values ​​of patch level scaling along the x-axis, y-axis, and z-axis for the current patch, respectively.

[0181] In addition, the syntax element "patch_offset_metadata_enabled_flag" of lines 114 and 210 of Syntax 4 indicates whether patch-level offset metadata parameters are signaled for the current patch. If "patch_offset_metadata_present_flag" is equal to one (or true), the decoder 550 is indicated that the patch-level offset metadata parameters along the x, y, and z axes are signaled for the current patch. Alternatively, if "patch_offset_metadata_present_flag" is equal to zero (or false), it indicates that the patch-level offset metadata parameters are not signaled for the current patch, so patch_offset_metadata_on_axis_x, patch_offset_metadata_on_axis_y, and patch_offset_metadata_on_axis_z are set to default values, such as zero. For example, the default value may be a value from the next highest access level (such as a frame access level). In another example, if the patch access level is the highest defined access level, the default value may be a predefined value. Note that "patch_offset_metadata_present_flag" is assumed to be zero if it is not present in Syntax 4. The syntax elements "patch_offset_metadata_on_axis_x", "patch_offset_metadata_on_axis_y", and "patch_offset_metadata_on_axis_z" indicate the values ​​of the patch level offsets along the x-axis, y-axis, and z-axis, respectively, for the current patch.

[0182] In addition, the syntax element "patch_rotation_metadata_enabled_flag" of lines 115 and 220 of Syntax 4 indicates whether patch-level rotation metadata parameters are signaled for the current patch. If "patch_rotation_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that patch-level rotation metadata parameters along the x, y, and z axes are signaled for the current patch. Alternatively, if "patch_rotation_metadata_present" is equal to zero (or false), it indicates that patch-level rotation metadata parameters are not signaled for the current patch. If patch-level rotation metadata parameters are not signaled for the current patch, "patchrotation_metadata_on_axis_x", "patch_rotation_metadata_on_axis_y", and "patch_rotation_metadata_on_axis_z" are set to default values, such as zero. For example, the default values ​​may be values ​​from the next highest access level, such as a frame access level. In another example, if the patch access level is the highest defined access level, the default value may be a predefined value. Note that if "patch_rotation_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero.

[0183] Referring to the rotation metadata, the syntax element "patch_rotation_metadata_on_axis_x" indicates the value of the patch-level rotation angle along the x-axis for the current patch, expressed in 2. -16 Similarly, the syntax element "patch_rotation_metadata_on_axis_y" indicates the value of the patch-level rotation angle along the y-axis for the current patch, in degrees. -16 In addition, the syntax element "patch_rotation_metadata_on_axis_z" indicates the value of the patch-level rotation angle along the z-axis for the current patch, in degrees. -16 The unit is degrees. The value range of "patch_rotation_metadata_on_axis_x", "patchrotation_metadata_on_axis_y" and "patch_rotation_metadata_on_axis_z" is -180*2 16 To 180*2 16 -1.

[0184] The syntax element "patch_point_size_metadata_present_flag" of lines 231 and 233 of Syntax 4 indicates whether a patch-level point size metadata parameter is signaled for the current patch. For example, if "patch_point_size_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that the patch-level point size metadata parameter is signaled for the current patch. Alternatively, if "patch_point_size_metadata_present_flag" is equal to zero (or false), it indicates that the patch-level point size metadata parameter is not signaled for the current patch. If the patch-level point size metadata parameter is not signaled for the current patch, patch_point_size_metadata is set to a default value, such as one. Note that if "patch_point_size_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero. The syntax element "patch_point_size_metadata" indicates the value of the patch-level point size for the current patch.

[0185] Finally, the syntax element "patch_point_shape_metadata_present_flag" of lines 239 and 241 of Syntax 4 indicates whether patch-level point shape metadata parameters are signaled for the current patch. For example, if "patch_point_shape_metadata_present_flag" is equal to one (or true), it indicates to the decoder 550 that patch-level point shape metadata parameters are signaled for the current patch. Alternatively, if "patch_point_shape_metadata_present_flag" is equal to zero (or false), it indicates that patch-level point shape metadata parameters are not signaled for the current patch. If patch-level point shape metadata parameters are not signaled for the current patch, "patch_point_shape_metadata" is set to a default value, such as zero. For example, the default value may be a value from the next highest access level, such as a frame access level. In another example, if the patch access level is the highest defined access level, the default value may be a predefined value. Note that if the syntax element "patch_point_shape_metadata_present_flag" is not present in Syntax 4, it is assumed to be zero. In addition, the syntax element "patch_point_shape_metadata" indicates the value of the patch level point shape for the current patch. The value of the patch level point shape for the current patch can be identified similarly to the values ​​in Table 1 above.

[0186] In a particular embodiment, at the highest access level, only the presence flag is signaled, and the enable flag is not signaled. In a particular embodiment, at the highest access level, the enable flag is assumed to be one (or true).

[0187] The type of metadata, such as scale, offset, rotation, point size, and point shape, can modify how the decoder 550 reconstructs the 3-D point cloud 502. Specifically, the metadata can (i) modify sparse point clouds, (ii) perform progressive scaling, indicate a ROI, and modify a portion of the point cloud to the indicated ROI differently from the rest of the point cloud, and (iii) perform adaptive point size and shape rendering to reduce or remove holes in the point cloud when the decoder 550 reconstructs the point cloud.

[0188] In certain embodiments, after projecting and mapping the points in the point cloud onto the 2-D frame, the points may be sorted and packed into the 2-D frame. Frame packing 512 sorts and packs the points into patches.

[0189] Frame packing 512 classifies and packs patches into one or more geometry frames 514 and one or more texture frames 518. In a particular embodiment, the patch generator generates one or more occupancy maps, and the frame packing 512 packs the occupancy maps into one or more occupancy map frames 522, wherein the one or more occupancy map frames 522 correspond to the one or more geometry frames 514 and the one or more texture frames 518. The geometry and texture information corresponding to the patches generated by the patch generator 504 are packed into separate video frames, such as one or more geometry frames 514 and one or more texture frames 518. In a particular embodiment, the frame packing 512 creates one or more geometry frames 514 and one or more texture frames 518. The frame packing 512 can determine the height and width of the frame based on how the patches are arranged and packed into the frame. In a particular embodiment, the patches included in a single frame are arranged to occupy the minimum space in the frame. That is, the size of the frame is determined based on the position of each patch within the frame. Therefore, under the constraint that the frame size is constant at least for each GOF, the frame is the smallest size possible based on how the patches are oriented and positioned within the frame. In certain embodiments, the frame size is also constant for each video sequence.

[0190] The occupancy map frame 522 represents an occupancy map indicating pixel locations in frames (such as one or more geometry frames 514 and one or more texture frames 518) that contain valid points of a point cloud projected or mapped onto the corresponding frame. For example, the occupancy map frame 522 indicates whether each pixel on each frame (such as one or more geometry frames 514 and one or more texture frames 518) is a valid pixel or blank. A valid pixel on the occupancy map indicates a pixel on a 2-D frame that corresponds to a 3-D point in a point cloud. In certain embodiments, the occupancy map included in the occupancy map frame 522 may be the same for each frame, similar to the one or more geometry frames 514 and one or more texture frames 518. In certain embodiments, the occupancy map frame 522 represents multiple occupancy maps, each of which corresponds to a single frame (such as one or more geometry frames 514 and one or more texture frames 518). In certain embodiments, the occupancy map frame 522 may be transmitted as metadata as a separate bitstream.

[0191] One or more geometry frames 514 and one or more texture frames 518 are encoded via encoding engines 516 and encoding engines 520, respectively. Encoding engine 524 also encodes occupancy map frame 522. In a particular embodiment, auxiliary information 506 is encoded by another encoding engine. Encoding engines 516, 520, and 524 can be separate encoding engines or the same encoding engine. For example, encoding engines 516, 520, and 524 are the same components, so that one or more geometry frames 514, one or more texture frames 518, and one or more occupancy map frames are encoded by the same encoding engine. In a particular embodiment, encoding engines 516, 520, and 524 can be configured to support 8-bit, 10-bit, 12-bit, 14-bit, or 16-bit data precision. Encoding engines 516, 520, and 524 can be video or image codecs such as HEVC, AVC, VP9, ​​VP8, JVNET, etc., which can be used to compress 2-D frames representing 3-D point clouds.

[0192] The auxiliary information 506, the encoded one or more geometry frames 514, the encoded one or more texture frames 518, and the encoded one or more occupancy map frames 522 are multiplexed via a multiplexer 526. The multiplexer 526 combines the auxiliary information 506, the encoded one or more geometry frames 514, the encoded one or more texture frames 518, and the encoded one or more occupancy map frames 522 to create a single encoded bitstream 528. In particular embodiments, the encoding engine 516, the encoding engine 520, and the encoding engine 524 are a single encoding engine.

[0193] The encoding engines 516, 520, and 522 may include lossy point cloud compression or lossless point cloud compression. If a lossy point cloud is used, an occupancy map frame 522 is included in the bitstream and transmitted to a decoder (such as decoder 550). The decoder 550 uses the occupancy map frame 522 to identify valid points in each frame (such as one or more geometry frames 514 and one or more texture frames 518). If lossless compression is used to compress the frames (such as one or more geometry frames 514), the decoder 550 may derive valid points directly from the one or more geometry frames 514. For example, the decoder 550 may use the one or more geometry frames 514 in a manner similar to the occupancy map included in the occupancy map frame 522. As a result, the occupancy map frame 522 may be omitted from the bitstream to improve the compression rate. For example, the patch generator 504 and the frame packing 512 do not create the occupancy map frame 522, so the multiplexer 526 does not multiplex the occupancy map frame 522 with the encoded one or more geometry frames 514 (from the encoding engine 516), the encoded one or more texture frames 518 (from the encoding engine 520), and the auxiliary information 506.

[0194] Equation 1 above solves for orthogonal projection. In certain embodiments, the 3-D point cloud 502 is sparse, such that holes exist in the point cloud. A sparse point cloud occurs when points or pixels are not densely populated in a point cloud. Holes in the point cloud can degrade the compression performance of encoding engines 516, 520, and 524. Non-orthogonal projections with scaling can subsample the point cloud, effectively condensing the point cloud itself. For example, g(y, z) is a geometry frame (such as one of geometry frames 514). Point p n = < x n ,y n , z n >An example non-orthogonal projection onto the geometry frame (y, z) plane is given by the following equation.

[0195] Equation 2:

[0196] G(<y n / scale_fractor_y+offset_y,

[0197] z n / scale_fractorz+offset_z)=x n / scale_fractorx

[0198] In Equation 2, "scale_fractor_y", "scale_fractor_z", and "scale_fractor_x" represent scaling factors used to scale the geometric coordinates in the projection of the 3-D point cloud 502 onto a 2-D frame, such as the geometry frame 514. "offset_y", "offset_z", and "offset_x" represent the locations in the image where the projection patches are stored. Note that projections onto other planes, such as (X, Z), (X, Y), or arbitrary planes, may be defined similarly. In particular embodiments, the offset of the patch locations in the frame is zero unless a specific offset value is identified in the equation or metadata. Equations and metadata indicating the scaling or offset may be included in the auxiliary information 506. In particular embodiments, if the scaling factor is set to "2", the 3-D point cloud is subsampled by a factor of eight. For example, if the original point cloud is represented by a 1024, 1024, 1024 voxel cube and then subsampled by a factor of 4, the subsampled point cloud is represented by a 512, 512, 512 voxel cube.

[0199] In a particular embodiment, the scaling factor is encoded as metadata and transmitted to a receiver including a decoder (such as decoder 550). Without loss of generality, one or more scaling factors may be transmitted to the decoder. In a particular embodiment, the scaling factor may have the same value for all coordinates. For example, the scaling factor for the x-coordinate is the same as the scaling factor for the y-coordinate, and the scaling factor for the y-coordinate is the same as the scaling factor for the z-coordinate. In a particular embodiment, the scaling factor may have a different value for each coordinate. For example, the scaling factor for the x-coordinate is different from the scaling factor for the y-coordinate, and the scaling factor for the y-coordinate is different from the scaling factor for the z-coordinate. Example syntax is provided below to illustrate the scaling factor metadata.

[0200] Syntax 5:

[0201]

[0202] Syntax 5 illustrates applying a scaling factor to points in a 3-D point cloud, such as 3-D point cloud 502. The term "single_scale_factor" is a flag indicating whether the same scaling factor is used for all geometric coordinates. The term "scale_factor" indicates a common scaling factor for all geometric coordinates. The term "scale_factor_x" is a common scaling factor for the "x" geometric coordinate. Similarly, the term "scale_factor_y" is a common scaling factor for the "y" geometric coordinate. In addition, the term "scale_factor_z" is a common scaling factor for the "z" geometric coordinate. In particular embodiments, the scaling factor may be transmitted once per point cloud, once per 2-D frame (such as one of geometry frames 514), once per region of a 2-D frame (such as one of geometry frames 514), or once per patch (included in one of geometry frames 514).

[0203] In certain embodiments, the point density in a point cloud can be varied by controlling the scaling factor at certain different patches or certain regions of the point cloud. For example, a face or hand of a humanoid point cloud can be encoded with a higher point cloud density (such as resolution), while another body part (such as an arm or leg) can be encoded with a lower point cloud density (such as resolution).

[0204] To scale a portion of a 3-D point cloud by a scaling factor, the encoder 500 may include a list of scaling factors in the compressed bitstream. For example, the list of scaling factors may be included in the auxiliary information 506. In another example, the list of scaling factors may be included in the encoded bitstream 528 via the multiplexer 526. For each patch, an index into the list of scaling factors may be used. The index may be used to signal the scaling factor to use for the points in the patch. Equation 3 below shows a non-orthogonal transform for scaling a portion of a 3-D point cloud.

[0205] Equation 3:

[0206] Equation 3 above shows an example matrix for applying a scaling factor in a non-orthogonal transformation. n ,y n , z n ) represents the X, Y, and Z coordinates of point "n" in 3-D point cloud 502 in 3-D space. The corresponding projection value is represented by (x' n , y' n , z' n ) indicates. Therefore, x' n =g(y' n , z' n ) indicates the geometric coordinate information encoded in a frame (such as geometry frame 514). Note that projections on other planes, such as z' n =g(x' n , y' n ) or y' n =g(x' n , z' n A decoder (such as decoder 550, discussed in more detail below) extracts from the video frame to obtain the location of a point in 3-D space (x n ,y n , z n ) coordinates. In certain embodiments, a matrix similar to Equation 3 can be used to apply metadata to a specific portion of a point cloud. For example, scaling metadata can be applied to a portion of a point cloud, rather than the entire point cloud.

[0207] In certain embodiments, point attribute information is stored at a co-located location. The co-located location corresponds to the projected location of the geometric coordinates in another video frame. For example, when x' n =g(y' n , z' n ), corresponding to the point (x n ,y n , z n ) can be stored in frame f. For example, the red component of a point in the point cloud can be represented by f(y' n , z'n , 1). Similarly, the blue component of a point in the point cloud can be represented by f(y' n , z' n , 2), and the green component of a point in the point cloud can be represented by f(y' n , z' n , 3) indicates. In certain embodiments, the point attribute information is stored in the same frame (such as one of the geometry frames 514) as the spatial offset.

[0208] Non-orthogonal projections can scale a portion of a 3-D point cloud, such as 3-D point cloud 502. For example, ROI encoding, progressive or scaled encoding, encoding of sparse point clouds, etc. can use non-orthogonal projections. Region of interest encoding is described below with reference to Fig. 9 The progressive or scalable encoding is described in more detail below with reference to FIG8. Encoding a sparse point cloud is described in more detail below with reference to Figure 7B and Figure 7C Describe in more detail.

[0209] Decoder 550 illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that decodes and reconstructs a point cloud for rendering. In a particular embodiment, the decoder extracts compressed geometry and attributes from the encoded bitstream 528. In a particular embodiment, the decoder 550 maps the geometry and attributes to generate a 3-D point cloud 502. The decoder 550 includes a demultiplexer 552, a decoding engine 554, a decoded geometry frame 556, a decoded texture frame 558, a decoded occupancy map frame 560, auxiliary information 562, and a reconstruction engine 564. In a particular embodiment, the decoder 650 includes two or more decoding engines 554. One or more geometry frames 514 of the encoder 500 are decoded geometry frames 556 of the decoder 550. Similarly, one or more texture frames 518 of the encoder 500 are decoded texture frames 558 of the decoder 550. In addition, one or more occupancy map frames 522 of the encoder 500 are decoded occupancy map frames 560 of the decoder 550. The side information 506 of the encoder 500 is the side information 562 of the decoder 550 .

[0210] The decoder 550 uses information within patches of various frames, such as one or more geometry frames 514, one or more texture frames 518, one or more occupancy map frames 522, and the auxiliary information 506 to reconstruct a 3-D point cloud. The decoder 550 receives an encoded bitstream 528 originating from the encoder 500. Upon receiving the encoded bitstream 528, the demultiplexer 552 separates various data streams from the encoded bitstream 528. For example, the demultiplexer 552 separates various data streams, such as geometry frame information, texture frame information, occupancy map information, and auxiliary information 562. The decoding engine 554 decodes the geometry frame information to generate a decoded geometry frame 556. Similarly, the decoding engine 554 decodes the texture frame information to generate a decoded texture frame 558. In a particular embodiment, there is a separate decoding engine for each set of information included in the encoded bitstream 528. For example, a separate parallel decoding engine similar to the decoding engine 554 can process each received stream. The decoding engine 554 decompresses the various data streams. In certain embodiments, the decoding engine 554 may be lossless or lossy. In certain embodiments, the decoding engine 554 may reconstruct a signal from a bitstream 528 that is either lossy or lossless encoded.

[0211] The auxiliary information 562 includes metadata such as (i) scale, (ii) offset, (iii) rotation, (iv) point size, (v) point shape, etc. The auxiliary information 562 may also include one or more flags (such as flag 508) that indicate where the metadata is to be applied with respect to reconstructing the 3-D point cloud. In certain embodiments, the auxiliary information 562 may include metadata and a technique for signaling when the metadata is to be applied. For example, metadata may be signaled using (i) a single control flag (such as flag 508), (ii) two flags such as an enable flag and a presence flag (such as flag 508), (iii) absolute signaling by reusing previously signaled metadata, (iv) conditional signaling by reusing previously signaled metadata, and (y) reusing metadata previously signaled using a reference identifier.

[0212] The reconstruction engine 564 reconstructs the 3-D point cloud based on the data and auxiliary information received from the various frames (geometry frame 556, texture frame 558, occupancy map frame 560). The 3-D point cloud 502 points are reconstructed from the points in the patches included in the geometry frame 556 and the texture frame 558. The patches included in each frame (geometry frame 556 and texture frame 558) from the decoded auxiliary patch information 670 can be modified based on the metadata included in the auxiliary information 562. The reconstruction engine 564 determines where to apply each type of metadata based on the flags. For example, the flags can signal which metadata is applied to a specific portion of a patch or frame and when the metadata is applied to a specific portion of a patch or frame.

[0213] As discussed above with respect to flag 508, flags may include two flags associated with each type of metadata: an enable flag and an existence flag. Flags control whether metadata parameters are signaled. Flags may be associated with various access levels (such as patch level, frame level, GOF level, sequence level, video level, etc.). Depending on which access level the flag is associated with, it will indicate whether metadata is applied to that particular access level and all data included in that access level, such as a hierarchical structure. For example, if metadata is signaled at the patch level, the particular patch is modified by the metadata. In this example, metadata may not be applied to other patches or other frames. In another example, if metadata is signaled at the frame level (such as in a frame in the geometry frame 556), metadata is applied to one or more patches within the frame. In this example, metadata may not be applied to patches of another frame. In another example, if metadata is signaled at the GOF level, metadata is applied to one or more frames and various patches included in the frame. In this example, metadata may not be applied to patches of another GOF.

[0214] As mentioned above about Figure 5A As discussed above, flag 508 may signal when metadata is available in a decoder (such as Figure 5B The auxiliary information 506 may include additional signaling techniques to indicate when metadata is applied to modify the point cloud in addition to absolute signaling using two binary control flags (an enable flag and a presence flag, discussed above with respect to syntaxes 1, 2, 3, and 4). In certain embodiments, metadata may be signaled by (i) absolute signaling using a single control binary flag, (ii) absolute signaling by reusing previously signaled metadata, (iii) conditional signaling by reusing previously signaled metadata, and (iv) reference identification.

[0215] For example, the encoder 500 may indicate via a single control binary flag when the decoder should apply metadata to the point cloud.Each parameter set has an enable flag that controls whether the metadata parameter set is signaled in the bitstream.

[0216] Syntax 6:

[0217]

[0218]

[0219] Syntax 6 shows that metadata parameters in the highest level parameter set are read from the bitstream first (if they are signaled), otherwise the parameter set is set to some default value. Next, metadata parameters in a lower level parameter set are read from the bitstream (if they are signaled). Otherwise, the parameter set is set to the value of its corresponding parameter at the next higher level. This process continues until metadata parameters in the lowest level parameter set are either read from the bitstream or assigned from corresponding parameters at the next higher level. Syntax 7 below shows metadata parameter sets for scaling.

[0220] Syntax 7:

[0221]

[0222]

[0223]

[0224] Syntax 7 shows that when a scaling parameter set does not exist at a particular level, the values ​​are inferred to be the parameter values ​​of the next higher level. For example, if the syntax element "patch_resolution_metadata" does not exist, the value of "patch_resolution_metadata" is inferred from "frame_resolution_metadata" at one access level higher. If a parameter set does not exist (so that the value of the presence flag for the parameter set is false), the values ​​of all metadata parameters in the parameter set are inferred from those of the next higher level. At the highest access level (such as the video parameter set in a particular embodiment), if a particular parameter does not exist, its value is inferred to be a default value. As a result, at each level, each parameter is assigned a valid value, either by inference or by explicit signaling in the bitstream.

[0225] In another example, the encoder 500 may indicate via absolute signaling by reusing previously signaled metadata to instruct the decoder 550 as to when to apply metadata to the point cloud. For example, a flag (such as flag 508 of FIG. 5 ) indicates whether the parameter set metadata is the same as the previous set. A single flag or different flags may indicate different metadata parameters in each parameter set. Similarly, a single flag or different flags may indicate different parameter sets. Syntax 8 below shows the scaling metadata parameters, where the presence flag is checked before the reuse flag. If the flag is not signaled, it is inferred to be false.

[0226] Syntax 8:

[0227]

[0228]

[0229] Similar syntax elements may be represented at sequence access level, GOF access level, frame access level and patch access level in syntax 8. In another example, a binary flag is signaled to determine whether the metadata parameter is read from the bitstream or assigned to a previously signaled one.

[0230] In another example, the encoder 500 can indicate via conditional signaling by reusing previously signaled metadata to instruct the decoder 550 as to when to apply metadata to the point cloud. For example, two binary flags (or one integer flag) can indicate whether the parameter set is assigned to or predicted from a previous set.

[0231] Syntax 9:

[0232]

[0233]

[0234] In Syntax 9, similar syntax elements can be expressed at the sequence access level, GOF access level, frame access level and patch access level. Syntax 9 shows that assignment to the previous set has priority over prediction from the previous set. Note that the predictions can be signaled in different orders.

[0235] In another example, the encoder 500 may indicate via a reference identification in order to instruct the decoder 550 as to when to apply metadata to the point cloud. That is, a list is used to maintain multiple previously explicitly signaled metadata parameter sets at different access levels. Thereafter, each parameter set uses a reference or identification parameter to reference one of those parameter sets in the list.

[0236] Syntax 10:

[0237]

[0238]

[0239] In syntax 10, a fixed size buffer of parameter sets is used with the setup rules for filling the lists: for example, all parameter sets of the sequence access level and the GOF access level are included. The remaining lists are filled with the previously signaled parameter sets.

[0240] Fig. 6A , Figure 6B and Figure 6C An example 3-D point cloud and a 2-D frame representing the 3-D point cloud according to an embodiment of the present disclosure are shown. Specifically, Fig. 6A 3-D point cloud 610 is shown, and Figure 6B , Figure 6CA 2-D frame is shown, including regular patches. Fig. 6A , Figure 6B and Figure 6C The embodiments shown are for illustration only, and other embodiments may be used without departing from the scope of the present disclosure.

[0241] 3-D point cloud 610 is similar to Figure 4 Point cloud 405, Figure 4 Grid 410, Figure 5A and Figure 5B 3-D point cloud 502. 3-D point cloud 610 is a collection of data points in 3-D space. Each point in the 3-D point cloud includes (i) a geometric location that provides the structure of the 3-D point cloud and (ii) one or more textures that provide information about each point, such as color, reflectivity, material, etc.

[0242] Figure 6B and Figure 6C 2-D frames 620 and 630 are shown, respectively. Frame 620 depicts a regular patch (such as patch 622) representing the texture of 3-D point cloud 610. Frame 630 depicts a regular patch (such as patch 632) representing the depth value of 3-D point cloud 610. Frames 620 and 630 may be similar to Figure 5A One or more texture frames 518 and one or more geometry frames 514.

[0243] Discussed in more detail below FIG. 7A to FIG. 10 B shows the use of metadata such as scale, offset, rotation, point size, and point shape to modify the 3-D point cloud. Non-orthogonal projection can adjust the scale of the 3-D point cloud to indicate ROI, progressive encoding, and manipulate sparse point clouds. For example, scaling and offsetting can transform a 3-D point cloud (similar to Figure 4 Point cloud 405, Figure 4 Grid 410, Figure 5A and Figure 5B 3-D point cloud 502 and Fig. 6A 610 ) is normalized to a voxelized grid. After receiving the compressed bitstream, a decoder (such as Figure 5B The decoder 550) decompresses the received bitstream. If the global scale and offset metadata are available to the decoder, the point cloud can be scaled back to the original coordinates.

[0244] Adaptive streaming is the streaming of the same 3-D point cloud at different bit rates and resolutions. For example, 3-D point clouds of different bit rates and resolutions can be created and stored in an information repository (such as a database or server). The rendering client requests a specific bit rate / resolution based on the network bandwidth. If the resolution of the received point cloud is different from the display resolution of the device, the device scales the point cloud appropriately. Point clouds of different resolutions are created by (i) subsampling the point cloud at the original voxel grid resolution, or (ii) reducing the voxel grid. Subsampling the point cloud can result in a sparse point cloud. Conversely, reducing the voxel grid can result in a dense point cloud by scaling the geometric coordinates. The scaling factor included in the scaling metadata changes the voxel grid at the encoder, and at the decoder, the metadata is used to return the point cloud to its original coordinate system.

[0245] The scaling point cloud is a sparse point cloud that can be decomposed via non-orthogonal projection. For example, x, y, and z are the coordinates of the sparse point cloud, and xs, ys, and zs are the input encoder (such as Figure 5A The scaled point cloud is represented in the x-coordinate as "xs=(x+offset_x)*scale_factor_x". The scaled point cloud is represented in the y-coordinate as "ys=(y+offset_y)*scale_factor_y". The scaled point cloud is represented in the z-coordinate as "zs=(z+offset_z)*scale_factor_z". If xr, yr, and zr are decoders (such as Figure 5B Where xo, yo and zo are the coordinates of the reconstructed point cloud at the output of the decoder 550), and xo, yo and zo are the coordinates of the point in the final scaled point cloud, the output of each point is expressed by the following equation 4.

[0246] Equation 4:

[0247] xo=xr / scale_factor_x+offset_x

[0248] yo=yr / scale_factor_y+offset_y

[0249] zo=zr / scale_factor_x+offset_z

[0250] The rotation metadata is used to control the plane of projection. For example, Figure 5AA patch generator 504 generates patches for the point cloud. When the patches are reconstructed into a point cloud at the decoder, patch boundary artifacts appear at the boundary lines of the patches. To avoid patch boundary artifacts, the point cloud can be rotated so that patches are not created on specific parts of the point cloud. For example, if a patch is created on the face of a humanoid point cloud, the encoder can rotate the point cloud to avoid the face of the point cloud being split between two or more patches. Rotating the point cloud can cause the face to appear on a single patch. The magnitude of the rotation is stored as rotation metadata to enable the decoder to reconstruct the point cloud. The decompressed point cloud is inversely rotated back to the original coordinates based on the rotation metadata.

[0251] Fig. 7A and Figure 7B The process of reconstructing a point cloud from a patch is shown. Specifically, Fig. 7A An example flow chart 700 is shown for receiving patches at a scaled resolution according to an embodiment of the present disclosure. Figure 7B and Figure 7C A process 760 of creating a bitstream including patches and a process 780 of receiving the bitstream according to an embodiment of the present disclosure are shown. Fig. 7A , Figure 7B and Figure 7C The embodiments shown are for illustration only, and other embodiments may be used without departing from the scope of the present disclosure.

[0252] Flowchart 700 is described as consisting of Figure 1 Client devices 106-114, Figure 5B Any one of the decoders 550 is implemented, and includes a decoder similar to Figure 2 Server 200 00, Figure 3 The process begins with the decoder 550 detecting a patch within a decompressed frame (such as a geometry frame 556 and a texture frame 558) (710). The process determines whether the patch is normal (720). That is, the decoder 550 determines whether the patch is normal by a method similar to Figure 5A The encoder 500 undergoes a change in density, resolution, or scaling.

[0253] If the patch has not undergone modification, the decoder 550 reconstructs the patch into a 3-D point cloud (740). If the patch has undergone a change in density, resolution, or scaling by the encoder, the patch data is modified to return the patch to its original form (730). After the patch is returned to its original form, the decoder 550 reconstructs the patch into a 3-D point cloud (740). After the decoder reconstructs the patch, the decoder outputs a reconstructed 3-D point cloud (750). The reconstructed 3-D point cloud may be displayed on a display such as a Figure 3 display 355) for the user to view.

[0254] Figure 7B A process 760 is shown for creating a bitstream of a 3-D point cloud 762a. The 3-D point cloud 762a is created by Figure 5A The encoder of encoder 500 is represented as a patch. The encoder scales and offsets the point cloud into a normalized 3-D point cloud 762b (764). The compression engine 768 compresses the normalized 3-D point cloud 762b into an encoded bitstream 770. The scale and offset metadata 766 are multiplexed into the encoded bitstream 770.

[0255] Figure 7C Shows such as Figure 5B The decoder 550 of the decoder receives the encoded bitstream 770. The encoded bitstream 770 is demultiplexed into scale and offset metadata 772 and a compressed normalized 3-D point cloud 762b. In a particular embodiment, the scale and offset metadata 766 and the scale and offset metadata 772 are the same, so that the scale and offset metadata 766 is created by an encoder (such as encoder 500) to be included in the encoded bitstream 770 and is used by a decoder (such as decoder 550) to reconstruct the 3-D point cloud 762a. The decompression engine 774 decompresses the compressed normalized 3-D point cloud 762b. The decoder then takes the inverse of the scale and offset metadata 772 to generate the original 3-D point cloud 762a (776).

[0256] Fig. 8A A process 800 of generating progressive non-orthogonal projections of a point cloud is shown according to an embodiment of the present disclosure. Figure 8B An example block diagram of a decoder receiving a scaled point cloud according to an embodiment of the present disclosure is shown. Fig. 8A Similar to Figure 5A Encoder 500. Similarly, Figure 8B Similar to Figure 5B Decoder 550. Fig. 8A and Figure 8B The embodiments shown are for illustration only, and other embodiments may be used without departing from the scope of the present disclosure.

[0257] From a source device such as Figure 1Progressive and adaptive bitrate transmission of a 3-D point cloud from a server 104 of the present invention to a client device (such as any of the client devices 106-116) allows for varying network conditions and individual capabilities of devices (such as the decoder 550 or the client devices 106-116). For example, when the network bandwidth, the processing power of the decoder, the processing power of the device rendering the 3D point cloud, or a combination thereof, is low, the decoder may receive a compressed bitstream including a lower level of detail (LLOD). Conversely, when the network bandwidth or the device processing power is high, the decoder may receive a compressed bitstream including a high level of detail (HLOD). The LLOD may be created by performing a non-orthogonal projection of the 3-D point cloud onto a 2-D frame, similar to Equations 2 and 3 above.

[0258] Process 800 is similar to Figure 5A The encoder 500 is configured such that the point cloud is projected onto a frame and multiplexed into the compressed bitstream. The 3-D point clouds 802a and 802b are similar to Figure 7B and Figure 7C 3-D point cloud 762a, Fig. 6A 3-D point cloud 610, Figure 5A and Figure 5B 3-D point cloud 502, Figure 4 4 and 5. 3-D point cloud 802a is an HLOD, while 3-D point cloud 802b is an LLOD. In certain embodiments, 3D point cloud 802b is a compact version of 3D point cloud 802a by scaling transformation or by non-orthogonal projection onto a smaller video frame (such as frame 806).

[0259] 3-D point cloud 802a may be projected onto frame 804 via orthogonal projection. 3-D point cloud 802b may be projected onto frame 806 via non-orthogonal projection. Frames 804 and 806 may be similar to Figure 5A 802a and 802b. Frame 806 is a 3D point cloud 802a that is smaller than frame 804 (non-orthogonal projection), frame 804 is a 3D point cloud 802b that is smaller than frame 804 (non-orthogonal projection), frame 804 is a 3D point cloud 802a that is smaller than frame 804 (non-orthogonal projection), frame 806 is a 3D point cloud 802b that is smaller than frame 804 (non-orthogonal projection), frame 804 is a 3D point cloud 802a that is smaller than frame 804 (non-orthogonal projection), frame 806 is a 3D point cloud 802b that is smaller than frame 804 (non-orthogonal projection), frame 806 ...

[0260] The upsampling module 808 upsamples the frame 806 so that the frame has the same or similar dimensions as the frame 804. In certain embodiments, the amount by which the upsampling module 808 upsamples the frame 806 is predetermined. In certain embodiments, the upsampling module 808 identifies the dimensions of the frame 804 and upsamples the frame 806 to a corresponding amount so that the dimensions of the frame 804 and the dimensions of the frame 806 are the same.

[0261] Comparison engine 810 compares frame 804 with upsampled frame 806 and extracts the residual between the frames. For example, comparison engine 810 extracts the residual between each geometry frame and each texture frame representing 3-D point cloud 802a and 3-D point cloud 802b. That is, comparison engine 810 extracts the difference between each of frames 804 and each of frames 806. The difference between each of frames 804 and each of frames 806 represents the difference in resolution between the HLOD of 3-D point cloud 802a and the LLOD of 3-D point cloud 802b.

[0262] Coding engine 812 and coding engine 814 are similar to Figure 5A 802a.

[0263] In a particular embodiment, coded bitstream 820 and coded bitstream 822 are transmitted to a decoder separately. Coded bitstream 822 is a base layer, and coded bitstream 820 is an enhancement layer. The enhancement layer and the base layer can be transmitted to the decoder, and the decoder determines whether to decode only the base layer, or decode both the base layer and the enhancement layer.

[0264] In certain embodiments, the encoded bit stream 820 and the encoded bit stream 822 are multiplexed together by a multiplexer 818 and transmitted to the decoder as a single compressed bit stream 824. The multiplexer 818 is similar to Figure 5A Multiplexer 526.

[0265] Figure 8B Decoder 850 is shown. Decoder 850 shows a high-level overview of an embodiment of the present disclosure of an electronic device that decodes and reconstructs a point cloud for rendering. Decoder 850 is similar to Figure 5BThe decoder 850 includes two sub-decoders, decoder 850a and decoder 850b. Decoder 850a decodes HLOD, while decoder 850b decodes LLOD. Decoder 850 receives Fig. 8A The decoder 850 includes a demultiplexer 852, an auxiliary information upscaling 854, a texture upscaling 856, a geometry upscaling 858, and an occupancy map upscaling 859.

[0266] Auxiliary information upscaling 854, texture upscaling 856, geometry upscaling 858, and occupancy map upscaling 859 represent metadata for upsampling the decoded content from decoder 850b (via bitstream 822a). The upscaled content from decoder 850b is augmented with information from decoder 850a (bitstream 820a). For example, decoder 850b decodes an LLOD frame ( Fig. 8A 806), and decoder 850a decodes the difference between the HLOD of the extracted 3-D point cloud 802a and the LLOD of the 3-D point cloud 802b. The content received by decoder 850a is insufficient to reconstruct the 3-D point cloud 802a because the content is the difference between the HLOD of the 3-D point cloud 802a and the LLOD of the 3-D point cloud 802b. Therefore, upscaling metadata (auxiliary information upscaling 854, texture upscaling 856, geometry upscaling 858, and occupancy map upscaling 859) is applied to various data elements of decoder 850b, which then increases the corresponding content of decoder 850a. This enables decoder 850a to reconstruct the HLOD of the 3-D point cloud 802a.

[0267] Demultiplexer 852 is similar to Figure 5B Demultiplexer 552. Note that decoder 850 includes two separate decoders similar to decoder 550, namely decoder 850a and decoder 850b. Decoder 850a includes demultiplexer 860, decoding engine 862 (which decodes geometry frame 864, texture frame 866, and occupancy map frame 868), auxiliary information 870, and reconstruction engine 872. The components of decoder 850a are similar to Figure 5B For example, demultiplexer 860 is similar to demultiplexer 552, decoding engine 862 is similar to decoding engine 554, and reconstruction engine 872 is similar to reconstruction engine 564. Similarly, decoder 850b includes demultiplexer 880, decoding engine 882 (which decodes geometry frame 884, texture frame 886, and occupancy map frame 888), auxiliary information 890, and reconstruction engine 892. The components of decoder 850b are similar to Figure 5BFor example, demultiplexer 880 is similar to demultiplexer 552, decoding engine 882 is similar to decoding engine 554, and reconstruction engine 892 is similar to reconstruction engine 564.

[0268] If a single bitstream is received, such as bitstream 824, then demultiplexer 852 separates the received bitstream into enhancement layer bitstream 820a (similar to Fig. 8A enhancement layer bit stream 820) and base layer bit stream 822a (similar to Fig. 8A The enhancement layer bitstream 820a is then further demultiplexed by a demultiplexer 860. The base layer bitstream 822a is then further demultiplexed by a demultiplexer 880.

[0269] In certain embodiments, decoder 850 may receive two separate bitstreams (rather than a single bitstream 824), namely enhancement layer bitstream 820a (similar to Fig. 8A enhancement layer bit stream 820) and base layer bit stream 822a (similar to Fig. 8A The base layer bit stream 822 (not shown) is directed to the corresponding decoders 850a and 850b and bypasses the demultiplexer 852. Fig. 8A The enhancement layer bit stream 820 is demultiplexed by the demultiplexer 860. Fig. 8A The) base layer bit stream 822 is demultiplexed by the demultiplexer 880.

[0270] The process of reconstructing the corresponding 3-D point clouds 802a and 802b is similar to that described above. Figure 5B The process of reconstructing the corresponding 3-D point cloud 802a augments the content of the received bitstream 820a with the upscaling metadata applied to the content included in the bitstream 822a. The geometry upscaling 858 upsamples the decoded LLOD geometry frame 884. The LLOD geometry frame 884 is from Fig. 8A The decoded geometry frame of the frame 806. The upsampling of the LLOD geometry frame 884 includes upsampling both the spatial values ​​and the depth values.

[0271] For example, upsampling of the LLOD geometry frame 884 spatially scales two of the geometric coordinates, while upsampling of the depth values ​​of the LLOD geometry frame 884 scales a third geometric coordinate of the point cloud. The upscaled geometric coordinates are then included in the decoded geometry frame 864. The decoded geometry frame 864 represents the total extracted difference between the upsampled LLOD frame 806, the HLOD frame 804, and the upscaled geometry frame 884.

[0272] Texture upscaling 856 upsamples the decoded LLOD texture frame 886. LLOD texture frame 886 is Fig. 8A The upscaled texture frame is then included in the decoded texture frame 866. The decoded texture frame 866 represents the entire extracted difference between the upsampled LLOD frame 806, the HLOD frame 804, and the upscaled texture frame 886.

[0273] The auxiliary information upscaling 854 upsamples any auxiliary information 890 associated with the decoded LLOD geometry frame 884 and texture frame 886, respectively. Thereafter, the upsampled auxiliary information 890 is applied to the auxiliary information 870.

[0274] The occupancy map included in the occupancy map frame 888 is a binary map. Therefore, the occupancy map upscaling 859 uses the nearest neighbor interpretation to upsample the occupancy map frame 888. The upsampled occupancy map frame 888 can be applied to the occupancy map frame 868.

[0275] Note that patch information between the base layer (represented in bitstream 822a) and the enhancement layer (represented in bitstream 820a) is synchronized to ensure correlation between LLOD frames and HLOD frames. In certain embodiments, the encoder synchronizes the respective bitstreams.

[0276] Fig. 9 FIG. 4 shows an adaptive region of point cloud density according to an embodiment of the present disclosure. Fig. 9 The embodiment of modifying the ROI is for illustration only, and other embodiments may be used without departing from the scope of the present disclosure.

[0277] ROI encoding can assign a higher quality to the ROI and a lower quality to areas of the 3-D point cloud that are not included in the ROI. For example, assigning a higher quality to the ROI can be performed by increasing the point cloud density and reducing the quantization error of the geometry and color attributes in the ROI. For example, if the ROI is assigned to the face of a humanoid point cloud, the face of the point cloud can be encoded differently from the rest of the point cloud.

[0278] Fig. 9 The 3-D point cloud 902 and two frames, frame 910 and frame 920, are included. The 3-D point cloud 902 is similar to Fig. 8A and Figure 8B 3-D point clouds 802a and 802b, Figure 7B and Figure 7C 3-D point cloud 762a, Fig. 6A 3-D point cloud 610, Figure 5A and Figure 5B 3-D point cloud 502, Figure 4Frames 910 and 920 represent patches of 3-D point cloud 902. Frame 910 is a uniform point density for all patches, while frame 920 indicates an area of ​​ROI based adaptive point density.

[0279] Frame 910 includes a number of patches, such as patch 914. Patch 914 depicts a face of 3-D point cloud 902. Patch 914 is encoded with the same point density as other patches included in the frame. Frame 920 includes a number of patches, such as patch 924. Patch 924 depicts a face of 3-D point cloud 902. Patch 924 is projected with a higher point density than other patches included in frame 920. Patch 924 has a higher point density by scaling it to a different scaling factor. Different scaling factors may be included in the image. Figure 5A 924. The flag 508 of the encoder 500 is used to signal the decoder 550 to apply a scaling factor on a per-patch basis when reconstructing the 3-D point cloud 902. When a face is specified as an ROI, encoding the face of the point cloud (as indicated by patch 924) with a higher point density than the rest of the point cloud makes the face of the 3-D point cloud 902 have a higher quality than the body of the 3-D point cloud 902. In a particular embodiment, if an ROI is indicated, frame 910 is not transmitted in the compressed bitstream, while frame 920 is transmitted in the compressed bitstream. In a particular embodiment, changing the scaling factor of the ROI can modify the point density, resolution, scaling, etc. on a per-patch basis.

[0280] Non-orthogonal projections can be used to indicate the ROI of a point cloud at the patch level. For example, x, y, and z are the coordinates of a point in a patch of a point cloud, and xs, ys, and zs are the inputs to an encoder such as Figure 5A The scaled point cloud is represented in the x-coordinate as "xs=(x+offset_x)*scale_factor_x". The scaled point cloud is represented in the y-coordinate as "ys=(y+offset_y)*scale_factor_y". The scaled point cloud is represented in the z-coordinate as "zs=(z+offset_z)*scale_factor_z". If xr, yr, and zr are decoders (such as Figure 5B Where xo, yo and zo are the coordinates of the reconstructed patch of the point cloud at the output of the decoder 550), and xo, yo and zo are the coordinates of the final scaled output point in the patch, the output of each point is expressed by the following equation 4.

[0281] Equation 4:

[0282] xo=xr / scale_factor_x+offset_x

[0283] yo=yr / scale_factor_y+offset_y

[0284] zo=zr / scale_factor_x+offset_z

[0285] In certain embodiments, non-adjacent portions of the point cloud may be located as adjacent patches on the 2-D frame. Figure 5A The frame packing 512 organizes and places the various patches at a certain position on the frame so as to minimize the size of the frame. As a result, non-adjacent parts of the point cloud can generally be located as adjacent patches on the 2-D frame. If non-adjacent patches are adjacent to each other on the 2-D frame, then when the patches are reconstructed into a point cloud, the pixels of one patch adjacent to another patch can be mixed with the adjacent patches, which causes artifacts. Artifacts are visible defects in the point cloud that appear when the decoder reconstructs the point cloud.

[0286] Fig.10 FIG. 1 shows an adaptive point size and shape of a point cloud according to an embodiment of the present disclosure. Fig.10 The embodiment of adapting the shape and size of points in the point cloud is for illustration only, and other embodiments may be used without departing from the scope of the present disclosure.

[0287] Fig.10 The reconstructed point cloud 1002a and the reconstructed point cloud 1002b are included. The reconstructed point cloud 1002a includes an artifact 1004. The reconstructed point cloud 1002b is the reconstructed point cloud 1002a after adjusting the shape of the points, the size of the points, or a combination thereof to remove the artifact 1004.

[0288] In certain embodiments, when reconstructing the point cloud, the decoder may perform a smoothing step at the boundary lines of the patch in order to minimize visible artifacts. In certain embodiments, the decoder may adjust the size of the points along the patch boundary in order to minimize visible artifacts. In certain embodiments, the decoder may adjust the shape of the points along the patch boundary in order to minimize visible artifacts.

[0289] Such as Figure 5B The decoder of the decoder 550 can reconstruct a 3-D point cloud from multiple 2-D frames. When reconstructing the 3-D point cloud, each point in the reconstructed point cloud is displayed as a single pixel on a display device (such as a Figure 3 The point cloud may be displayed on a display 355 of the electronic device 300. For example, the ratio of the point size to the pixel on the display may be adjusted to remove artifacts at the patch boundary. The resolution of the displayed point cloud may be reduced by increasing the number of individual points.

[0290] To adjust the point size at the patch boundary, the decoder identifies points near the boundary of the reconstructed point cloud. The decoder can then exclude points that are different from adjacent points. For example, the decoder can identify points that are dissimilar to adjacent points based on color, texture, geometric position, etc. The decoder can compare these points to identify any points that exceed a threshold level of dissimilarity. If the point is excluded, the decoder can increase the point size of the points near the patch boundary to reduce the presence of any holes in the reconstructed point cloud. The encoder may include metadata that signals the average point size that the decoder should use when rendering the point cloud. In a particular embodiment, the decoder may change the shape of the points near the patch boundary to reduce the presence of any holes in the reconstructed point cloud. The encoder may include metadata similar to the metadata of Table 1 to indicate the shape of one or more points. In addition, based on the received metadata and signaling, the point size can be changed on a patch, ROI, or region of the point cloud.

[0291] Fig.11 An example flow chart for decoding a point cloud according to an embodiment of the present disclosure is shown. Fig.11 A flowchart 1100 is depicted for point cloud decoding. For example, Fig.11 The process depicted in is described as Figure 1 Client devices 106-114, Figure 5B Decoder 550, Figure 8B Any implementation of a decoder of , and including a decoder similar to Figure 2 Server 200 00, Figure 3 The internal components of the electronic device 300. The embodiment of the flowchart 1100 is for illustration only, and other embodiments may be used without departing from the scope of the present disclosure.

[0292] The process begins with a decoder, such as decoder 550, receiving a compressed bitstream 1102. The received bitstream may include an encoded point cloud that is mapped onto multiple 2-D frames, compressed, then transmitted and ultimately received by the decoder.

[0293] The process decodes the compressed bitstream (1104). The decoder 550 decodes the compressed bitstream into two 2-D frames representing the 3-D point cloud. Each of the 2-D frames includes a patch set. Each patch in the patch set includes a cluster of points in the 3-D point cloud. The clusters of points correspond to attributes (such as geometry and texture) associated with the 3-D point cloud. A patch in the patch set, the patch set, and the 2-D frame correspond to respective access levels representing the 3-D point cloud.

[0294] The process then identifies metadata indicating a modification of one of the respective access levels 1106. The metadata may indicate a modification of at least one of a scale, a resolution, a point density, an offset, a rotation, a point size, and a point shape for the respective access levels representing the 3-D point cloud.

[0295] The process may also identify a first flag and a second flag (1108). The first flag signals when metadata is applied to a first access level among the access levels. The first flag may be referred to as an enable flag. The second flag indicates metadata associated with a modification of the first access level. The second flag may be referred to as an existence flag. If the first flag may be included at an access level above the first access level, the first flag indicates whether a portion of the first access level is modified. The second flag is included at the first access level to indicate the portion of the first access level that is modified by the metadata.

[0296] When the first access level corresponds to a set of patches included in a 2-D frame, a first flag is signaled in the 2-D frame. Similarly, when the first access level corresponds to a set of patches included in a 2-D frame, a second flag is signaled at the first access level, and the second flag indicates a specific patch that is modified by the metadata.

[0297] When the first access level corresponds to a 2-D frame, a first flag is signaled in the 2-D frame. Similarly, when the first access level corresponds to a 2-D frame, a second flag is signaled at the first access level, and the second flag indicates that a patch set included in the 2-D frame is modified through metadata.

[0298] When the first access level corresponds to a 2-D frame, a first flag is signaled in the frame sequence. Similarly, when the first access level corresponds to a 2-D frame, a second flag is signaled at the first access level, and the second flag indicates that each of the 2-D frames included in the frame sequence is modified by metadata.

[0299] In response to recognizing both the first flag and the second flag, the process can also read metadata from the bitstream (1108). The bitstream includes metadata indicating the modification of the access level. For example, when the decoder recognizes both flags, it indicates to the decoder that metadata for modifying content within the 2-D frame when generating the 3-D point cloud is present in the bitstream.

[0300] In response to recognizing the first flag or the second flag, the process may also assign predetermined parameters to the metadata (1110). That is, if the decoder recognizes only one of the two flags, the predetermined parameters are assigned to the metadata. When the access level to which the metadata is applied is the highest level, the predetermined parameters are default values. For example, if the access level includes (i) patch, (ii) frame, (iii) GOF, (iv) sequence, and (v) video, the highest level is the video level. The predetermined parameters may be metadata for an access level that is one level higher than the current access level. For example, metadata applied to the frame level may be metadata applied to the GOF level.

[0301] The process generates a 3-D point cloud using the set of 2-D frames and the metadata (1112). When reconstructing the 3-D point cloud, the decoder can apply any received metadata to the patch. In a particular embodiment, if metadata is identified with the patch, the decoder 550 identifies whether the metadata is a scaling factor. In response to identifying that the metadata is a scaling factor, the decoder 550 applies the inverse of the scaling factor to each point in the 3-D point cloud to increase the resolution of the set of 2-D frames.

[0302] In a particular embodiment, if the decoder 550 receives two bitstreams of different resolutions, one of which is lower than the other. The decoder can decode the two bitstreams. The decoder 550 can identify whether the metadata includes a scaling factor and an offset. Thereafter, the decoder applies the scaling factor, offset, and patch included in the higher resolution bitstream.

[0303] The decoder may further identify points included in a set of patches located at the border of each patch based on the generated occupancy map. Thereafter, the decoder 550 may adjust at least one of a size of a point located at the border of each patch and a shape of a point located at the border of each patch based on the metadata.

[0304] Fig.12 An example flowchart for encoding a point cloud according to an embodiment of the present disclosure is shown. Fig.12 A flowchart 1200 is depicted for point cloud encoding. For example, Fig.12 The process depicted in is described as Figure 1 Server 104, Figure 1 any one of the client devices 106-114, Figure 5A The encoder 500 is implemented and includes Figure 2 Server 200 00, Figure 3 The internal components of the electronic device 300 are similar to the internal components of the electronic device 300. The embodiment of the flowchart 1200 is for illustration only, and other embodiments may be used without departing from the scope of the present disclosure.

[0305] The process begins with an encoder such as Figure 5A The encoder 500 of the embodiment of the present invention generates a 2-D frame (1202) representing a 3-D point cloud. Each of the 2-D frames includes a set of patches. Each patch in the set of patches includes a cluster of points in the 3-D point cloud. The cluster of points corresponds to an attribute associated with the 3-D point cloud. A patch in the set of patches, the set of patches, and the 2-D frame correspond to respective access levels representing the 3-D point cloud.

[0306] In a particular embodiment, the 2-D frame includes a first set of 2-D frames and a second set of 2-D frames. When generating the second set of 2-D frames, the encoder 500 applies scaling to each point in the 3-D point cloud to reduce the resolution of the second set of 2-D frames compared to the first set of 2-D frames. The encoder 500 upsamples the second set of 2-D frames to match the dimensional size of the first set of 2-D frames. The encoder then compares the first set of 2-D frames with the upsampled second set of 2-D frames. After comparing the first set of 2-D frames with the upsampled second set of 2-D frames, the encoder 500 extracts the difference in points between the first set of 2-D frames and the upsampled second set of 2-D frames.

[0307] In a particular embodiment, when the encoder 500 is generating a 2-D frame, the encoder 500 identifies points included in a set of patches located at the border of each patch based on the generated occupancy map. The encoder 500 then generates metadata to adjust at least one of a size of the points located at the border of each patch and a shape of the points located at the border of each patch.

[0308] The process generates metadata indicating a modification of one of the respective access levels. (1204) The metadata indicates a modification of at least one of a scale, a resolution, a point density, an offset, a rotation, a point size, and a point shape for the respective access levels representing the 3-D point cloud.

[0309] The process generates a flag to signal when metadata is applied to a first access level among the access levels (1206). In a particular embodiment, the flag includes a first flag and a second flag. For example, the first flag is an enable flag and the second flag is an existence flag. The first flag can signal when metadata is applied to the first access level, and the second flag indicates metadata associated with a modification of the first access level. In order to signal when metadata is applied to the first access level, the encoder 500 includes a first flag that is one access level higher than the first access level, and the first flag indicates whether a portion of the first access level is modified. The decoder also includes a second flag for the first access level to indicate the portion of the first access level that is modified by the metadata.

[0310] When the first access level corresponds to a set of patches included in a 2-D frame, the first flag is signaled in the 2-D frame. Similarly, the first access level corresponds to a set of patches included in the 2-D frame, the second flag is signaled at the first access level, and the second flag indicates a specific patch that is modified by the metadata.

[0311] When the first access level corresponds to a 2-D frame, the first flag is signaled at the 2-D frame, and the second flag is signaled at the first access level. Similarly, when the first access level corresponds to a 2-D frame, the first flag is signaled at the 2-D frame, and the second flag indicates that a patch set included in the 2-D frame is modified by metadata.

[0312] When the first access level corresponds to a 2-D frame, the first flag is signaled at the frame sequence. Similarly, when the first access level corresponds to a 2-D frame, the second flag is signaled at the first access level, and the second flag indicates that each of the 2-D frames included in the frame sequence is modified by metadata.

[0313] The process encodes the 2-D frame, flags, and metadata to generate a compressed bitstream (1208). The process transmits the compressed bitstream (1210). The compressed bitstream may ultimately be transmitted to a decoder, such as decoder 550.

[0314] Although these drawings illustrate different examples of user equipment, various changes may be made to these drawings. For example, the user equipment may include any number of each component of any suitable arrangement. Generally, the drawings do not limit the scope of the present disclosure to any (multiple) specific configurations. In addition, although the drawings illustrate operating environments in which various user equipment features disclosed in this patent document may be used, these features may be used in any other suitable system.

[0315] Any description in this application should not be interpreted as implying that any particular element, step or function is an essential element that must be included in the scope of the claims. The scope of the subject matter of the patent application is limited only by the claims. In addition, no claim is intended to cause exclusion unless the exact words "means for..." are interspersed with participles. Any other terms in the claims, including but not limited to the use of "mechanism", "module", "device", "unit", "component", "element", "member", "device", "machine", "system", "processor" or "controller", are understood by the applicant to refer to structures known to technicians in the relevant field and are not intended to cause exclusion.

[0316] Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method for point cloud decoding performed by a decoding device, the method comprising: receiving a compressed bitstream for a three-dimensional 3-D point cloud, including scaling information and offset information for processing the 3-D point cloud; decoding a compressed bitstream into two-dimensional 2-D frames representing a 3-D point cloud, each of the 2-D frames comprising a set of patches, each patch in the set of patches comprising a point of an object represented by the 3-D point cloud, each of the points being represented by a coordinate in a coordinate system; identifying scaling information indicating at least one scaling factor for each patch in the set of patches, wherein each of the at least one scaling factor is applied to coordinates of a point; identifying offset information indicating at least one offset for each patch in the set of patches, wherein each of the at least one offset is applied to a coordinate of a point; and A 3D point cloud including the object is rendered based on the at least one scaling factor and the at least one offset.

2. The method according to claim 1, in, signaling, via the information for processing the 3-D point cloud, for one or more patches in the patch set, a point size and a point shape for each patch; and The rendering of the 3-D point cloud includes adjusting the point size and point shape of the points in the one or more patches.

3. The method according to claim 1, further comprising: in, The rendering of the 3-D point cloud includes scaling the one or more patches based on the scaling factor when rendering the 3-D point cloud.

4. The method according to claim 3, in, The scaling information includes, for the first patch, a first scaling factor for a first dimension of the first patch and a second scaling factor for a second dimension of the first patch.

5. The method according to claim 1, further comprising: in, The rendering of the 3-D point cloud includes: when rendering the 3-D point cloud, moving the positions of points in one or more patches based on the offset information.

6. The method according to claim 1, in, The offset information includes: for the first patch, a first dimensional offset for a first dimension of the first patch and a second dimensional offset for a second dimension of the first patch.

7. The method according to claim 1, in, The scaling information includes: for the first patch, the scaling factor of the "x" dimension and the scaling factor of the "y" dimension, The offset information includes: for the first patch, the "x" dimension offset and the "y" dimension offset, wherein the 3-D point cloud is rendered by reconstructing a first patch at an output position based on the scaling information and the offset information, and The method further comprises: identifying a first coordinate of an output position of the patch as a function of an "x" dimension offset plus a result of scaling a first input position coordinate of the patch by an "x" dimension scaling factor; and A second coordinate of the reconstructed position of the patch is identified as a function of the "y" dimension offset plus a result of scaling the second input position coordinate of the patch by the "y" dimension scaling factor.

8. A method for point cloud encoding performed by an encoding device, the method comprising: generating a compressed bitstream comprising two-dimensional 2-D frames representing a three-dimensional 3-D point cloud, each of the 2-D frames comprising a set of patches, each patch in the set of patches comprising a point of an object represented by the 3-D point cloud, each of the points being represented by a coordinate in a coordinate system; generating scaling information and offset information for processing a 3-D point cloud, wherein the scaling information indicates at least one scaling factor for each patch in the set of patches, wherein each of the at least one scaling factor is applied to coordinates of a point; generating offset information indicating at least one offset for each patch in the set of patches, wherein each of the at least one offset is applied to a coordinate of a point; and A compressed bitstream for the 3-D point cloud including the scaling information and the offset information is sent to a decoding device.

9. The method according to claim 8, in, Via information for processing the 3-D point cloud, for one or more patches in the patch set, a dot size and a dot shape are signaled for each patch to adjust the dot size and the dot shape of the points in the one or more patches.

10. The method according to claim 8, in, The point size values ​​and point shape identifiers are used to indicate adjustments to the size and shape of the object's points based on proximity to the boundaries of the patch in the 2-D.

11. The method according to claim 8, in, The scaling information for the one or more patches includes, for a first patch, a first scaling factor for a first dimension of the first patch and a second scaling factor for a second dimension of the first patch.

12. The method according to claim 8, in, The offset information for the one or more patches includes, for a first patch, a first dimensional offset for a first dimension of the first patch and a second dimensional offset for a second dimension of the first patch.

13. A decoding device, comprising: Transceiver; as well as A processor coupled to the transceiver and configured to: receiving a compressed bitstream for a three-dimensional 3-D point cloud, including scaling information and offset information for processing the 3-D point cloud; decoding a compressed bitstream into two-dimensional 2-D frames representing a 3-D point cloud, each of the 2-D frames comprising a set of patches, each patch in the set of patches comprising a point of an object represented by the 3-D point cloud, each of the points being represented by a coordinate in a coordinate system; identifying scaling information indicating at least one scaling factor for each patch in the set of patches, wherein each of the at least one scaling factor is applied to coordinates of a point; identifying offset information indicating at least one offset for each patch in the set of patches, wherein each of the at least one offset is applied to a coordinate of a point; and A 3D point cloud including the object is rendered based on the at least one scaling factor and the at least one offset.

14. An encoding device comprising: Transceiver; as well as A processor coupled to the transceiver and configured to: generating a compressed bitstream comprising two-dimensional 2-D frames representing a three-dimensional 3-D point cloud, each of the 2-D frames comprising a set of patches, each patch in the set of patches comprising a point of an object represented by the 3-D point cloud, each of the points being represented by a coordinate in a coordinate system; generating scaling information and offset information for processing a 3-D point cloud, wherein the scaling information indicates at least one scaling factor for each patch in the set of patches, wherein each of the at least one scaling factor is applied to coordinates of a point; generating offset information indicating at least one offset for each patch in the set of patches, wherein each of the at least one offset is applied to a coordinate of a point; and A compressed bitstream for the 3-D point cloud including the scaling information and the offset information is sent to a decoding device.

Images