Single-pass boundary detection in video-based point cloud compression

Abstract

The present disclosure relates to a communication method and system for integrating a fifth generation (5G) communication system for supporting a higher data rate than a fourth generation (4G) system with Internet of Things (IoT) technology. The present disclosure can be applied to smart services based on 5G communication technology and IoT-related technologies, such as smart homes, smart buildings, smart cities, smart cars, connected cars, healthcare, digital education, smart retail, security and safety services. A decoding device includes: a communication interface configured to receive a bit stream; and a processor configured to decode a first frame, a second frame, and an occupancy map frame from the bit stream, reconstruct a 3D point cloud using the first frame, the second frame, and the occupancy map frame, and perform a single pass of the occupancy map frame to identify points of the reconstructed 3D point cloud.

Description

Technical Field

[0001] The present disclosure relates generally to multimedia data and more particularly to devices and methods for compressing and decompressing point clouds. Background Art

[0002] In order to meet the increased demand for wireless data services since the deployment of 4G communication systems, efforts have been made to develop improved 5G or quasi-5G communication systems. Therefore, 5G or quasi-5G communication systems are also referred to as "super 4G networks" or "post-LTE systems". 5G communication systems are considered to be implemented in higher frequency (millimeter wave) bands (e.g., 60GHz bands) in order to achieve higher data rates. In order to reduce the propagation loss of radio waves and increase the transmission distance, beamforming, massive multiple-input multiple-output (MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, and massive antenna technology are discussed in 5G communication systems. In addition, in 5G communication systems, development is being carried out for system network improvements based on advanced small cells, cloud radio access networks (RANs), ultra-dense networks, device-to-device (D2D) communications, wireless backhaul, mobile networks, collaborative communications, coordinated multipoint (CoMP), receiving-end interference cancellation, etc. In 5G systems, hybrid FSK and QAM modulation (FQAM) and sliding window superposition coding (SWSC) have been developed as advanced coded modulation (ACM), and filter bank multi-carrier (FBMC), non-orthogonal multiple access (NOMA), and sparse code multiple access (SCMA) as advanced access technologies.

[0003] The Internet, a human-centric network of connected devices where humans generate and consume information, is now evolving into the Internet of Things (IoT), in which distributed entities such as things exchange and process information without human intervention. The Internet of Everything (IoE), a combination of IoT technology and big data processing technologies connected to cloud servers, has emerged. Since IoT implementation requires technological elements such as sensing technology, wired / wireless communication and network infrastructure, service interface technology, and security technology, sensor networks, machine-to-machine (M2M) communication, and machine-type communication (MTC) have recently been researched. Such an IoT environment can provide intelligent internet technology services that create new value for human life by collecting and analyzing data generated between connected things. Through the convergence and combination of existing information technology (IT) and various industrial applications, the IoT can be applied in a variety of fields, including smart homes, smart buildings, smart cities, smart cars or connected vehicles, smart grids, healthcare, smart appliances, and advanced medical services.

[0004] In line with this, various attempts have been made to apply 5G communication systems to IoT networks. For example, technologies such as sensor networks, machine-type communications (MTC), and machine-to-machine (M2M) communications can be implemented using beamforming, MIMO, and array antennas. The application of cloud radio access networks (RAN), which are the aforementioned big data processing technologies, can also be considered an example of the convergence between 5G and IoT technologies.

[0005] Thanks to the ready availability of powerful handheld devices such as smartphones, three-hundred-sixty-degree (360°) video is becoming a new way to experience immersive video. 360° video can provide consumers with an immersive "real-life," "present" experience by capturing the world in 360° view. Users can interactively change their viewpoint and dynamically view any part of the captured scene or object they desire. Display and navigation sensors can track the user's head movements in real time to determine the area of ​​the 360° video that the user wants to view. Multimedia data that is inherently three-dimensional (3D), such as point clouds, can be used in immersive environments. Summary of the Invention

[0006] Technical issues

[0007] A point cloud is a collection of points representing an object in 3D space. Point clouds are common in various applications such as games, 3D maps, visualization, medical applications, augmented reality, virtual reality, autonomous driving, multi-view playback, 6-degree-of-freedom (6DoF) immersive media, etc. If uncompressed, point clouds typically require a large amount of bandwidth for transmission. Due to the requirement for large bit rates, point clouds are typically compressed before transmission. Compressing 3D objects such as point clouds typically requires dedicated hardware. In order to avoid dedicated hardware compression of 3D point clouds, 3D point clouds can be manipulated onto traditional two-dimensional (2D) frames that can be compressed and reconstructed on different devices for viewing by users. Compressing and decompressing 2D frames can produce artifacts that reduce the visual quality of the point cloud.

[0008] Technical Solution

[0009] The present disclosure provides single-pass boundary detection in video-based point cloud compression.

[0010] In one embodiment, a decoding device for point cloud decoding is provided. The decoding device includes a communication interface and a processor. The communication interface is configured to receive a bitstream. The processor is configured to decode a first frame including pixels from the bitstream. A portion of the pixels of the first frame represent the geometric structure positions of points of a 3D point cloud organized into tiles corresponding to respective point clusters of the 3D point cloud. The processor is also configured to decode a second frame including pixels from the bitstream. A portion of the pixels of the second frame represent attribute information of points of the 3D point cloud, and the positions of the pixels in the second frame correspond to the respective positions of the pixels in the first frame. The processor is also configured to decode an occupancy map frame from the bitstream. The occupancy map frame includes pixels that identify the portion of pixels in the first frame that represent the geometric structure positions of points of the 3D point cloud and the portion of pixels in the second frame that represent the attribute information of points of the 3D point cloud. In addition, the processor is configured to reconstruct the 3D point cloud using the first frame, the second frame, and the occupancy map frame. The processor is further configured to perform a single pass of the occupancy map frame to identify, based on values ​​of pixels within the occupancy map frame, points of the reconstructed 3D point cloud corresponding to a subset of pixels in the first and second frames that are within a proximity threshold from a boundary of any of the tiles.

[0011] In another embodiment, a method for point cloud decoding is provided. The method includes receiving a bitstream. The method also includes decoding a first frame comprising pixels from the bitstream. A portion of the pixels of the first frame represent geometric locations of points of a 3D point cloud organized into tiles corresponding to respective point clusters of the 3D point cloud. The method further includes decoding a second frame comprising pixels from the bitstream. A portion of the pixels of the second frame represent attribute information of points of the 3D point cloud, and the positions of the pixels in the second frame correspond to the respective positions of the pixels in the first frame. The method also includes decoding an occupancy map frame from the bitstream. The occupancy map frame includes pixels identifying the portion of pixels in the first frame representing the geometric locations of points of the 3D point cloud and the portion of pixels in the second frame representing the attribute information of the points of the 3D point cloud. Additionally, the method includes reconstructing the 3D point cloud using the first frame, the second frame, and the occupancy map frame. The method also includes performing a single pass through the occupancy map frame to identify, based on the values ​​of the pixels within the occupancy map frame, points of the reconstructed 3D point cloud corresponding to a subset of pixels in the first and second frames that are within a proximity threshold from a boundary of any of the tiles.

[0012] In another embodiment, an encoding device for point cloud encoding is provided. The encoding device includes a communication interface and a processor. The processor is configured to generate a first frame and a second frame for a 3D point cloud. The first frame includes pixels, wherein a portion of the pixels represents the geometric structure positions of points of the 3D point cloud organized into tiles corresponding to respective point clusters of the 3D point cloud. The second frame includes pixels, wherein a portion of the pixels of the second frame represents attribute information of the points of the 3D point cloud. The positions of the pixels in the second frame correspond to the respective positions of the pixels in the first frame. The processor is further configured to generate an occupancy map frame. The occupancy map frame includes pixels that identify the portion of the pixels in the first frame that represent the geometric structure positions of the points of the 3D point cloud and the portion of the pixels in the second frame that represent the attribute information of the points of the 3D point cloud. The processor is further configured to encode the first frame to generate a geometry substream, encode the second frame to generate an attribute substream, and encode the occupancy map frame to generate an occupancy map substream. In response to decoding the geometry substream, the attribute substream, and the occupancy map substream, the processor is configured to reconstruct the 3D point cloud using the first frame, the second frame, and the occupancy map frame. Additionally, the processor is configured to perform a single pass of the occupancy map frame to identify points of the reconstructed 3D point cloud corresponding to a subset of pixels in the first and second frames that are within a proximity threshold from a boundary of any of the tiles based on values ​​of the pixels within the occupancy map frame. The processor is further configured to combine the geometry substream, the attribute substream, and the occupancy map substream to generate a bitstream. The communication interface is configured to transmit the bitstream.

[0013] Other technical features may be apparent to those skilled in the art from the following drawings, descriptions, and claims.

[0014] Beneficial effects of the present invention

[0015] Embodiments of the present disclosure provide systems and methods for converting a point cloud into a 2D representation that can be transmitted and then reconstructed into a point cloud for rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0017] Figure 1 An example communication system according to an embodiment of the present disclosure is shown;

[0018] Figure 2 and Figure 3 An example electronic device according to an embodiment of the present disclosure is shown;

[0019] Figure 4AAn example 3D point cloud according to an embodiment of the present disclosure is shown;

[0020] Figure 4B A diagram showing a point cloud surrounded by multiple projection planes according to an embodiment of the present disclosure;

[0021] Figure 4C and 4D The embodiment of the present disclosure includes Figure 4A Example 2D frames of tiles of a 3D point cloud;

[0022] Figure 4E shows example color artifacts in a reconstructed 3D point cloud according to an embodiment of the present disclosure;

[0023] Figure 5A A block diagram illustrating an example environment architecture according to an embodiment of the present disclosure is shown;

[0024] Figure 5B An example block diagram of an encoder according to an embodiment of the present disclosure is shown;

[0025] Figure 5C An example block diagram of a decoder according to an embodiment of the present disclosure is shown;

[0026] Figure 6A An example method for identifying boundary points around a single query pixel according to an embodiment of the present disclosure is shown;

[0027] Figure 6B An example method for identifying boundary points around a single query pixel according to an embodiment of the present disclosure is shown;

[0028] Figure 6C An example method for identifying boundary points according to an embodiment of the present disclosure is shown;

[0029] Figure 7A An example sliding window for identifying boundary points according to an embodiment of the present disclosure is shown;

[0030] Figure 7B shows an example portion of an occupancy map including tiles according to an embodiment of the present disclosure;

[0031] Figures 7C-7M An example process of identifying boundary points by moving a sliding window on an occupancy map to identify boundary points according to an embodiment of the present disclosure is shown;

[0032] Figure 8 An example method for encoding a point cloud according to an embodiment of the present disclosure is shown; and

[0033] Figure 9An example method for decoding a point cloud according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0034] Before proceeding with the detailed description below, it may be beneficial to set forth the definitions of specific words and phrases used throughout this patent document. The term "couple" and its derivatives refer to any direct or indirect communication between two or more elements, regardless of whether these elements are in physical contact with each other. The terms "send," "receive," and "communicate," and their derivatives encompass both direct and indirect communication. The terms "include," "comprise," and their derivatives mean, but are not limited to. The term "or" is inclusive, meaning and / or. The phrase "associated with..." and its derivatives mean including, included within, interconnected with, including, contained within, connected to or connected with, coupled to or coupled with, communicable with, collaborative with, interlaced, juxtaposed, close to, bound to or bound with, having, having the property of, having a relationship with, etc. The term "controller" refers to any device, system, or part thereof that controls at least one operation. Such a controller can be implemented in hardware or a combination of hardware and software and / or firmware. The functions associated with any particular controller can be centralized or distributed, whether local or remote. The phrase "at least one of" when used with a list of items means that different combinations of one or more of the listed items can be used, and only one item in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A, B, and C.

[0035] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, or a portion thereof that are suitable for implementation with suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), random access memory (RAM), a hard drive, a compact disk (CD), a digital video disc (DVD), or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical, or other communication links that transmit transient electrical signals or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and later rewrite data, such as rewritable optical disks or erasable memory devices.

[0036] Definitions for other specific words and phrases are provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0037] Discussed below Figures 1 to 9 The various embodiments used to describe the principles of the present disclosure in this patent document are illustrative only and should not be interpreted in any way as limiting the scope of the present disclosure. Those skilled in the art will understand that the principles of the present disclosure can be implemented in any suitably arranged system or device.

[0038] Augmented reality (AR) is an interactive experience in which objects residing in a real-world environment are enhanced with virtual objects, virtual information, or both. Virtual reality (VR) is a rendered version of a visual scene where the entire scene is generated by a computer. In certain embodiments, AR and VR include both visual and audio experiences. Visual rendering is designed to mimic the visual stimuli and auditory sensory stimuli (if available) of the real world as naturally as possible to the observer or user as the user moves within the limits defined by the application or AR or VR scene. For example, VR places the user in an immersive world that responds to the user's head movements. At the video level, VR is achieved by providing a video experience that covers as much of the field of view (FOV) as possible and synchronizing the perspective of the rendered video with the head movements.

[0039] Many different types of devices are capable of providing an immersive experience associated with AR or VR. One example device is a head-mounted display (HMD). An HMD represents one of the many types of devices that provide AR and VR experiences to users. An HMD is a device that enables a user to view a VR scene and adjust the displayed content based on the movement of the user's head. Typically, an HMD relies on a dedicated screen that is integrated into the device and connected to an external computer (tethered), or on a device (such as a smartphone) that is plugged into the HMD (untethered). The first approach utilizes one or more lightweight screens and benefits from high computing power. In contrast, smartphone-based systems utilize greater mobility and can be produced more cheaply. In both cases, the generated video experience is the same. Note that as used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0040] A point cloud is a virtual representation of a three-dimensional object. For example, a point cloud is a collection of points in 3D space, and each point is located at a specific geometric location within the 3D space and includes one or more attributes such as color. A point cloud can be similar to a virtual object in a VR or AR environment. A mesh is another type of virtual representation of an object in a VR or AR environment. A point cloud or mesh can be an object, multiple objects, a virtual scene (which includes multiple objects), etc. Point clouds and meshes are commonly used in various applications, including gaming, 3D projection, visualization, medicine, AR, VR, autonomous driving, multi-view playback, 6-DOF immersive media, to name a few. As used herein, the terms point cloud and mesh are used interchangeably.

[0041] Point clouds represent volumetric visual data. A point cloud consists of multiple points located in a 3D space, where each point in the 3D point cloud includes a geometric structure position represented by a 3-tuple (X, Y, Z) coordinate value. When each point is identified by three coordinates, the precise position in the 3D environment or space is identified. The position of each point in the 3D environment or space can be relative to the origin, other points of the point cloud, or a combination thereof. The origin is the location where the X, Y, and Z axes intersect. In some embodiments, the points are located on the outer surface of the object. In other embodiments, the points are distributed throughout both the internal structure and the outer surface of the object. In other embodiments, the points are located along the surface of the object and can be located within the internal area of ​​the point cloud.

[0042] In addition to the geometric location of the point (the position of the point in 3D space), each point in the point cloud can also include attributes such as color, texture, reflectivity, intensity, surface normal, etc. In some embodiments, a single point in the 3D point cloud can have multiple attributes. In some applications, point clouds can also be used to approximate light field data, where each point includes color information related to multiple views (R, G, B triples or Y, U, V triples).

[0043] A single point cloud can include billions of points, each associated with a geometric location and one or more attributes. The geometric location and each additional attribute associated with a point occupy a certain number of bits. For example, the geometric location of a single point in a point cloud may occupy thirty bits. For example, if each geometric location of a single point is defined using X, Y, and Z values, then each coordinate (X, Y, and Z) uses ten bits, for a total of thirty bits. Similarly, the attribute that specifies the color of a single point may consume twenty-four bits. For example, if the color components of a single point are defined based on red, green, and blue values, then each color component (red, green, and blue) uses eight bits, for a total of twenty-four bits. As a result, a single point with 10 bits of geometric attribute data per coordinate and eight bits of color attribute data per color value occupies fifty-four bits. Each additional attribute increases the number of bits required for a single point. If a frame includes one million points, the number of bits per frame is fifty-four million bits (fifty-four bits per point multiplied by one million points per frame). If the frame rate is thirty frames per second and is uncompressed, 1.62 gigabytes per second (54 million bits per frame multiplied by thirty frames per second) will be transmitted from one electronic device to another electronic device in order for the second device to display the point cloud. Therefore, due to the size and complexity of the data associated with a single point cloud, transmitting an uncompressed point cloud from one electronic device to another electronic device uses a large amount of bandwidth. Therefore, the point cloud is compressed before transmission.

[0044] Embodiments of the present disclosure take into account that, due to the bandwidth required to transmit the point cloud, compressing the point cloud is necessary to speed up and improve the transmission of the point cloud from one device (such as a source device) to another device (such as a display device). Certain dedicated hardware components can be used to meet real-time requirements or reduce delays or lags in transmitting and rendering 3D point clouds; however, such hardware components are typically expensive. In addition, many video codecs are not able to encode and decode 3D video content (such as point clouds). By leveraging existing 2D video codecs to compress and decompress point clouds, the encoding and decoding of point clouds can be widely available without the need for new or dedicated hardware. According to embodiments of the present disclosure, when a point cloud is converted from a 3D representation to a 2D representation, leveraging existing video codecs can be used to compress and reconstruct the point cloud. In some embodiments, the conversion of the point cloud from a 3D representation to a 2D representation includes projecting point clusters of the 3D point cloud onto a 2D frame by creating patches. Thereafter, a video codec such as HEVC, AVC, VP9, ​​VP8, VVC, etc. can be used to compress the 2D frames representing the 3D point cloud similar to a 2D video.

[0045] To send a point cloud from one device to another, the 3D point cloud is divided into multiple point clusters, which are represented as tiles on a 2D frame. The 2D frame may include projections of the 3D point cloud relative to different projection planes. The frame may include values ​​representing the geometric positions of the points. The frame may also represent different attributes of the point cloud. For example, an attribute frame may include values ​​representing color information associated with each point. The decoder uses the frames to reconstruct the 3D point cloud so that the point cloud can be rendered, displayed, and then viewed by the user.

[0046] When the point cloud is deconstructed to fit into multiple 2D frames and compressed, the frames can be transmitted using less bandwidth than that used to transmit the original point cloud. Figures 4A to 4D The various stages of projecting a point cloud onto different planes and subsequently storing the projections into a 2D frame are shown. For example, the point cloud can be a 360° view of an object. Figure 4A Two views of the 3D point cloud are shown. Figure 4B The process of projecting 3D point clouds onto different planes is shown. Figure 4A After projecting the point cloud onto different planes, Figure 4C and Figure 4D A geometry frame and an attribute frame (the attribute frame represents the color of the points of the 3D point cloud) are shown separately, which include tiles corresponding to various projections.

[0047] Embodiments of the present disclosure provide systems and methods for converting a point cloud into a 2D representation that can be transmitted and then reconstructed into a point cloud for rendering. In some embodiments, the point cloud is deconstructed into a plurality of tiles that are packed into frames. In some embodiments, the frames include tiles of the same attributes. When a tile in one frame and another tile in a second frame are located at the same coordinates, the point of the 3D point cloud represented by the one tile in the one frame corresponds to the same point represented by the other tile in the second frame. For example, a pixel at a position (u, v) in a frame representing a geometric structure is the geometric structure position of a pixel at the same (u, v) position in a frame representing an attribute such as color. In other embodiments, a tile in a frame represents multiple attributes associated with a point of a point cloud, such as the geometric structure position and color of a point in 3D space.

[0048] The encoder can separate the geometric structure information and attribute information from each point. The encoder groups (or clusters) the points of the 3D point cloud for different projection planes and then stores the point groups as tiles on a 2D frame. The tiles representing the geometric structure and attribute information are packaged into geometry video frames and attribute video frames, respectively, where each pixel within any tile corresponds to a point in 3D space. The geometry video frame is used to encode the geometric structure information, and the corresponding attribute video frame is used to encode the attributes of the point cloud (such as color). The two lateral coordinates of the 3D point (relative to the projection plane) correspond to the column index and row index in the geometry video frame (u, v) plus a lateral offset that determines the position of the entire tile within the video frame. The depth of the 3D point is encoded as the value of the pixel in the video frame plus the depth offset of the tile. The depth of the 3D point cloud depends on whether the projection of the 3D point cloud is obtained from XY coordinates, YZ coordinates or XZ coordinates.

[0049] After the frame is generated, it can be compressed by making full use of various video compression codecs, image compression codecs, or both. For example, the encoder first generates a geometry frame and then compresses the geometry frame using a 2D video codec such as HEVC. In order to encode the attribute frame (such as the color of the 3D point cloud), the encoder decodes the encoded geometry frame for reconstructing the 3D coordinates of the 3D point cloud. The encoder smoothes the reconstructed point cloud. Thereafter, the encoder interpolates the color value of each point from the color value of the original point cloud. The interpolated color values ​​are then packed into a compressed color frame.

[0050] The encoder may also generate an occupancy map (also called an occupancy map frame) that shows the locations of projected points in the 2D video frame. For example, because a tile may not occupy the entire generated frame, the occupancy map indicates which pixels of the geometry frame and attribute frame correspond to points of the point cloud, and which pixels are empty / invalid and do not correspond to points of the point cloud. In some embodiments, the occupancy map frame is compressed. The compressed geometry frame, compressed color frame (and any other attribute frames), and occupancy map frame may be multiplexed to generate a bitstream. The encoder or another device then sends the bitstream comprising the 2D frame to a different device.

[0051] The decoder receives the bitstream, decompresses it into frames, and reconstructs the point cloud based on the information within the frames. After reconstructing the point cloud, the 3D point cloud can be rendered and displayed for user observation. In some embodiments, the frames representing the geometry and the frames representing the different attributes are encoded and decoded separately. In other embodiments, the frames representing the geometry and the frames representing the different attributes are encoded and decoded together.

[0052] Embodiments of the present disclosure provide systems and methods for improving the reconstruction of 3D point clouds. Improving the reconstruction of 3D point clouds reduces the processing power required to reconstruct the 3D point clouds and improves the visual quality of the 3D point clouds. For example, smoothing points of a 3D point cloud corresponding to pixels located at or near tile boundaries can improve the visual appearance of the point cloud while reducing the amount of data transmitted to represent the points.

[0053] According to an embodiment of the present disclosure, an architecture and method for performing point cloud compression and decompression using a video codec are provided. When a 3D point cloud is converted from a 3D representation to a 2D representation, the points of the 3D point cloud are clustered into groups and projected onto a frame, where the clustered points generate tiles that are packed onto the 2D frame. Due to the size constraints of certain 2D frames, two tiles that are not adjacent to each other on the 3D point cloud may be packed to be adjacent to each other in a single frame. When two non-adjacent tiles of a point cloud are packed adjacent to each other in a 2D frame, a block-based video codec may unintentionally mix pixels from one tile with pixels from another tile. When pixels from one tile are unintentionally included in another tile, visible artifacts may occur at the tile boundaries when the decoder reconstructs the point cloud.

[0054] Therefore, embodiments of the present disclosure provide systems and methods for identifying points of a reconstructed 3D point cloud that are represented by pixels at or near tile boundaries in a frame. Because the identified points are represented by pixels at or near tile boundaries in a frame, these points are referred to as boundary points. The boundary points can then be smoothed to remove visual artifacts, thereby improving the visual appearance of the point cloud. When using a single pass of the occupied frame to identify boundary points, the runtime of the encoder or decoder can be reduced by more than 30%.

[0055] Figure 1 An example communication system 100 is shown in accordance with an embodiment of the present disclosure. Figure 1 The embodiment of the communication system 100 shown is for illustration only. Other embodiments of the communication system 100 may be used without departing from the scope of the present disclosure.

[0056] The communication system 100 includes a network 102 that facilitates communication between various components in the communication system 100. For example, the network 102 can transmit IP packets, frame forwarding frames, asynchronous transfer mode (ATM) cells, or other information between network addresses. The network 102 includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other one or more communication systems at one or more locations.

[0057] In this example, network 102 facilitates communication between server 104 and various client devices 106-116. Client devices 106-116 may be, for example, smartphones, tablets, laptops, personal computers, wearable devices, HMDs, etc. Server 104 may represent one or more servers. Each server 104 includes any suitable computing or processing device that can provide computing services to one or more client devices (such as client devices 106-116). Each server 104 may, for example, include one or more processing devices, one or more memories for storing instructions and data, and one or more network interfaces to facilitate communication over network 102. As described in more detail below, server 104 may send a compressed bitstream representing a point cloud to one or more display devices, such as client devices 106-116. In some embodiments, each server 104 may include an encoder.

[0058] Each client device 106-116 represents any suitable computing or processing device that interacts with at least one server (such as server 104) or other computing device via network 102. Client devices 106-116 include desktop computers 106, mobile phones or mobile devices 108 (such as smartphones), PDAs 110, laptop computers 112, tablet computers 114, and head-mounted display (HMD) 116. However, any other or additional client devices may be used in communication system 100. Smartphones represent one type of mobile device 108, which is a handheld device with a mobile operating system and an integrated mobile broadband cellular network connection for voice, short message service (SMS), and internet data communications. Head-mounted display (HMD) 116 can display a 360° scene including one or more 3D point clouds. In some embodiments, any of client devices 106-116 may include an encoder, a decoder, or both. For example, mobile device 108 may record a video and then encode the video so that it can be sent to one of client devices 106-116. In another example, laptop computer 112 may be used to generate a virtual 3D point cloud that is then encoded and sent to one of client devices 106 - 116 .

[0059] In this example, some client devices 108-116 communicate indirectly with the network 102. For example, the mobile device 108 and the PDA 110 communicate via one or more base stations 118, such as cellular base stations or eNodeBs (eNBs). Additionally, the laptop 112, the tablet 114, and the HMD 116 communicate via one or more wireless access points 120, such as IEEE 802.11 wireless access points. Note that these are for illustration only, and each of the client devices 106-116 can communicate directly with the network 102 or indirectly with the network 102 via any suitable intermediary device or network. In some embodiments, the server 104 or any of the client devices 106-116 can be used to compress the point cloud, generate a bitstream representing the point cloud, and send the bitstream to another client device, such as any of the client devices 106-116.

[0060] In certain embodiments, any of the client devices 106-114 securely and efficiently transmits information to another device, such as, for example, the server 104. In addition, any of the client devices 106-116 can trigger information transmission between itself and the server 104. Any of the client devices 106-114 can function as a VR display when attached to a mobile phone via a cradle and can function similarly to the HMD 116. For example, the mobile device 108 can function similarly to the HMD 116 when attached to a cradle system and worn on the user's eyes. The mobile device 108 (or any other client device 106-116) can trigger information transmission between itself and the server 104.

[0061] In some embodiments, any of the client devices 106-116 or the server 104 may create a 3D point cloud, compress a 3D point cloud, send a 3D point cloud, receive a 3D point cloud, render a 3D point cloud, or a combination thereof. For example, the server 104 receives the 3D point cloud, decomposes the 3D point cloud to fit on a 2D frame, and compresses the frame to generate a bitstream. The bitstream may be sent to a storage device (such as an information repository) or one or more of the client devices 106-116. For another example, one of the client devices 106-116 may receive the 3D point cloud, decompose the 3D point cloud to fit on a 2D frame, and compress the frame to generate a bitstream that may be sent to a storage device (such as an information repository), another of the client devices 106-116, or the server 104.

[0062] although Figure 1 One example of a communication system 100 is shown, but may be Figure 1Various changes may be made. For example, the communication system 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of this disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be employed, but these features may be employed in any other suitable system.

[0063] although Figure 1 One example of a communication system 100 is shown, but may be Figure 1 Various changes may be made. For example, the communication system 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of this disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be employed, but these features may be employed in any other suitable system.

[0064] Figure 2 and Figure 3 An example electronic device according to an embodiment of the present disclosure is shown. Specifically, Figure 2 An example server 200 is shown and may represent Figure 1 The server 200 may represent one or more encoders, decoders, local servers, remote servers, cluster computers, components acting as a single seamless resource pool, cloud-based servers, etc. The server 200 may be composed of Figure 1 One or more of the client devices 106-116 or another server accesses.

[0065] Server 200 may represent one or more local servers, one or more compression servers, or one or more encoding servers, such as encoders. In some embodiments, the encoders may perform decoding. Figure 2 As shown, server 200 includes a bus system 205 that supports communication between at least one processing device (such as processor 210 ), at least one storage device 215 , at least one communication interface 220 , and at least one input / output (I / O) unit 225 .

[0066] Processor 210 executes instructions that may be stored in memory 230. Processor 210 may include any suitable number and type of processors or other devices in any suitable arrangement. Example types of processors 210 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application-specific integrated circuits, and discrete circuits. In certain embodiments, processor 210 may encode a 3D point cloud stored in storage device 215. In certain embodiments, when a 3D point cloud is encoded by an encoder, the encoder also decodes the encoded 3D point cloud to ensure that when the point cloud is reconstructed, the reconstructed 3D point cloud matches the 3D point cloud before encoding.

[0067] Memory 230 and permanent storage 235 are examples of storage 215 that represent any structure capable of storing and facilitating retrieval of information, such as temporary or permanent data, program code, or other suitable information. Memory 230 may represent random access memory or any other suitable volatile or non-volatile storage. For example, the instructions stored in memory 230 may include instructions for decomposing a point cloud into tiles, instructions for packing tiles onto 2D frames, instructions for compressing 2D frames, and instructions for encoding 2D frames in a particular order to generate a bitstream. The instructions stored in memory 230 may also include instructions for rendering, such as through a VR headset, such as a Figure 1 The persistent storage 235 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, flash memory, or an optical disk.

[0068] The communication interface 220 supports communication with other systems or devices. For example, the communication interface 220 may include a Figure 1 The communication interface 220 may include a network interface card or wireless transceiver for communicating with the network 102. The communication interface 220 may support communication via any suitable physical or wireless communication link. For example, the communication interface 220 may send a bitstream containing a 3D point cloud to another device, such as one of the client devices 106-116.

[0069] The I / O unit 225 allows for the input and output of data. For example, the I / O unit 225 can provide a connection for user input via a keyboard, mouse, keypad, touch screen, or other suitable input device. The I / O unit 225 can also send output to a display, printer, or other suitable output device. However, it is noted that the I / O unit 225 can be omitted, such as when I / O interaction with the server 200 occurs via a network connection.

[0070] Note that although Figure 2 Described as indicating Figure 1The server 104 of FIG. 106 may be configured as a server 104, but the same or similar architecture may be used in one or more of the various client devices 106-116. For example, a desktop computer 106 or a laptop computer 112 may have a server 104 configured as a server 104. Figure 2 The same or similar structure as shown in .

[0071] Figure 3 An example electronic device 300 is shown and may represent Figure 1 The electronic device 300 may be a mobile communication device such as, for example, a mobile station, a user station, a wireless terminal, a desktop computer (similar to a Figure 1 Desktop computer 106), portable electronic device (similar to Figure 1 In some embodiments, Figure 1 One or more of the client devices 106-116 may include the same or similar configuration as the electronic device 300. In some embodiments, the electronic device 300 is an encoder, a decoder, or both. For example, the electronic device 300 may be used for data transmission, image or video compression, image or video decompression, encoding, decoding, and media rendering applications.

[0072] like Figure 3 As shown, the electronic device 300 includes an antenna 305, a radio frequency (RF) transceiver 310, a transmit (TX) processing circuit 315, a microphone 320, and a receive (RX) processing circuit 325. The RF transceiver 310 may include, for example, an RF transceiver, a Bluetooth transceiver, a WI-FI transceiver, a ZIGBEE transceiver, an infrared transceiver, and various other wireless communication signal transceivers. The electronic device 300 also includes a speaker 330, a processor 340, an input / output (I / O) interface (IF) 345, an input unit 350, a display 355, a memory 360, and a sensor 365. The memory 360 includes an operating system (OS) 361 and one or more applications 362.

[0073] The RF transceiver 310 receives incoming RF signals from an antenna 305 transmitted from an access point (such as a base station, a WI-FI router, or a Bluetooth device) or other device in a network 102 (such as WI-FI, Bluetooth, cellular, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network). The RF transceiver 310 downconverts the incoming RF signal to generate an intermediate frequency (IF) or baseband signal. The IF or baseband signal is sent to the RX processing circuitry 325, which generates a processed baseband signal by filtering, decoding, and / or digitizing the baseband or IF signal. The RX processing circuitry 325 sends the processed baseband signal to the speaker 330 (such as for voice data) or to the processor 340 for further processing (such as for web browsing data).

[0074] TX processing circuitry 315 receives analog or digital voice data from microphone 320 or other outgoing baseband data from processor 340. The outgoing baseband data may include web data, email, or interactive video game data. TX processing circuitry 315 encodes, multiplexes, and / or digitizes the outgoing baseband data to generate a processed baseband or intermediate frequency (IF) signal. RF transceiver 310 receives the outgoing processed baseband or IF signal from TX processing circuitry 315 and up-converts the baseband or IF signal into an RF signal that is transmitted via antenna 305.

[0075] The processor 340 may include one or more processors or other processing devices. The processor 340 may execute instructions stored in the memory 360 (such as the OS 361) to control the overall operation of the electronic device 300. For example, the processor 340 may control the reception of forward channel signals and the transmission of reverse channel signals via the RF transceiver 310, the RX processing circuit 325, and the TX processing circuit 315 according to well-known principles. The processor 340 may include any suitable number and type of processors or other devices in any suitable arrangement. For example, in some embodiments, the processor 340 includes at least one microprocessor or microcontroller. Example types of processors 340 include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application-specific integrated circuits, and discrete circuits.

[0076] The processor 340 is also capable of executing other processes and programs residing in the memory 360, such as operations to receive and store data. The processor 340 can move data into or out of the memory 360 as needed for the executed process. In some embodiments, the processor 340 is configured to execute one or more applications 362 based on the OS 361 or in response to a signal received from an external source or operator. For example, the application 362 may include an encoder, a decoder, a VR or AR application, a camera application (for still images and video), a video phone call application, an email client, a social media client, an SMS messaging client, a virtual assistant, etc. In some embodiments, the processor 340 is configured to receive and send media content.

[0077] The processor 340 is also coupled to an I / O interface 345 that provides the electronic device 300 with the ability to connect to other devices, such as the client devices 106 - 114 . The I / O interface 345 is the communication path between these accessories and the processor 340 .

[0078] The processor 340 is also coupled to an input unit 350 and a display 355. The operator of the electronic device 300 can use the input unit 350 to input data or input into the electronic device 300. The input unit 350 can be a keyboard, a touch screen, a mouse, a trackball, a voice input unit, or other device capable of serving as a user interface to allow the user to interact with the electronic device 300. For example, the input unit 350 may include a voice recognition processing unit to allow the user to enter voice commands. In another example, the input unit 350 may include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel can, for example, recognize touch input using at least one scheme (such as a capacitive scheme, a pressure-sensitive scheme, an infrared scheme, or an ultrasonic scheme). The input unit 350 can be associated with (one or more) sensors 365 and / or a camera by providing additional input to the processor 340. In some embodiments, the sensors 365 include one or more inertial measurement units (IMUs) (such as accelerometers, gyroscopes, and magnetometers), motion sensors, optical sensors, cameras, pressure sensors, heart rate sensors, altimeters, and the like. The input part 350 may further include a control circuit. In a capacitive scheme, the input part 350 may recognize a touch or proximity.

[0079] Display 355 can be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED), an active matrix OLED (AMOLED), or other display capable of presenting text and / or graphics, such as from a website, video, game, image, etc. Display 355 can be sized to fit within the HMD. Display 355 can be a single display screen or multiple display screens capable of creating a stereoscopic display. In some embodiments, display 355 is a heads-up display (HUD). Display 355 can display 3D objects, such as a 3D point cloud.

[0080] Memory 360 is coupled to processor 340. A portion of memory 360 may include RAM, and another portion of memory 360 may include flash memory or other ROM. Memory 360 may include a permanent storage device (not shown) representing any structure capable of storing and facilitating retrieval of information (such as data, program code, and / or other suitable information). Memory 360 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, flash memory, or an optical disk. Memory 360 may also include media content. Media content may include various types of media, such as images, videos, three-dimensional content, VR content, AR content, 3D point clouds, and the like.

[0081] The electronic device 300 also includes one or more sensors 365 that can measure physical quantities or detect the activation state of the electronic device 300 and convert the measured or detected information into electrical signals. For example, the sensor 365 may include one or more buttons for touch input, a camera, a gesture sensor, an IMU sensor (such as a gyroscope or gyro sensor and an accelerometer), an eye tracking sensor, an air pressure sensor, a magnetic sensor or a magnetometer, a grip sensor, a proximity sensor, a color sensor, a biophysical sensor, a temperature / humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalography (EEG) sensor, an electrocardiography (ECG) sensor, an IR sensor, an ultrasonic sensor, an iris sensor, a fingerprint sensor, a color sensor (such as a red, green, and blue (RGB) sensor), etc. The sensor 365 may also include a control circuit for controlling any of the sensors included therein.

[0082] As discussed in more detail below, one or more of these sensors 365 can be used to control a user interface (UI), detect UI input, determine the user's orientation and the direction the user is facing for three-dimensional content display recognition, etc. Any of these sensors 365 can be located within the electronic device 300, within a secondary device operably connected to the electronic device 300, within a cell phone configured to accommodate the electronic device 300, or within a single device in which the electronic device 300 includes a cell phone.

[0083] The electronic device 300 can create media content, such as generating a 3D point cloud or capturing (or recording) content via a camera. The electronic device 300 can encode the media content to generate a bitstream (similar to the server 200 described above) so that the bitstream can be sent directly to another electronic device or to a server such as a server. Figure 1 The electronic device 300 may receive the bit stream directly from another electronic device, or may receive the bit stream indirectly from another electronic device, such as through a Figure 1 The network 102 receives the bit stream indirectly.

[0084] When encoding media content such as a point cloud, the electronic device 300 may project the point cloud into multiple tiles. For example, point clusters of a point cloud may be grouped together and represented as tiles on a 2D frame. Tiles may represent individual attributes of the point cloud, such as geometry, color, etc. Tiles representing the same attributes may be packaged into separate 2D frames. The 2D frames are then encoded to generate a bitstream. During the encoding process, additional content such as metadata, flags, occupancy maps, geometry smoothing parameters, one or more attribute smoothing parameters, tile substreams, etc. may be included in the bitstream.

[0085] Similarly, when decoding media content including a representation of a 3D point cloud in a bitstream, the electronic device 300 decodes the received bitstream into frames. In some embodiments, the decoded bitstream also includes an occupancy map, a 2D frame, auxiliary information, etc. The geometry frame may include pixels indicating the geographic coordinates of points of the point cloud in 3D space. Similarly, the attribute frame may include pixels indicating the RGB (or YUV) color (or any other attribute) of each geometry point in 3D space. The auxiliary information may include one or more flags, or quantization parameter sizes, one or more thresholds, geometry smoothing parameters, one or more attribute smoothing parameters, tile substreams, or any combination thereof. After reconstructing the 3D point cloud, the electronic device 300 may render the 3D point cloud in three dimensions via the display 355.

[0086] although Figure 2 and Figure 3 An example of an electronic device is shown, but the Figure 2 and Figure 3 Make various changes. For example, you can combine, further subdivide or omit Figure 2 and Figure 3 As a specific example, the processor 340 may be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, as with computing and communications, electronic devices and servers may have a variety of configurations, and Figure 2 and Figure 3The present disclosure is not limited to any particular electronic device or server.

[0087] Figure 4A 、 Figure 4B 、 Figure 4C 、 Figure 4D and Figure 4E The various stages of generating a frame representing a 3D point cloud are shown. Specifically, Figure 4A An example 3D point cloud 400 is shown, in accordance with an embodiment of the present disclosure. Figure 4B Diagram 405 shows a point cloud surrounded by multiple projection planes according to an embodiment of the present disclosure. Figure 4C and 4D The embodiment of the present disclosure includes Figure 4A 2D frame of tiles of the 3D point cloud 400. For example, Figure 4C A 2D frame 430 is shown representing the geometric positions of points of the 3D point cloud 400, and Figure 4D Frame 440 is shown representing the colors (or another attribute) associated with points of 3D point cloud 400 . Figure 4E 4 shows example color artifacts in the reconstructed point cloud 450. In some embodiments, the reconstructed point cloud 450 is similar to Figure 4A 400 3D point clouds.

[0088] Figure 4A 3D point cloud 400 is a collection of data points in 3D space. Each point in 3D point cloud 400 includes a geometric location that provides the structure of the 3D point cloud and one or more attributes that provide information about each point (such as color, reflectivity, material, etc.). 3D point cloud 400 represents the entire 360° object. That is, the point cloud can be viewed from various angles, such as the front 402, the side and back 402, the top, and the bottom.

[0089] Figure 4B The graph 405 includes a point cloud 406. The point cloud 406 may be similar to Figure 4A3D point cloud 400. That is, point cloud 406 represents the entire 360° object. Point cloud 406 is surrounded by multiple projection planes (such as projection planes 410, 412, 414, 416, 418, and 420). Projection plane 410 is separated from projection plane 412 by a predefined distance. For example, projection plane 410 corresponds to projection plane XZ0, and projection plane 412 corresponds to projection plane XZ1. Similarly, projection plane 414 is separated from projection plane 416 by a predefined distance. For example, projection plane 414 corresponds to projection plane YZ0, and projection plane 416 corresponds to projection plane YZ1. In addition, projection plane 418 is separated from projection plane 420 by a predefined distance. For example, projection plane 418 corresponds to projection plane XY0, and projection plane 420 corresponds to projection plane XY1. Note that additional projection planes may be included, and the shapes formed by the projection planes may be different.

[0090] During the partitioning process, each point of the point cloud 406 is assigned to a specific projection plane (such as projection planes 410, 412, 414, 416, 418, and 420). Points that are close to each other and assigned to the same projection plane are grouped together to form a point cloud represented as a tile (such as Figure 4C and Figure 4D A cluster of any of the tiles shown in ).

[0091] Figure 4C and 4D 2D frame 430 and 2D frame 440 are shown respectively. Since frame 430 provides Figure 4A The geometric structure position of each point of the 3D point cloud 400 is represented by the frame 430, so the frame 430 is a geometric structure frame. The frame 430 includes a plurality of tiles (such as tile 432) representing the depth value of the 3D point cloud 400. The value of each pixel in the frame 430 is represented as a lighter or darker color and is related to the distance of each pixel from a specific projection plane (such as Figure 4B corresponds to a distance to one of the projection planes 410, 412, 414, 416, 418 and 420).

[0092] Because frame 440 provides Figure 4A Frame 440 is a color frame (a type of attribute) because it represents the color of each point in 3D point cloud 400. Frame 440 includes multiple tiles (such as tile 442) representing values ​​corresponding to the colors of points in 3D point cloud 400.

[0093] Figure 4C and Figure 4D Each tile in the frame can be identified by an index number. Similarly, each pixel within a tile can be identified by its position within the frame and the index number of the tile in which the pixel is located.

[0094] There is a correspondence (or mapping) between frame 430 and frame 440. That is, each pixel in frame 430 corresponds to a pixel at the same location in frame 440. Each color pixel in frame 440 corresponds to a pixel of a specific geometric structure in frame 430. For example, a mapping is generated between each pixel in frame 430 and each pixel in frame 440. For example, each pixel within tile 432 corresponds to a point in 3D space, and each pixel within tile 442 provides a color for the point of the 3D point cloud represented at the same location in tile 432. As shown in frames 430 and 440, some of the pixels correspond to valid pixels representing 3D point cloud 400, while other pixels (the black areas in the background) correspond to invalid pixels that do not represent 3D point cloud 400.

[0095] Non-adjacent points in the 3D space may ultimately be represented as pixels adjacent to each other in the frames 430 and 440. For example, two tiles that are not adjacent to each other in the 3D space may be represented as tiles adjacent to each other in the frames 430 and 440.

[0096] The frames 430 and 440 may be encoded using a video codec such as HEVC, AVC, VP9, ​​VP8, VVC, AV1, etc. The decoder receives a bitstream including the frames 430 and 440, reconstructs the geometry of the 3D point cloud from the frame 430, and colors the geometry of the point cloud based on the frame 440 to generate the following image: Figure 4E The reconstructed point cloud 450 is shown.

[0097] Figure 4E The reconstructed point cloud 450 should be similar to the 3D point cloud 400. When frames 430 and 440 are encoded and compressed, the values ​​corresponding to the pixels may be mixed by the block-based video codec. If pixels within a single tile of frame 430 are mixed, the effect is generally negligible when reconstructing the point cloud because the colors adjacent to each other within the tile are generally similar. However, if pixels at the boundary of one tile (such as tile 432) in the tiles of frame 430 are mixed with pixels of another tile, artifacts may be created when reconstructing the point cloud, similar to the following example. Figure 4E . Since the tiles may come from completely different parts of the point cloud, the coloring of the tiles can be different. In block-based video codecs, a coded block may contain pixels from tiles with very different coloring. This causes color to leak from a block to another block with a very different texture. As a result, visual artifacts are generated that reduce the visual quality of the point cloud. Similarly, if pixels at the border of one of the tiles are mixed with empty pixels (indicated by a black background and not corresponding to points of the point cloud), artifacts may be generated when reconstructing the point cloud because the points corresponding to the pixels switched by the empty pixels will not be reconstructed.

[0098] Reconstructed point cloud 450 shows artifact 452. Artifact 452 may be generated when a tile corresponding to the forehead of the model represented by 3D point cloud 400 and a tile corresponding to another portion of 3D point cloud 400 (such as a dress of the model represented by 3D point cloud 400) are packed into frame 430 close together. In this way, the color values ​​of the tile representing the portion of the dress may leak into the tile corresponding to the forehead of the model represented by 3D point cloud 400. In this example, the mixing of the color values ​​results in an artifact that appears as a crack or hole in the user's face, reducing the visual quality of reconstructed point cloud 450. Embodiments of the present disclosure provide systems and methods for removing artifacts by smoothing the reconstructed point cloud in the area of ​​the artifact while maintaining the quality of the point cloud. For example, points near tile boundaries of the reconstructed point cloud are identified and smoothed.

[0099] although Figure 4A 、 Figure 4B 、 Figure 4C and Figure 4D An example point cloud and a 2D frame representing the point cloud are shown, but Figure 4A 、 Figure 4B 、 Figure 4C and Figure 4D Various changes may be made. For example, a point cloud or mesh may represent a single object, while in other embodiments, the point cloud or mesh may represent multiple objects, scenery (such as a landscape), virtual objects in AR, etc. In another example, the tiles included in the 2D frame may represent other properties, such as brightness, material, etc. Figure 4A 、 Figure 4B 、 Figure 4C and Figure 4D The present disclosure is not limited to any particular 3D object and 2D frame representing the 3D object.

[0100] Figure 5A 、 Figure 5B and Figure 5C 1 shows a block diagram of an embodiment according to the present disclosure. Specifically, Figure 5A A block diagram of an example environment architecture 500 is shown, in accordance with an embodiment of the present disclosure. Figure 5B The embodiment according to the present disclosure is shown Figure 5A An example block diagram of an encoder 510, and Figure 5C The embodiment according to the present disclosure is shown Figure 5A An example block diagram of a decoder 550 is shown. Figure 5A 、 Figure 5B and Figure 5C The embodiments shown are for illustration only. Other embodiments may be used without departing from the scope of this disclosure.

[0101] like Figure 5AAs shown, the example environment architecture 500 includes an encoder 510 and a decoder 550 communicating via a network 502. The network 502 can be connected to Figure 1 In some embodiments, network 502 represents a "cloud" of computers interconnected by one or more networks, where a network is a computing system utilizing clustered computers and components that act as a single seamless resource pool when accessed. Additionally, in some embodiments, network 502 is connected to one or more servers (such as, Figure 1 Server 104, server 200), one or more electronic devices (such as, Figure 1 The network 502 is connected to the client devices 106-116, the electronic device 300), the encoder 510, and the decoder 550. In addition, in some embodiments, the network 502 can be connected to an information library (not shown) containing VR and AR media content, which can be encoded by the encoder 510, decoded by the decoder 550, or rendered and displayed on the electronic device.

[0102] In some embodiments, the encoder 510 and the decoder 550 may represent the server 104, Figure 1 one of the client devices 106-116, Figure 2 Server 200, Figure 3 300 or another suitable device. In certain embodiments, the encoder 510 and the decoder 550 can be a "cloud" of computers interconnected by one or more networks, each of which is a computing system that utilizes cluster computers and components to act as a single seamless resource pool when accessed through the network 502. In certain embodiments, a part of the components included in the encoder 510 or the decoder 550 can be included in different devices, such as multiple servers 104 or 200, multiple client devices 106-116 or other combinations of different devices. In certain embodiments, the encoder 510 is operably connected to the electronic device or server, and the decoder 550 is operably connected to the electronic device. In certain embodiments, the encoder 510 and the decoder 550 are the same device or are operably connected to the same device.

[0103] Below Figure 5B The encoder 510 is described in more detail in . Typically, the encoder 510 is received from a server such as a Figure 1 Server 104, Figure 2 The encoder 510 may receive 3D media content, such as a point cloud, from a server 200 or an information repository (such as a database) or another device at one of the client devices 106-116. In some embodiments, the encoder 510 may receive media content from multiple cameras and stitch the content together to generate a 3D scene including one or more point clouds.

[0104] The encoder 510 projects the points of the point cloud into a plurality of tiles representing the projections. The encoder 510 clusters the points of the point cloud into groups that are projected onto different planes such as the XY plane, the YZ plane, and the XZ plane. When projected onto a plane, each cluster of points is represented by a tile. The encoder 510 can project the point cloud into a two-dimensional space on a 2D frame. Note that the points of a 3D point cloud are located in a 3D space based on (X, Y, Z) coordinate values, but when the point is projected onto a 2D frame, the pixel representing the projected point is represented by the column and row index of the frame indicated by the coordinates (u, v). In addition, 'u' and 'v' can range from zero to the number of rows or columns in the depth image, respectively.

[0105] The encoder 510 packs the tiles representing the point cloud into 2D frames. The 2D frames can be video frames. Each of the 2D frames represents a specific attribute, such as one set of frames can represent geometry and another set of frames can represent an attribute (such as color). It should be noted that additional frames can be generated based on more layers and each with additional defined attributes.

[0106] The encoder 510 also generates an occupancy map based on the geometry frame and the attribute frame to indicate which pixels within the frame are valid. Generally, for each pixel within the frame, the occupancy map indicates whether the pixel is valid or invalid. For example, if the pixel at coordinate (u, v) in the occupancy map is valid, the corresponding pixel at coordinate (u, v) in the geometry frame and the corresponding attribute frame is also valid. If the pixel at coordinate (u, v) in the occupancy map is invalid, the decoder skips the corresponding pixel at coordinate (u, v) in the geometry frame and the attribute frame. In some embodiments, the occupancy map at position (u, v) can be either 1 or 0. Generally, the occupancy map is binary, so that each pixel has a value of 1 or 0. When the value of the pixel at position (u, v) in the occupancy map is 1, it indicates that the pixel at (u, v) in the attribute frame and the geometry frame is valid. In contrast, when the value of the pixel at position (u, v) in the occupancy map is zero, it indicates that the pixel at (u, v) in the attribute frame and the geometry frame is invalid.

[0107] In certain embodiments, the encoder 510 reconstructs a point cloud using the geometry frame, the attribute frame, and the occupancy map frame. The encoder 510 may identify points on the reconstructed 3D point cloud that are located on or near a boundary of one of the tiles in one of the frames. The encoder 510 may then determine whether to perform smoothing on the identified boundary points.

[0108] The encoder 510 sends the frames representing the point cloud as an encoded bitstream. If smoothing is performed on some or all of the identified boundary points, the bitstream may also include parameters from the smoothing. The bitstream may be sent over the network 502 to an information repository such as a database or an electronic device including a decoder such as decoder 550, or decoder 550 itself. Figure 5B The encoder 510 is described in more detail in .

[0109] Below Figure 5C The decoder 550, described in more detail in

[0045] , receives a bitstream representing media content, such as a point cloud. The bitstream may include data representing a 3D point cloud. In some embodiments, the decoder 550 may decode the bitstream and generate multiple frames, such as one or more geometry frames, one or more attribute frames, and one or more occupancy map frames. The decoder 550 uses the multiple frames to reconstruct a point cloud that can be rendered and viewed by a user. The decoder 550 may identify points on the reconstructed 3D point cloud that are represented on or near a boundary of one of the tiles on one of the frames. The decoder 550 may then determine whether to perform smoothing on the identified boundary points.

[0110] Figure 5B An encoder 510 is shown that receives a 3D point cloud 512 and generates a bitstream 546. The bitstream 546 includes data representing the 3D point cloud 512. The bitstream 546 may include multiple bitstreams and may be transmitted via Figure 5A The decoder 550 or the information repository is sent to another device, such as a decoder 550 or an information repository, over a network 502. The encoder 510 includes a tile generator and packer 514, one or more encoding engines (such as encoding engines 520a, 520b, 520c, and 520d, collectively referred to as encoding engines 520), a reconstruction engine 528, a boundary detection engine 530, a geometry smoothing engine 538a, an attribute smoothing engine 538b, and a multiplexer 542.

[0111] The 3D point cloud 512 may be stored in a memory (not shown) or received from another electronic device (not shown). The 3D point cloud 512 may be a single 3D object (similar to Figure 4A The 3D point cloud 512 may be a stationary object or a moving object.

[0112] The tile generator and packer 514 generates tiles by obtaining projections of the 3D point cloud 512 and packs the tiles into frames. In some embodiments, the tile generator and packer 514 separates the geometric structure information and attribute information of each point of the 3D point cloud 512. The tile generator and packer 514 can use two or more projection planes (such as Figure 4BThe points of the 3D point cloud 512 are clustered using two or more projection planes 410-420 to generate tiles. The geometry tiles are finally packed into a geometry frame 518.

[0113] The tile generator and packer 514 determines the best projection plane for each point in the 3D point cloud 512. When projected, each point cluster in the 3D point cloud 512 appears as a tile (also called a regular tile). A single point cluster can be represented by multiple tiles (located on different frames), where each tile represents a specific aspect of each point in the cluster. For example, a tile representing the geometric location of a point cluster is located on the geometry frame 518, and a tile representing the attributes of the point cluster is located on the attributes frame 534.

[0114] After determining the best projection plane for each point of the 3D point cloud 512, the tile generator and packer 514 segments the points into a tile data structure that is a packed frame, such as the geometry frame 518. As discussed above Figure 4C and 4D As shown, the tiles are organized by attributes and placed in corresponding frames, such as tile 432 being included in geometry frame 430 and tile 442 being included in attribute frame 440. Note that tiles representing different attributes of the same point cluster include a correspondence or mapping, where a pixel in one tile corresponds to the same pixel in another tile based on the positions of the pixels being at the same location in each frame.

[0115] The tile generator and packer 514 also generates tile information (providing information about the tiles, such as an index number associated with each tile), an occupancy frame 516 , a geometry frame 518 , and attribute information (which is used by the attribute generator 532 to generate an attribute frame 534 ).

[0116] The occupancy map frame 516 represents an occupancy map that indicates valid pixels in a frame, such as the geometry frame 518. For example, the occupancy map frame 516 indicates whether each pixel in the geometry frame 518 is a valid pixel or an invalid pixel. Each valid pixel in the occupancy map frame 516 corresponds to a pixel in the geometry frame 518 that represents a point in the 3D point cloud 512 in 3D space. Conversely, an invalid pixel is a pixel within the occupancy map frame 516 that corresponds to a point in the geometry frame 518 that does not represent the 3D point cloud 512 (such as a pixel in the geometry frame 518). Figure 4C and 4D 40). In some embodiments, one of the occupancy map frames 516 may correspond to both the geometry frame 518 and the attribute frame 534 (discussed below).

[0117] For example, when the tile generator and packer 514 generates the occupancy map frame 516, the occupancy map frame 516 includes a predefined value for each pixel, such as zero or one. For example, when the pixel at position (u, v) of the occupancy map is a zero value, it indicates that the pixel at (u, v) in the geometry frame 518 is invalid. Similarly, when the pixel at position (u, v) of the occupancy map is a value of one, it indicates that the pixel at (u, v) in the geometry frame 518 is valid and, therefore, includes information representing a point of the 3D point cloud.

[0118] The geometry frame 518 includes pixels representing the geometry values ​​of the 3D point cloud 512. The geometry frame 518 includes the geographic location of each point of the 3D point cloud 512. The geometry frame 518 is used to encode the geometry information of the point cloud. For example, the two lateral coordinates of the 3D point (relative to the projection plane) correspond to the column index and row index in the geometry video frame (u, v) plus a lateral offset that indicates the location of the entire tile within the video frame. The depth of the 3D point is encoded as the value of the pixel in the video frame plus the depth offset of the tile. The depth of the 3D point cloud depends on whether the projection of the 3D point cloud is obtained from XY coordinates, YZ coordinates, or XZ coordinates.

[0119] The encoder 510 includes one or more encoding engines. For example, the encoder 510 may include an encoding engine 520a, an encoding engine 520b, an encoding engine 520c, and an encoding engine 520d (collectively referred to as encoding engines 520). In some embodiments, a single encoding engine performs the operations of all individual encoding engines 520. In some embodiments, the encoding engine 520 may be configured to support data with 8-bit, 10-bit, 12-bit, 14-bit, or 16-bit precision. The encoding engine 520 may include a video or image codec such as HEVC, AVC, VP9, ​​VP8, VVC, EVC, AV1, etc. to compress 2D frames representing 3D point clouds.

[0120] One or more of the encoding engines 520 may compress information in a lossy or lossless manner.

[0121] In some embodiments, as shown, information such as tile information, occupancy map frame 516, geometry frame 518, and attribute frame 534 is encoded by independent encoding engines 520. For example, encoding engine 520a receives tile sequence information and performs tile sequence compression to generate tile substream 522. Encoding engine 520b receives occupancy map frame 516, performs occupancy map compression to generate occupancy map substream 524a. Encoding engine 520c receives geometry frame 518, performs geometry compression to generate geometry substream 526a. Encoding engine 520d receives attribute frame 534, performs attribute compression to generate attribute substream 536.

[0122] In some embodiments, the encoding engines 520b, 520c, and 520d may encode the occupancy map frame 516, the geometry frame 518, and the attribute frame 534 at a lower resolution. The lower resolution may result in a reduction in the overall size of the frame. When the occupancy map frame 516, the geometry frame 518, and the attribute frame 534 are encoded at a lower resolution, the frame may be Figure 5C After decoding by the decoding engine of decoder 550, the frame will need to be upsampled to the original size of the frame.

[0123] After encoding engine 520b generates occupancy map substream 524a, a decoding engine (not shown) may decode occupancy map substream 524a to generate a reconstructed occupancy map frame 524b. Similarly, after encoding engine 520c generates geometry substream 526a, a decoding engine (not shown) may decode geometry substream 524a to generate a reconstructed geometry frame 526b. In certain embodiments, a reconstruction engine 528 reconstructs the geometric positions of points of the 3D point cloud based on the reconstructed occupancy map frame 524b and the reconstructed geometry frame 526b.

[0124] The attribute generator 532 generates the attribute frame 534 . The attribute generator 532 may generate the attribute frame 534 based on the attribute information from the 3D point cloud 512 , the reconstructed occupancy map frame 524 b , the reconstructed geometry frame 526 b , and information provided by the tile generator and packer 514 .

[0125] For example, to generate one of the attribute frames 534 representing color, the geometry frame 518 is compressed by the encoding engine 520c using a 2D video codec such as HEVC. The geometry substream 526a is decoded by the reconstruction engine 528 and the geometry of the 3D point cloud is reconstructed. The reconstruction engine 528 reconstructs the geometric positions of the points of the 3D point cloud based on the reconstructed geometry frame 526b. The attribute generator 532 interpolates the attribute value (such as color) of each point from the color value of the input point cloud to the reconstructed point cloud (and, if smoothing is performed, to the smoothed geometric coordinates of the point cloud). The interpolated color is then segmented by the attribute generator 532 to match the same tiles as the geometry information. The attribute generator 532 then packs the interpolated attribute values ​​into the attribute frame 534 representing color.

[0126] Attribute frames 534 represent different attributes of the point cloud. For example, for each of the geometry frames 518, there may be one or more corresponding attribute frames 534. Attribute frames may include color, texture, normals, material properties, reflection, motion, and the like. In some embodiments, one of the attribute frames 534 may include a color value for each of the geometry points within one of the geometry frames 518, while another attribute frame may include a reflection value indicating the reflection level of each corresponding geometry point within the same geometry frame 518. Each additional attribute frame 534 represents other attributes associated with a particular geometry frame 518. In some embodiments, each geometry frame 518 has at least one corresponding attribute frame 534.

[0127] When the occupancy map frame 516, geometry frame 518, and attribute frame 534 are encoded by the encoding engine 520 and later decoded at the decoder 550, pixels from one tile may be unintentionally included in another tile. As a result, visible artifacts may appear in the reconstructed point cloud, thereby reducing the visual quality of the point cloud. For example, pixels within the geometry frame 518 may be slightly offset due to the encoding and decoding process. Typically, when a pixel is located in the middle of a tile, a slight offset may not significantly reduce the visual quality of the point cloud. However, a slight offset or switch of a pixel away from a tile to a location indicated as empty (or invalid) by the occupancy map may result in considerable artifacts because part of the image will not be rendered. Similarly, a slight offset or switch of pixels from one tile to another can cause considerable artifacts. For example, if Figure 4A If a tile of a face in the 3D point cloud 400 is packed next to a tile of a dress in the 3D point cloud 400, and the encoding / decoding process causes pixel groups to be moved from one tile to another, the reconstructed point cloud will have pixels on the dress that correspond to the face, and conversely, the reconstructed point cloud will have pixels on the face in the reconstructed and rendered point cloud that correspond to the dress. This shift can result in visible artifacts that reduce the visual quality of the point cloud.

[0128] To reduce the appearance of artifacts, points of the 3D point cloud that are represented as pixels near the boundaries of tiles in the 2D frame may be smoothed. To reduce the occurrence or appearance of visual artifacts and improve compression efficiency, smoothing may be applied to the positions of the points of the point cloud, to each identified attribute of the point cloud (such as color, reflectivity, etc.), or to both the geometry and attributes of the point cloud.

[0129] To smooth the geometry, attributes, or both of the 3D point cloud, the encoder 510, via a boundary detection engine 530, identifies pixels within the geometry frame 518 and the attribute frame 534 that are located at or near the boundaries of each tile. In certain embodiments, after the encoding engine 520d generates the attribute substream 536, a decoding engine (not shown) may decode the attribute substream 536 to generate a reconstructed attribute frame. The reconstructed attribute frame may be used to reconstruct the 3D point cloud by applying attribute information to the points of the reconstructed 3D point cloud. The boundary detection engine 530 then identifies pixels within the reconstructed geometry frame 526b and the reconstructed attribute frame that are located at or near the boundaries of each tile.

[0130] Boundary detection engine 530 examines pixels of reconstructed occupancy map frame 524b to identify boundaries of tiles within reconstructed geometry frame 526b. Alternatively, boundary detection engine 530 examines pixels of occupancy map frame 516 to identify boundaries of tiles within geometry frame 518.

[0131] The boundary detection engine 530 performs a single pass on the occupancy map frame to identify a subset of pixels corresponding to points of the point cloud that are located at or near the boundaries of the tiles in the 2D frame. To identify boundary points of the point cloud, the boundary detection engine 530 examines pixels within the reconstructed occupancy map frame 524b that are adjacent to the query pixel. For example, the boundary detection engine 530 identifies points of the reconstructed 3D point cloud that correspond to a subset of pixels in the geometry frame 518 based on the values ​​of the pixels within the occupancy map frame, the subset of pixels being located within a proximity threshold to the boundaries of any of the tiles. The subset of pixels includes one or more pixels that are a predefined distance from the query pixel. For example, the distance may include pixels within a distance of one pixel from the query pixel, a distance of two pixels from the query pixel, a distance of three pixels from the query pixel, and so on. As the distance increases, the number of identified boundary points will also increase.

[0132] The boundary detection engine 530 checks each pixel within the occupancy map frame 516. This check includes selecting a query pixel and identifying whether the query pixel is valid based on the value of the query pixel. If the query pixel is invalid, the boundary detection engine 530 continues to select new pixels within the occupancy map frame 516 until a valid query pixel is identified.

[0133] When identifying a valid query pixel, the boundary detection engine 530 examines the pixels of the first layer surrounding the query pixel. The pixels of the first layer include any pixels that are adjacent (neighboring) and touching the query pixel. For example, if the query pixel is in the middle of one of the occupied image frames 516, there are eight neighboring pixels. Figure 7A Pixels 704 of the first layer surrounding the query pixel 702 are shown.

[0134] The boundary detection engine 530 examines the pixels of the first layer to determine whether any of the pixels of the first layer are invalid. The boundary detection engine 530 also examines the pixels of the first layer to determine whether any of the neighboring pixels belong to a different tile than the tile to which the query pixel belongs. If the value of any of the pixels in the first layer indicates an invalid pixel, the query pixel is identified as a boundary pixel. Similarly, if any of the pixels in the first layer belong to a different tile than the tile to which the query pixel belongs, the query pixel is also identified as a boundary pixel.

[0135] The boundary detection engine 530 also examines the pixels of the first layer to determine whether any of the neighboring pixels are located at the edge of one of the occupancy map frames 516. The query pixel is identified as a boundary pixel if any of the pixels in the first layer are located at the edge of the occupancy map frame 516. For example, if there are typically eight pixels in the first layer, and the boundary detection engine 530 identifies fewer than eight pixels in the first layer, then the query pixel is identified as a boundary pixel.

[0136] Upon identifying the query pixel as a boundary pixel, the boundary detection engine 530 identifies a point of the 3D point cloud (corresponding to the query pixel) as a query point.

[0137] If all pixels in the first layer are valid, belong to the same tile as the query pixel, and are not located at the edge of the occupied frame 516, the boundary detection engine 530 also checks the pixels of the second layer surrounding the query pixel.

[0138] For example, if the query pixel is in the middle of one of the occupancy map frames 516, there are sixteen neighboring pixels. Figure 7A A second layer of pixels 706 is shown surrounding the query pixel 702. The pixels of the second layer are adjacent to and touching the pixels of the first layer such that the pixels of the second layer are a distance of two pixels from the query point.

[0139] The boundary detection engine 530 examines the pixels of the second layer to determine whether any of the pixels of the second layer are invalid. The boundary detection engine 530 also examines the pixels of the second layer to determine whether any of the neighboring pixels (in the second layer) belong to a tile different from the tile to which the query pixel belongs. If the value of any of the pixels in the second layer indicates an invalid pixel, the query pixel is identified as a boundary pixel. Similarly, if any of the pixels in the second layer belong to a tile different from the tile to which the query pixel belongs, the query pixel is also identified as a boundary pixel.

[0140] The boundary detection engine 530 also examines the pixels of the second layer to determine whether any of the neighboring pixels are located at the edge of one of the occupancy map frames 516. The query pixel is identified as a boundary pixel if any of the pixels in the second layer are located at the edge of the occupancy map frame 516. For example, if there are typically sixteen pixels in the second layer and the boundary detection engine 530 identifies fewer than sixteen pixels, the query pixel is identified as a boundary pixel.

[0141] Upon identifying the query pixel as a boundary pixel, the boundary detection engine 530 identifies the point of the 3D point cloud (corresponding to the query pixel) as the query point. If all pixels in the second layer are valid, belong to the same tile as the query pixel, and are not located at the edge of the occupancy map frame 516, the boundary detection engine 530 then selects a new valid query pixel. The boundary detection engine 530 continues to select new query pixels until all points of the point cloud have been examined. In some embodiments, the boundary detection engine 530 continues to select new query pixels until all occupancy map frames 516 have been examined.

[0142] The following syntax (1) is provided to illustrate the use of a single-pass process to identify boundary points of a 3D point cloud. The input of syntax (1) includes (i) the occupancy frame oFrame[i][j], i = 0 ... asps_frame_height-1, j = 0 ... asps_frame_width-1, (ii) x, the column index of the reconstructed point on the canvas, and (iii) y, the row index of the reconstructed point on the canvas. The output of syntax (1) is the variable boundaryPoint, which is a Boolean variable that identifies whether the point is near the tile boundary, which is used to indicate whether the reconstructed point is smoothly related to the geometric structure or attribute.

[0143]

[0144] Additionally, with respect to the above syntax (1), if i is not in the range of 0 to (asps_frame_height-1) (inclusive), or j is not in the range of 0 to (asps_frame_width-1) (inclusive), then oFrame[i][j] is assumed to be equal to 0.

[0145] After the boundary detection engine 530 identifies a subset of pixels within a proximity threshold of a boundary of any one of the tiles, the geometry smoothing engine 538a and the attribute smoothing engine 538b can perform smoothing on the identified points of the 3D point cloud. To perform smoothing, smoothing parameters (such as geometry smoothing parameters 540a and attribute smoothing parameters 540b) are generated.

[0146] A multiplexer 542 combines the tile substream 522 , the occupancy map substream 524 a , the geometry substream 526 a , the geometry smoothing parameters 540 a , the attribute substream 536 , and the attribute smoothing parameters 540 b to create a bitstream 546 .

[0147] Figure 5C Decoder 550 is shown, which includes a demultiplexer 552, one or more decoding engines (such as decoding engine 554a, decoding engine 554b, decoding engine 554c, and decoding engine 554d), a reconstruction engine 556, a geometry smoothing engine 558a, an attribute smoothing engine 558b, and a boundary detection engine 530.

[0148] The decoder 550 receives a bitstream 546, such as the bitstream generated by the encoder 510. The demultiplexer 552 separates the bitstream 546 into one or more substreams representing different information. For example, the demultiplexer 552 separates various data streams into separate substreams, such as the tile substream 522, the occupancy map substream 524a, the geometry substream 526a, the attribute substream 536, the geometry smoothing parameters 540a, and the attribute smoothing parameters 540b.

[0149] Decoder 550 includes one or more decoding engines. For example, decoder 550 may include decoding engine 554a, decoding engine 554b, decoding engine 554c, and decoding engine 554d (collectively referred to as decoding engines 554). In some embodiments, a single decoding engine performs the operations of all individual decoding engines 554.

[0150] The decoding engine 554a decodes the attribute substream 536 into an attribute frame 534a. The attribute frame 534a is similar to Figure 5B Attribute frame 534a, except that one or more pixels may be offset due to encoding and decoding of the frame to produce attribute frame 534a.

[0151] The decoding engine 554b decodes the geometry substream 526a into a geometry frame 526c. The geometry frame 526c is similar to Figure 5B 526b, except that one or more pixels may be offset due to encoding and decoding of the frame to produce geometry frame 526c.

[0152] The decoding engine 554d decodes the occupancy map substream 524a into an occupancy map frame 524c. The occupancy map frame 524c is similar to Figure 5B occupancy map frame 524b, except that one or more pixels may be offset due to encoding and decoding of the frame to produce occupancy map frame 524c.

[0153] The decoding engine 554d decodes the tile substream 522 into tile information 522a. The tile information 522a is similar to Figure 5B The tile information shown is before being encoded.

[0154] After decoding the tile information 522a, occupancy map frame 524c, geometry frame 526c, and attribute frame 534a, the reconstruction engine 556 generates a reconstructed point cloud. The reconstruction engine 556 reconstructs the point cloud. The reconstruction engine 556 is similar to Figure 5B Reconstruction engine 528.

[0155] The boundary detection engine 530 identifies boundary points of the reconstructed point cloud based on the values ​​of the pixels within the occupancy map frame 524c. Figure 5B The boundary detection engine 530 of the encoder 510 identifies boundary points of the reconstructed point cloud in a similar manner.

[0156] After identifying boundary points of the reconstructed point cloud, the geometry smoothing engine 558a applies geometry smoothing parameters 540a and smoothes the identified points of the reconstructed point cloud.

[0157] The attribute transformer 560 modifies the attributes of the reconstructed point cloud based on the reconstructed point cloud modified by the geometry smoothing engine. After the attribute transformer 560 modifies the attributes, the attribute smoothing engine 558b applies the attribute smoothing parameters 540b and smoothes the attributes of the reconstructed point cloud. Thereafter, the reconstructed point cloud 564 can be rendered and displayed.

[0158] although Figure 5A shows an environment architecture 500, Figure 5B An encoder 510 is shown, and Figure 5C Decoder 550 is shown, but can be Figure 5A 、 Figure 5B 、 Figure 5C and Figure 4D For example, the environment architecture 500 may include any number of encoders or decoders.

[0159] Figure 6A 、 Figure 6B and Figure 6B A process for identifying boundary points that are within a boundary proximity threshold from any of the tiles is described. For example, Figure 6A An example method 600 for identifying boundary points around a single query point is shown according to an embodiment of the present disclosure. Figure 6B An example method 620 for identifying boundary points around a single query point is shown according to an embodiment of the present disclosure. Figure 6C An example method 650 for identifying boundary points based on the values ​​of pixels in an occupancy map according to an embodiment of the present disclosure is shown. Methods 600, 620, and 650 may be performed by the server 104, Figure 1Any client device 106-116, Figure 2 Server 200, Figure 3 electronic device 300, Figure 5A and Figure 5B Encoder 510, Figure 5A and Figure 5C For ease of explanation, methods 600, 620, and 650 are described as being performed by the decoder 550 or any other suitable device or system. Figure 5A and Figure 5B The boundary detection engine 530 of the encoder 510 and decoder 550 is executed.

[0160] Figure 6A The method 600 describes the overall process of determining whether a single point of a 3D point cloud represented on a 2D frame (such as a geometry frame, an attribute frame, and an occupancy map frame) is a boundary point.

[0161] At step 602, the boundary detection engine 530 selects a single pixel from the occupancy map frame as a query pixel. The pixel can be located anywhere within the occupancy map frame. In some embodiments, the boundary detection engine 530 first selects a specific pixel, such as the pixel represented by index number zero. Thereafter, the boundary detection engine 530 selects the next pixel, such as the pixel represented by index number 1. This process continues until all pixels in the occupancy map or the selected points or all points of the 3D point cloud have been identified as boundary points or non-boundary points.

[0162] When a pixel is selected as a query pixel, at step 604, the boundary detection engine 530 determines whether the query pixel is valid. As described above, the occupancy map includes values. In some embodiments, these values ​​are binary, such as zero or one. For example, when the value of the pixel at position (u, v) in the occupancy map is a first value (such as zero), it indicates that the corresponding pixel at the same position in the geometry frame and the attribute frame does not correspond to a point in the 3D point cloud (and is considered invalid). When the value of the pixel at position (u, v) in the occupancy map is a second value (such as one), it indicates that the corresponding pixel at the same position in the geometry frame and the attribute frame represents a point in the 3D point cloud (and is considered valid).

[0163] When it is determined that the query pixel is zero (indicating an invalid pixel), method 600 ends because there is no point in the 3D point cloud corresponding to a pixel with a zero value at position (u, v) in the occupancy map. Note that when it is determined that the query pixel is zero, the boundary detection engine 530 can return to step 602 and select a new pixel in the occupancy map as the query pixel.

[0164] When it is determined that the query pixel is one (indicating a valid pixel), at step 606, the boundary detection engine 530 checks the neighboring pixels around the query pixel. The neighboring points are pixels in the first layer that are adjacent to and surround the query pixel. For example, if the query pixel is not at the edge of the occupied image frame, the query pixel will have eight neighboring pixels. For example, if the query pixel is at (y, x), the pixels in the first layer are defined as the pixels at (y, x-1), (y, x+1), (y-1, x), (y+1, x), (y-1, x-1), (y-1, x+1), (y+1, x-1), and (y+1, x+1) relative to the query pixel.

[0165] At step 608, the boundary detection engine 530 determines whether any of the pixels within the first layer surrounding the query pixel are invalid (have values ​​representing pixels that do not correspond to the 3D point cloud). If any of the pixels within the first layer are invalid, this indicates that the query pixel is located at the boundary of the tile, and therefore the corresponding point of the 3D point cloud will be identified as a boundary point. Since the query pixel was previously determined to be valid, but the neighboring pixels are considered invalid pixels, the query pixel is identified as being located at the boundary of the tile. Therefore, since the query pixel is adjacent to the invalid pixels, it is identified as being located at the boundary of the tile.

[0166] At step 614, if at least one of the pixels within the first layer surrounding the query point is invalid (has a value representing a pixel that does not correspond to a 3D point cloud), the pixel in the attribute frame and the corresponding pixel in the geometry frame at the same position as the query pixel in the occupancy map are indicated to correspond to a boundary point of the 3D point cloud. That is, if any one of the pixels within the first layer is invalid, the point of the 3D point cloud represented by the pixels in the attribute frame and the geometry frame at the same position as the query pixel is identified as a boundary pixel.

[0167] If all pixels in the first layer surrounding the query pixel are valid (each pixel has a value representing a valid pixel in the geometry frame and the attribute frame), then the boundary detection engine 530 checks the pixels of the second layer at step 610. The pixels of the second layer are pixels adjacent to and surrounding the pixels of the first layer. For example, if the query pixel is located at a distance of at least two pixels from the edge of the occupancy frame, there are sixteen pixels in the second layer of the query pixel. For example, if the query pixel is at (y,x), the pixels in the second layer are defined as the pixels at (y-2,x-2), (y-2,x-1), (y-2,x), (y-2,x+1), (y-2,x+2), (y-1,x-2), (y-1,x+2), (y,x-2), (y,x+2), (y+1,x-2), (y+1,x+2), (y+2,x-2), (y+1,x-1), (y+1,x-1), (y+1,x), (y+1,x+1), and (y+1,x+2) relative to the query pixel.

[0168] At step 612, the boundary detection engine 530 determines whether any of the pixels in the second layer surrounding the query pixel are invalid (have a value representing a pixel that does not correspond to a 3D point cloud). If any of the pixels in the second layer are invalid, this indicates that the query pixel is located at the boundary of the tile, and therefore the corresponding point of the 3D point cloud will be identified as a boundary point. That is, when one or more pixels in the second layer are invalid, this indicates that the query pixel is within a proximity threshold to the boundary of any of the tiles. In this case, the proximity threshold is two pixels from the edge of the tile.

[0169] At step 614, if at least one of the pixels within the second layer surrounding the query point is invalid (has a value representing a pixel that does not correspond to a 3D point cloud), the pixel in the attribute frame and the pixel in the geometry frame at the same position as the query pixel in the occupancy map are indicated to correspond to a boundary point of the 3D point cloud. That is, if any one of the pixels within the second layer is invalid, the point of the 3D point cloud represented by the pixels in the attribute frame and the geometry frame at the same position as the query pixel is identified as a boundary pixel.

[0170] If all pixels in the first and second layers surrounding the query pixel are valid (each pixel has a value representing a valid pixel in the geometry frame and the attribute frame), the encoder may select another pixel in the occupancy map frame as the query pixel. When another pixel in the occupancy map frame is selected as the query pixel, method 600 restarts at step 604.

[0171] In some embodiments, method 600 can be expanded to include examining pixels beyond the first and second layers. For example, method 600 can be expanded so that boundary detection engine 530 examines pixels in layers 3 through n, where each new layer is adjacent to and surrounds pixels in the layer below it. The more pixels that are examined, the more points of the 3D point cloud are identified as boundary points.

[0172] Figure 6B The method 620 is similar to Figure 6A Method 600 is described in detail in conjunction with method 620, because both methods 600 and 620 describe the overall process of determining whether a single point of a 3D point cloud represented on a 2D frame (such as a geometry frame, an attribute frame, and an occupancy map frame) is a boundary point. However, method 620 adds the ability to identify a point in the 3D point cloud as a boundary point when the pixel in the 2D frame corresponding to the point is close to the edge of the frame or when the pixel is close to another pixel belonging to another tile.

[0173] In step 622 (similar to Figure 6A In step 602), the boundary detection engine 530 selects a single pixel from the occupancy map frame as a query pixel. The pixel can be located anywhere within the occupancy map frame. Thereafter, in step 624 (similar to Figure 6A In step 604 of the embodiment, the boundary detection engine 530 determines whether the value of the query pixel indicates a valid pixel. For example, when the value of the pixel at the position (u, v) in the occupancy map is a first value (such as zero), it indicates that the corresponding pixel at the same position in the geometry frame and the attribute frame does not correspond to a point of the 3D point cloud. When the value of the pixel at the position (u, v) in the occupancy map is a second value (such as one), it indicates that the corresponding pixel at the same position in the geometry frame and the attribute frame represents information of a point of the 3D point cloud. For example, when the value of the pixel in the occupancy map is zero, it indicates that the pixel at the same position in the geometry frame and the attribute frame is invalid. When the value of the pixel in the occupancy map is one, it indicates that the pixel at the same position in the geometry frame and the attribute frame is valid and thus represents a specific point of the 3D point cloud.

[0174] When it is determined that the value of the query pixel is zero (indicating an invalid pixel), method 600 ends because there is no point in the 3D point cloud corresponding to the pixel at position (u, v) in the attribute frame and the geometry frame because the value of the pixel at that position in the occupancy map is zero. In some embodiments, when it is determined that the value of the query pixel is zero, the boundary detection engine 530 can return to step 622 and select a new pixel in the occupancy map as the query pixel.

[0175] At step 626, upon determining that the query pixel has a value of one (indicating a valid pixel), the boundary detection engine 530 identifies the tile to which the pixel belongs. Figure 4C and Figure 4DAs described in

[0045] , multiple tiles (such as tiles 432 and 442) are included in a single frame. Since each tile can be identified by a tile index number, the boundary detection engine 530 can identify which tile a pixel is within. For example, the query pixel can be within a tile identified by an index number between one and the total number of tiles (or if the tile starts at index number zero, between index number zero and the total number of tiles minus one).

[0176] At step 628, the boundary detection engine 530 determines whether any of the pixels in the first layer surrounding the query pixel are invalid (similar to Figure 6A The neighboring points in the first layer are within the pixels of the first layer that are adjacent to and surround the query pixel. For example, if the query pixel is not at the edge of the occupied frame, the query pixel will have eight neighboring pixels, such as pixels at (y, x-1), (y, x+1), (y-1, x), (y+1, x), (y-1, x-1), (y-1, x+1), (y+1, x-1), and (y+1, x+1).

[0177] The boundary detection engine 530 checks neighboring pixels around the query pixel at step 628 to identify whether the value of the query pixel in the occupancy map indicates a valid pixel or an invalid pixel and the tile to which each pixel in the first layer belongs.

[0178] If at least one of the pixels within the first layer surrounding the query point is invalid (has a value representing a pixel that does not correspond to a 3D point cloud), the boundary detection engine 530 identifies the pixel in the attribute frame and the corresponding pixel in the geometry frame at the same location as the query pixel in the occupancy map as a boundary point of the 3D point cloud, at step 636. That is, if any one of the pixels within the first layer is invalid, the point of the 3D point cloud represented by the pixel in the attribute frame and the geometry frame at the same location as the query pixel is identified as a boundary point.

[0179] Similarly, if at least one of the pixels in the first layer surrounding the query point belongs to a different tile than the query point, the boundary detection engine 530 identifies the pixel in the attribute frame and the corresponding pixel in the geometry frame at the same location as the query pixel in the occupancy map as a boundary point of the 3D point cloud at step 636. For example, the boundary detection engine 530 compares the tile index number of the query pixel with the tile index number of each of the neighboring pixels in the first layer. When any of the pixels in the first layer belongs to a different tile, the boundary detection engine 530 identifies the point of the 3D point cloud corresponding to the query pixel as a boundary point (based on the pixels in the geometry frame and the attribute frame that are at the same location as the query pixel in the occupancy map frame).

[0180] When all pixels in the first layer belong to the same tile and all pixels in the first layer are valid, the boundary detection engine 530 determines whether the query pixel is located at the edge of the occupancy frame at step 630. For example, the boundary detection engine 530 determines whether any of the pixels in the first layer do not exist. If the query pixel is located on the boundary of the occupancy frame, the pixel in the first layer will not exist.

[0181] When it is determined that the query pixel is along the boundary of the occupancy map frame, the boundary detection engine 530 identifies the pixel in the attribute frame and the corresponding pixel in the geometry frame at the same location as the query pixel in the occupancy map as a boundary point of the 3D point cloud, at step 636. That is, if none of the pixels within the first layer are present (such as when the query pixel is on the boundary of the occupancy map frame), then the point of the 3D point cloud represented by the pixels in the attribute frame and the geometry frame at the same location as the query pixel is identified as a boundary point.

[0182] When all pixels in the first layer are present, at step 632 the boundary detection engine 530 determines whether any of the pixels in the second layer surrounding the query pixel are invalid (similar to Figure 6A The second layer is a pixel that is adjacent to and surrounds the pixel of the first layer. For example, if the query pixel is at a distance of at least two pixels from the edge of the occupied frame, there are sixteen pixels in the second layer of the query pixel, such as pixels at (y-2, x-2), (y-2, x-1), (y-2, x), (y-2, x+1), (y-2, x+2), (y-1, x-2), (y-1, x+2), (y, x-2), (y, x+2), (y-1, x+2), (y, x-2), (y, x+2), (y+1, x+2), (y+2, x-2), (y+1, x+2), (y+2, x-2), (y+1, x-1), (y+1, x), (y+1, x+1), and (y+1, x+2).

[0183] The boundary detection engine 530 checks neighboring pixels at the second layer surrounding the query pixel to identify, at step 632 , whether the value of the query pixel in the occupancy map indicates a valid pixel or an invalid pixel and the block to which each pixel in the second layer belongs.

[0184] If at least one of the pixels within the second layer surrounding the query point is invalid (has a value representing a pixel that does not correspond to a 3D point cloud), the boundary detection engine 530 identifies the pixel in the attribute frame and the corresponding pixel in the geometry frame at the same location as the query pixel in the occupancy map as a boundary point of the 3D point cloud at step 636. That is, if any one of the pixels within the second layer is invalid, the point of the 3D point cloud represented by the pixels in the attribute frame and the geometry frame at the same location as the query pixel is identified as a boundary point.

[0185] Similarly, if at least one of the pixels in the second layer surrounding the query point belongs to a different tile than the query point, the boundary detection engine 530 identifies the pixel in the attribute frame and the corresponding pixel in the geometry frame at the same location as the query pixel in the occupancy map as a boundary point of the 3D point cloud at step 636. For example, the boundary detection engine 530 compares the tile index number of the query pixel with the tile index number of each of the neighboring pixels in the second layer. When any of the pixels in the second layer belongs to a different tile, the boundary detection engine 530 identifies the point of the 3D point cloud corresponding to the query pixel as a boundary point (based on the pixels in the geometry frame and the attribute frame at the same location as the query pixel in the occupancy map frame).

[0186] When all pixels in the second layer belong to the same tile as the query pixel and all pixels in the second layer are valid, the boundary detection engine 530 determines whether the query pixel is located 2 pixels from the edge of the occupied image frame at step 634. For example, the boundary detection engine 530 determines whether any of the pixels in the second layer do not exist. If the query pixel is located 2 pixels from the edge of the occupied image frame, the pixel in the second layer will not exist.

[0187] When it is determined that the query pixel is along the boundary of the occupancy map frame, the boundary detection engine 530 identifies the pixel in the attribute frame and the corresponding pixel in the geometry frame at the same location as the query pixel in the occupancy map as a boundary point of the 3D point cloud at step 636. That is, if any of the pixels within the second layer do not exist (such as when the query pixel is located at a distance of 2 pixels from the boundary of the occupancy map frame), then the point of the 3D point cloud represented by the pixels in the attribute frame and the geometry frame at the same location as the query pixel is identified as a boundary point.

[0188] If all pixels in the second layer are present, the boundary detection engine 530 determines that the query pixel is not located 2 pixels from the edge of the frame. When the query pixel is not located 2 pixels from the edge of the frame, the method 620 ends. In some embodiments, the boundary detection engine 530 may select another pixel in the occupancy map frame as the query pixel. When another pixel in the occupancy map frame is selected as the query pixel, the method 600 returns to step 624 and determines whether the new query pixel is valid. In some embodiments, the method 620 can be extended to include checking pixels beyond the first and second layers. For example, the method 620 can be extended so that the boundary detection engine 530 checks pixels in the third through nth layers, where each new layer is adjacent to and surrounds pixels in the layer below it. The more pixels that are checked, the more points of the 3D point cloud are identified as boundary points.

[0189] Figure 6C Method 650 describes an overall process for identifying which points of a 3D point cloud correspond to pixels that are within a proximity threshold from the boundary of any one of the tiles within a frame. That is, method 650 is used to identify boundary points within a point cloud based on the values ​​of the pixels within the frame that are occupied. Method 650 is limited to identifying boundary points of a 3D point cloud that correspond to pixels that are a distance of two pixels between a query pixel and a boundary of a tile. In some embodiments, method 650 can be extended to include identifying boundary points that correspond to pixels that are a distance greater than two pixels between a query pixel and a boundary of a tile.

[0190] At step 652, the boundary detection engine 530 sets the variable n to zero. The variable n serves as a count of pixels used to identify the occupancy map.

[0191] At step 654, the boundary detection engine 530 determines whether the occupancy value of pixel n in the occupancy map is zero. Based on the information in the geometry frame and the occupancy map frame, a zero value for a pixel in the occupancy map indicates that there is no corresponding point in the 3D point cloud, while a non-zero value (such as a 1 in the occupancy map) indicates that there is a corresponding point in the 3D point cloud. If the pixel value of pixel n is zero, then at step 662, the boundary detection engine 530 increases the value of n by 1 to correspond to the new pixel in the occupancy map.

[0192] When the value of the pixel is non-zero, then the boundary detection engine 530 identifies the values ​​of the pixels immediately adjacent to pixel n at step 656. The boundary detection engine 530 then determines whether any of the values ​​of the pixels immediately adjacent to pixel n (eg, pixels within the first layer) are equal to zero.

[0193] If, at step 656, the boundary detection engine 530 determines that one or more values ​​of the pixels immediately adjacent to pixel n (pixels within the first layer) are equal to zero, then, at step 660, the boundary detection engine 530 (based on the geometry frame) identifies the point of the 3D point cloud corresponding to pixel n as a boundary point. Alternatively, if, at step 656, the boundary detection engine 530 determines that none of the pixels immediately adjacent to pixel n (pixels within the first layer) have a value equal to zero, then, at step 658, the boundary detection engine 530 identifies the value of the pixel (second layer) two pixels away from pixel n. The boundary detection engine 530 then determines whether any of the values ​​of the pixels (e.g., pixels within the second layer) two pixels away from the neighboring pixel n are equal to zero.

[0194] If, at step 658, the boundary detection engine 530 determines that one or more values ​​of the pixels (pixels within the second layer) that are two pixels away from pixel n are equal to zero, then the boundary detection engine 530 (based on the geometry frame) identifies the point of the 3D point cloud corresponding to pixel n as a boundary point at step 660. Alternatively, if, at step 658, the boundary detection engine 530 determines that none of the pixels (pixels within the second layer) that are two pixels away from pixel n have a value equal to zero, then, at step 662, the boundary detection engine 530 increases the value of n by one to correspond to the new pixel within the occupancy map.

[0195] Additionally, after the boundary detection engine 530 (based on the geometry frame) identifies the point of the 3D point cloud corresponding to pixel n as a boundary point, at step 660 , the boundary detection engine 530 increases the value of n by 1 to correspond to the new pixel within the occupancy map.

[0196] At step 664, the boundary detection engine 530 compares the value of n to the total number of pixels in the occupancy map frame. When the value of n is less than the total number of pixels in the occupancy map frame, the boundary detection engine 530 returns to step 654 and determines whether the new pixel n (based on the incremented value of n from step 662) is equal to zero. When the value of n is equal to or greater than the total number of pixels, the method 650 ends because the boundary detection engine 530 has identified all boundary points of the 3D point cloud.

[0197] In some embodiments, the method 650 may be extended to include pixels that are more than 2 pixels away from the pixel n. For example, the boundary detection engine 530 may identify the values ​​of pixels that are three or more pixels away from the pixel n and then determine whether any of the values ​​of those pixels (the pixels that are three or more pixels away from the pixel n) are equal to zero.

[0198] In some embodiments, instead of n corresponding to pixels in the occupancy map as described above, n can correspond to points in a 3D point cloud. When n corresponds to points in a 3D point cloud, method 650 ends when all points have been checked. Since the occupancy map frame includes both valid and invalid pixels, only the points in the point cloud are checked instead of every pixel in the occupancy map frame, which requires less processing.

[0199] although Figure 6A 、 Figure 6B and Figure 6C An example of identifying boundary points is shown, but Figure 6A 、 Figure 6B and Figure 6C For example, although shown as a series of steps, Figure 6A 、 Figure 6B and Figure 6C The steps in can overlap, occur in parallel, or occur any number of times. In another example, the boundary detection engine 530 can use other methods to identify boundary points.

[0200] Figures 7A-7M A diagram is shown for identifying boundary points that are within a proximity threshold from the boundary of any of the tiles. Figure 7A An example sliding window for identifying boundary points according to an embodiment of the present disclosure is shown. Figure 7B An example portion of an occupancy map including tiles is shown according to an embodiment of the present disclosure. Figure 7C-7L An example process of identifying boundary points by moving a sliding window in an occupancy map according to an embodiment of the present disclosure is shown. Figure 7M An example occupancy map frame is shown indicating which pixels correspond to boundary points of a 3D point cloud. The process of identifying boundary points may be performed by the server 104, Figure 1 Any client device 106-116, Figure 2 Server 200, Figure 3 electronic device 300, Figure 5A and 5B Encoder 510, Figure 5A and 5C For ease of explanation, as Figures 7A-7M The process of identifying boundary points shown is described by Figure 5A and Figure 5B The boundary detection engine 530 of the encoder 510 and decoder 550 is executed.

[0201] Figure 7AAn example sliding window 700 is shown. Sliding window 700 includes a query pixel 702 at its center. Sliding window 700 also includes pixels of a first layer adjacent to and surrounding query pixel 702. The pixels of the first layer include pixel 704a, pixel 704b, pixel 704c, pixel 704d, pixel 704e, pixel 704f, pixel 704g, and pixel 704h (collectively, pixels 704). The pixels of the first layer include the pixels closest to query pixel 702.

[0202] Sliding window 700 also includes pixels in a second layer that are adjacent to and surround pixel 704 of the first layer. The pixels in the second layer are two pixels away from query pixel 702. The pixels in the second layer include pixels 706a, 706b, 706c, 706d, 706e, 706f, 706g, 706h, 706i, 706j, 706k, 706l, 706m, 706n, 706o, and 706p (collectively referred to as pixels 706 of the second layer). The pixels in the second layer include pixels within a distance of two pixels from query pixel 702.

[0203] In some embodiments, additional layers may be included around the query pixel. The sliding window 700 can identify pixels that are a certain distance from the query pixel. The distance may be referred to as a proximity threshold because the distance may include pixels from two or more layers.

[0204] Figure 7B A portion of an occupancy map frame 710 is shown. The portion of the occupancy map frame 710 is similar to Figure 5B 712. Pixel 712 includes both valid pixels and invalid pixels based on the value associated with each individual pixel. Figure 7B As shown, invalid pixels (pixels having a value of zero) are represented by no shading, while valid pixels (pixels having a non-zero value, such as 0) are represented by shading. The portion of the occupied frame 710 includes tile 714. Tile 714 includes twenty-five valid pixels.

[0205] Figure 5B and Figure 5C The boundary detection engine 530 performs boundary detection for the tiles in the portion of the occupancy map frame 710. For example, the boundary detection engine 530 identifies a boundary between valid pixels and invalid pixels. In another example, the boundary detection engine 530 identifies a boundary between invalid pixels and valid pixels that are within a threshold distance from the query pixel. The boundary detection engine 530 performs a single pass over the occupancy map frame to identify pixels that are within a proximity threshold to a boundary of any one of the tiles.

[0206] Figure 7C Sliding window 700 is shown at one location of occupancy map frame 710. Boundary detection engine 530 determines whether query pixel 702 of sliding window 700 is located on a valid pixel of occupancy map frame 710 or on an invalid pixel of occupancy map frame 710. Upon determining that query pixel 702 corresponds to an invalid pixel in occupancy map frame 710, boundary detection engine 530 identifies the pixel (a pixel of occupancy map frame 710) as corresponding to a non-boundary point as shown in identified boundary point 720a.

[0207] Figure 7D The sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7C 7. In comparison, sliding window 700 is shifted one pixel to the right. Boundary detection engine 530 determines whether query pixel 702 of sliding window 700 is located on a valid pixel of occupancy map frame 710 or on an invalid pixel of occupancy map frame 710. Upon determining that query pixel 702 corresponds to an invalid pixel in occupancy map frame 710, boundary detection engine 530 identifies the pixel (a pixel of occupancy map frame 710) as corresponding to a non-boundary point as shown in identified boundary point 720b.

[0208] Figure 7E The sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7D 7. In comparison, sliding window 700 is shifted one pixel to the right. Boundary detection engine 530 determines whether query pixel 702 of sliding window 700 is located on a valid pixel of occupancy map frame 710 or on an invalid pixel of occupancy map frame 710. Upon determining that query pixel 702 corresponds to an invalid pixel in occupancy map frame 710, boundary detection engine 530 identifies the pixel (a pixel of occupancy map frame 710) as corresponding to a non-boundary point as shown in identified boundary point 720c.

[0209] Figure 7F The sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7E 7. In comparison, sliding window 700 is shifted one pixel to the right. Boundary detection engine 530 determines whether query pixel 702 of sliding window 700 is located on a valid pixel of occupancy map frame 710 or on an invalid pixel of occupancy map frame 710. Upon determining that query pixel 702 corresponds to an invalid pixel in occupancy map frame 710, boundary detection engine 530 identifies the pixel (a pixel of occupancy map frame 710) as corresponding to a non-boundary point as shown in identified boundary point 720d.

[0210] Figure 7G The sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7FCompared to the first layer, the sliding window 700 is shifted one pixel to the right. The boundary detection engine 530 determines whether the query pixel 702 of the sliding window 700 is located on a valid pixel of the occupancy map frame 710 or on an invalid pixel of the occupancy map frame 710. Upon determining that the query pixel 702 corresponds to a valid pixel in the occupancy map frame 710, the boundary detection engine 530 determines whether any of the pixels within the pixels 704 of the first layer are invalid (corresponding to pixels having a value of zero in the occupancy map frame 710). Figure 7G As shown, five pixels within the first layer of pixels 704 of the sliding window 700 are located on invalid pixels, while three pixels are located on valid pixels. Since at least one pixel within the first layer of pixels 704 of the sliding window 700 is located on an invalid pixel, the boundary detection engine 530 identifies the pixel (the pixel of the occupancy map frame 710 indicated by the query pixel 702) as corresponding to a boundary point as shown in identified boundary point 720e.

[0211] Figure 7H The sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7G Compared to the first layer, the sliding window 700 is shifted one pixel to the right. The boundary detection engine 530 determines whether the query pixel 702 of the sliding window 700 is located on a valid pixel of the occupancy map frame 710 or on an invalid pixel of the occupancy map frame 710. Upon determining that the query pixel 702 corresponds to a valid pixel in the occupancy map frame 710, the boundary detection engine 530 determines whether any of the pixels within the pixels 704 of the first layer are invalid (corresponding to pixels having a value of zero in the occupancy map frame 710). Figure 7H As shown, five pixels within the first layer of pixels 704 of the sliding window 700 are located on valid pixels, while three pixels are located on invalid pixels. Since at least one pixel within the first pixel layer 704 of the sliding window 700 is located on an invalid pixel, the boundary detection engine 530 identifies the pixel (the pixel of the occupancy map frame 710 indicated by the query pixel 702) as corresponding to a boundary point as shown in identified boundary point 720f.

[0212] Figure 7I The sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7H Compared to the first layer, the sliding window 700 is shifted one pixel to the right. The boundary detection engine 530 determines whether the query pixel 702 of the sliding window 700 is located on a valid pixel of the occupancy map frame 710 or on an invalid pixel of the occupancy map frame 710. Upon determining that the query pixel 702 corresponds to a valid pixel in the occupancy map frame 710, the boundary detection engine 530 determines whether any of the pixels within the pixels 704 of the first layer are invalid (corresponding to pixels having a value of zero in the occupancy map frame 710). Figure 7IAs shown, five pixels within the first layer of pixels 704 of sliding window 700 are located on valid pixels, while three pixels are located on invalid pixels. Since at least one pixel within the first layer of pixels 704 of sliding window 700 is located on an invalid pixel, boundary detection engine 530 identifies the pixel (the pixel of occupancy map frame 710 indicated by query pixel 702) as corresponding to a boundary point as shown in identified boundary point 720g.

[0213] Figure 7J The sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7I Compared to the first layer, the sliding window 700 is shifted one pixel to the right. The boundary detection engine 530 determines whether the query pixel 702 of the sliding window 700 is located on a valid pixel of the occupancy map frame 710 or on an invalid pixel of the occupancy map frame 710. Upon determining that the query pixel 702 corresponds to a valid pixel in the occupancy map frame 710, the boundary detection engine 530 determines whether any of the pixels within the pixels 704 of the first layer are invalid (corresponding to pixels having a value of zero in the occupancy map frame 710). Figure 7J As shown, five pixels within the first layer of pixels 704 of the sliding window 700 are located on valid pixels, while three pixels are located on invalid pixels. Since at least one pixel within the first layer of pixels 704 of the sliding window 700 is located on an invalid pixel, the boundary detection engine 530 identifies the pixel (the pixel of the occupancy map frame 710 indicated by the query pixel 702) as corresponding to a boundary point as shown in identified boundary point 720h.

[0214] Figure 7K The sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7J Compared to the first layer, the sliding window 700 is shifted one pixel to the right. The boundary detection engine 530 determines whether the query pixel 702 of the sliding window 700 is located on a valid pixel of the occupancy map frame 710 or an invalid pixel of the occupancy map frame 710. Upon determining that the query pixel 702 corresponds to a valid pixel in the occupancy map frame 710, the boundary detection engine 530 determines whether any of the pixels within the pixels 704 of the first layer are invalid (corresponding to pixels having a value of zero in the occupancy map frame 710). Figure 7K As shown, five pixels within the first layer of pixels 704 of the sliding window 700 are located on invalid pixels, while three pixels are located on valid pixels. Since at least one pixel within the first layer of pixels 704 of the sliding window 700 is located on an invalid pixel, the boundary detection engine 530 identifies the pixel (the pixel of the occupancy map frame 710 indicated by the query pixel 702) as corresponding to a boundary point as shown in the identified boundary point 720i.

[0215] Figure 7LThe sliding window 700 is shown at the new position of the occupancy map frame 710. That is, Figure 7K 7. In comparison, sliding window 700 is shifted one pixel to the right. Boundary detection engine 530 determines whether query pixel 702 of sliding window 700 is located on a valid pixel of occupancy map frame 710 or on an invalid pixel of occupancy map frame 710. Upon determining that query pixel 702 corresponds to an invalid pixel in occupancy map frame 710, boundary detection engine 530 identifies the pixel (a pixel of occupancy map frame 710) as corresponding to a non-boundary point as shown in identified boundary point 720j.

[0216] Note that if the boundary detection engine 530 determines that all pixels within the pixels 704 of the first layer of the sliding window 700 are located on valid pixels, the boundary detection engine 530 will check the pixels of the second layer of pixels 706. If any of the pixels in the second layer of pixels 706 are located on an invalid pixel, the boundary detection engine 530 will determine that the pixel (the pixel of the occupancy map frame 710 indicated by the query pixel 702) corresponds to a boundary point. Alternatively, if all pixels in the second layer of pixels 706 are located on valid pixels, the boundary detection engine 530 will identify the pixel (the pixel of the occupancy map frame 710 indicated by the query pixel 702) as corresponding to a non-boundary point.

[0217] After determining that the pixel (the pixel indicated by query pixel 702 of the occupancy map frame 710) corresponds to a boundary point, or after determining that the pixel (the pixel indicated by query pixel 702 of the occupancy map frame 710) corresponds to a non-boundary point, the sliding window moves to another pixel in the occupancy map. In some embodiments, the sliding window 700 moves horizontally from one pixel to another in a row, and when it reaches the end of the row, it then moves vertically one pixel to a new row and continues to move horizontally along the new row until every pixel has been examined. In some embodiments, the sliding window 700 moves across the pixels in the occupancy map frame in a predefined pattern.

[0218] Figure 7M Boundary point 720k is shown. Figure 7M A mapping is shown indicating which pixels of the occupancy map frame 710 correspond to boundary points of the 3D point cloud based on tiles 714 as identified by the sliding window 700 and the boundary detection engine 530 .

[0219] although Figures 7A-7M An example of identifying boundary points is shown, but Figures 7A-7M Various changes may be made. For example, the sliding window may include more or fewer layers. For another example, any number of pixels may be included in the portion of the occupancy map frame 710, and any one of the pixels may be a valid point or an invalid point on the frame.

[0220] Figure 8 An example method 800 for encoding a point cloud according to an embodiment of the present disclosure is shown. The method 800 may be performed by Figure 1 any of the server 104 or client devices 106-116, Figure 2 Server 200, Figure 3 electronic device 300, Figure 5A and 5B For ease of explanation, the method 800 is described as being performed by the encoder 510 or any other suitable device or system. Figure 5A and 5B The encoder 510 is executed.

[0221] At step 802, the encoder 510 generates a geometry frame for the 3D point cloud. The geometry frame includes pixels. A portion of the pixels included in the geometry frame represents the geometric positions of points of the 3D point cloud, organized into tiles corresponding to respective point clusters of the 3D point cloud. Additionally, the encoder 510 generates an attribute frame. The attribute frame includes pixels, and a portion of the pixels in the attribute frame represent attribute information of the points of the 3D point cloud. The positions of the pixels in the second frame correspond to the respective positions of the pixels in the first frame.

[0222] At step 804, the encoder 510 generates an occupancy map frame. The occupancy map frame includes pixels that identify portions of pixels in the geometry frame that represent geometric locations of points of the 3D point cloud. The pixels of the occupancy map frame also identify portions of pixels in the attribute frame that represent attribute information of the points of the 3D point cloud.

[0223] At step 806, the encoder 510 encodes the geometry frame to generate a geometry substream, encodes the attribute frame to generate an attribute substream, and encodes the occupancy map frame to generate an occupancy map substream. In response to decoding the geometry substream, the attribute substream, and the occupancy map substream, the encoder 510 reconstructs a 3D point cloud using the geometry frame and the occupancy map frame (step 808).

[0224] At step 810, the encoder 510 performs a single pass of the occupancy map frame to identify points of the reconstructed 3D point cloud corresponding to a subset of pixels in the geometry frame and the attribute frame that are within a proximity threshold from a boundary of any of the tiles. The encoder 510 identifies the points of the reconstructed 3D point cloud corresponding to the subset of pixels based on the values ​​of the pixels within the occupancy map frame.

[0225] For example, the encoder 510 selects a query pixel in the occupancy frame, and upon determining that the query pixel corresponds to a valid pixel in the geometry frame, the encoder 510 checks the neighboring pixels of the query pixel. The encoder 510 first checks the pixels in the first layer surrounding the query pixel. For the first layer, the encoder 510 identifies that (i) any of the pixels in the first layer is invalid, (ii) any of the pixels in the first layer belongs to a different tile than the query pixel, or (iii) any of the pixels in the first layer is at the edge of the frame, and then identifies the point in the point cloud corresponding to the query pixel as a boundary point. If (i) all pixels in the first layer are valid, (ii) all pixels in the first layer belong to the same tile as the query pixel, or (iii) none of the pixels in the first layer are at the edge of the frame, the encoder 510 checks the pixels in the second layer surrounding the query pixel.

[0226] For the second layer, the encoder 510 identifies that (i) any of the pixels in the second layer is invalid, (ii) any of the pixels in the second layer belongs to a different tile than the query pixel, or (iii) any of the pixels in the second layer is located at the edge of the frame, then the point in the point cloud corresponding to the query pixel is identified as a boundary point. If (i) all pixels in the second layer are valid, (ii) all pixels in the second layer belong to the same tile as the query pixel, or (iii) no pixel in the second layer is located at the edge of the frame, the encoder 510 selects a new query pixel.

[0227] In some embodiments, the decoder 550 may then perform smoothing on the identified boundary points and generate smoothing parameters. The decoder 550 may apply the smoothing parameters to improve the visual quality of the reconstructed point cloud by removing any possible artifacts.

[0228] After the frames representing the 3D point cloud are encoded and the boundary points are identified, the encoder 510 may multiplex the frames into a bitstream. At step 812, the encoder 510 sends the bitstream. The bitstream may ultimately be transmitted to a decoder, such as the decoder 550.

[0229] although Figure 8 One example of a method 800 for encoding a point cloud is shown, but may be used for Figure 8 For example, although shown as a series of steps, Figure 8 The steps in a process can overlap, occur in parallel, or occur any number of times.

[0230] Figure 9 An example method 900 for decoding a point cloud according to an embodiment of the present disclosure is shown. The method 900 may be performed by Figure 1any of the server 104 or client devices 106-116, Figure 2 Server 200, Figure 3 electronic device 300, Figure 5A and 5C For ease of explanation, the method 900 is described as being performed by the decoder 550 or any other suitable device or system. Figure 5A and 5C The decoder 550 performs

[0231] Method 900 begins with a decoder (such as decoder 550) receiving a compressed bitstream (step 902). The received bitstream may include an encoded point cloud that is mapped onto multiple 2-D frames, compressed, and then transmitted and ultimately received by decoder 550.

[0232] At step 904, the decoder 550 decodes a geometry frame and an attribute frame from the bitstream. The geometry frame includes pixels, and a portion of the pixels of the geometry frame represent the geometric positions of points in the 3D point cloud. The pixels of the geometry frame are organized into tiles corresponding to clusters of points in the 3D point cloud. The attribute frame includes pixels, and a portion of the pixels of the attribute frame represent attribute information of the points in the 3D point cloud, and the positions of the pixels in the attribute frame correspond to the corresponding positions of the pixels in the geometry frame.

[0233] At step 906, the decoder 550 decodes an occupancy map frame from the bitstream. The occupancy map frame includes pixels that identify portions of pixels in the geometry frame representing geometric locations of points of the 3D point cloud. The pixels of the occupancy map frame also identify portions of pixels in the attribute frame representing attribute information of the points of the 3D point cloud.

[0234] At step 908 , the decoder 550 reconstructs a 3D point cloud using the geometry frame, the attribute frame, and the occupancy map frame.

[0235] At step 910, the decoder 550 performs a single pass of the occupancy map frame to identify points of the reconstructed 3D point cloud corresponding to a subset of pixels in the geometry frame that are within a proximity threshold from a boundary of any of the tiles. The decoder 550 identifies the points of the reconstructed 3D point cloud corresponding to the subset of pixels based on the values ​​of the pixels within the occupancy map frame.

[0236] For example, the decoder 550 selects a query pixel in the occupancy frame, and upon determining that the query pixel corresponds to a valid pixel in the geometry frame, the decoder 550 checks the neighboring pixels of the query pixel. The decoder 550 first checks the pixels in the first layer surrounding the query pixel. For the first layer, the decoder 550 identifies that (i) any of the pixels in the first layer is invalid, (ii) any of the pixels in the first layer belongs to a different tile than the query pixel, or (iii) any of the pixels in the first layer is at the edge of the frame, and then identifies the point in the point cloud corresponding to the query pixel as a boundary point. If (i) all pixels in the first layer are valid, (ii) all pixels in the first layer belong to the same tile as the query pixel, or (iii) none of the pixels in the first layer are at the edge of the frame, the decoder 550 checks the pixels in the second layer surrounding the query pixel.

[0237] For the second layer, the decoder 550 identifies that (i) any of the pixels in the second layer is invalid, (ii) any of the pixels in the second layer belongs to a different tile than the query pixel, or (iii) any of the pixels in the second layer is located at the edge of the frame, then the point in the point cloud corresponding to the query pixel is identified as a boundary point. If (i) all pixels in the second layer are valid, (ii) all pixels in the second layer belong to the same tile as the query pixel, or (iii) any of the pixels in the second layer is not located at the edge of the frame, the decoder 550 selects a new query pixel.

[0238] The decoder 550 may then perform smoothing on the identified boundary points to improve the visual quality of the reconstructed point cloud by removing any possible artifacts. Thereafter, the decoder 550 may render the point cloud for viewing by a user.

[0239] although Figure 9 One example of a method 900 for decoding a point cloud is shown, but may be used for Figure 9 For example, although shown as a series of steps, Figure 9 The steps in the process can overlap, occur in parallel, or occur any number of times.

[0240] Although the accompanying drawings illustrate various examples of user equipment, various modifications may be made to the drawings. For example, the user equipment may include any number of each component in any suitable arrangement. Generally, the drawings do not limit the scope of the present disclosure to any particular configuration. Furthermore, although the accompanying drawings illustrate operating environments in which various user equipment features disclosed in this patent document may be used, these features may be used in any other suitable system.

[0241] Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. It is intended that the present disclosure encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A decoding device for point cloud decoding, the decoding device comprising: a communication interface configured to receive a bit stream; as well as a processor operatively coupled to the communication interface, wherein the processor is configured to: decoding a first frame comprising pixels from the bitstream, wherein a portion of the pixels of the first frame represent geometric positions of points of a three-dimensional 3D point cloud organized into tiles corresponding to respective point clusters of the 3D point cloud, decoding a second frame comprising pixels from the bitstream, wherein a portion of the pixels of the second frame represent attribute information of points of the 3D point cloud, and positions of the pixels in the second frame correspond to respective positions of the pixels in the first frame, decoding an occupancy map frame from the bitstream, wherein the occupancy map frame comprises pixels identifying the portion of pixels in the first frame representing geometric locations of points of the 3D point cloud, reconstructing the 3D point cloud using the first frame, the second frame, and the occupancy map frame, and performing a single pass of the occupancy map frame to identify points of a reconstructed 3D point cloud based on values ​​of pixels within the occupancy map frame, wherein the points of the reconstructed 3D point cloud correspond to a subset of pixels in the first frame and the second frame that are within a proximity threshold from a boundary of any of the tiles, In order to identify the points of the reconstructed 3D point cloud corresponding to the subset of pixels, the processor is configured to: Select the query pixel that occupies the image frame, determining whether the query pixel is valid or invalid based on the value of the query pixel, In response to determining that the query pixel is valid, identifying the query pixel as a first query pixel, The valid pixel indicates that a pixel in the first frame located at the same position as a valid pixel in the occupancy map frame represents a geometric structure position of a point in the 3D point cloud.

2. The decoding device according to claim 1, wherein: The processor is further configured to: identifying a geometry smoothing parameter or an attribute smoothing parameter from the bitstream; Incrementally selecting pixels that occupy the frames until all points of the 3D point cloud are identified as boundary points or non-boundary points, wherein boundary points correspond to the subset of pixels in the first frame and the second frame; and Smoothing is performed on the identified boundary points based on the geometry smoothing parameter or the attribute smoothing parameter.

3. The decoding device according to claim 1, wherein: The processor is further configured to: In response to determining that the query pixel is invalid, continuing to check the values ​​of pixels in the occupancy map frame until a first query pixel is identified, The invalid pixel indicates that a pixel in the first frame located at the same position as the invalid pixel in the occupancy map frame does not represent a geometric structure position of a point in the 3D point cloud.

4. The decoding device according to claim 3, wherein: In response to identifying a first query pixel, the processor is configured to: identifying pixels of a first layer adjacent to a first query pixel in the occupancy map frame; determining whether any of the pixels in the first layer are invalid based on the values ​​of the pixels in the first layer; as well as In response to determining that at least one of the pixels in the first layer is invalid, a point of the reconstructed 3D point cloud represented by a pixel in the first frame at the same location as the first query pixel in the occupancy map frame is identified as a boundary point.

5. The decoding device according to claim 4, wherein: In response to determining that all pixels in the first layer are valid, the processor is configured to: Identifying pixels of a second layer adjacent to pixels of the first layer in the occupancy map frame; determining whether any of the pixels in the second layer are invalid based on the values ​​of the pixels in the second layer; In response to determining that at least one of the pixels in the second layer is invalid, identifying a point of the reconstructed 3D point cloud represented by a pixel in the first frame at the same location as the first query pixel in the occupancy map frame as a boundary point; as well as In response to determining that all pixels in the second layer are valid or identifying a point of the reconstructed 3D point cloud as a boundary point, subsequent query pixels corresponding to boundary points of the reconstructed 3D point cloud are identified occupying the image frame.

6. The decoding device according to claim 3, wherein: In response to identifying the first query pixel, the processor is further configured to: Identifying a first tile in a first frame to which a geometry pixel belongs, wherein the geometry pixel is located at the same position as a first query pixel in an occupancy frame; identifying pixels of the first layer adjacent to the geometric structure pixel in the first frame; determining (i) whether any of the pixels in the first layer belong to the second tile and (ii) whether any of the pixels in the first layer are invalid, wherein the first tile and the second tile correspond to different point clusters of the 3D point cloud; In response to determining that at least one of the pixels in the first layer belongs to the second tile or at least one of the pixels in the first layer is invalid, identifying a point of the reconstructed 3D point cloud represented by the geometric structure pixel as a boundary point; In response to determining that the pixel in the first layer belongs to the first tile and all pixels in the first layer are valid, identifying a pixel in the second layer adjacent to the pixel in the first layer in the first frame; determining whether (i) any of the pixels in the second layer belongs to the second tile and (ii) any of the pixels in the second layer is invalid; and In response to determining that at least one of the pixels in the second layer belongs to the second tile or at least one of the pixels in the second layer is invalid, identifying a point of the reconstructed 3D point cloud represented by the geometry pixel as a boundary point.

7. The decoding device according to claim 3, wherein: In response to identifying the first query pixel, the processor is further configured to: determining whether the first query pixel is within a proximity threshold from an edge of the occupancy map frame, and In response to determining that the first query pixel is within the proximity threshold from an edge of the occupancy map frame, a point of the reconstructed 3D point cloud corresponding to the first query pixel is identified as a boundary point.

8. A method for point cloud decoding, the method comprising: Receive bit stream; decoding a first frame comprising pixels from the bitstream, wherein a portion of the pixels of the first frame represent geometric positions of points of a three-dimensional 3D point cloud organized into tiles corresponding to respective point clusters of the 3D point cloud; Decoding a second frame comprising pixels from the bitstream, wherein a portion of the pixels of the second frame represent attribute information of points of the 3D point cloud, and positions of the pixels in the second frame correspond to respective positions of the pixels in the first frame; decoding an occupancy map frame from the bitstream, wherein the occupancy map frame comprises pixels identifying the portion of pixels in the first frame that represent geometric locations of points of the 3D point cloud; reconstructing the 3D point cloud using the first frame, the second frame, and the occupancy map frame; and performing a single pass of the occupancy map frame to identify points of a reconstructed 3D point cloud based on values ​​of pixels within the occupancy map frame, wherein the points of the reconstructed 3D point cloud correspond to a subset of pixels in the first frame and the second frame that are within a proximity threshold from a boundary of any of the tiles, The step of identifying points of the reconstructed 3D point cloud corresponding to the pixel subset comprises: Select the query pixel that occupies the image frame, determining whether the query pixel is valid or invalid based on the value of the query pixel, In response to determining that the query pixel is valid, identifying the query pixel as a first query pixel, The valid pixel indicates that a pixel in the first frame located at the same position as a valid pixel in the occupancy map frame represents a geometric structure position of a point in the 3D point cloud.

9. The method of claim 8, wherein: The step of identifying points of the reconstructed 3D point cloud corresponding to the subset of pixels further comprises: In response to determining that the query pixel is invalid, continuing to check the values ​​of pixels in the occupancy map frame until a first query pixel is identified, The invalid pixel indicates that a pixel in the first frame located at the same position as the invalid pixel in the occupancy map frame does not represent a geometric structure position of a point in the 3D point cloud.

10. The method of claim 9, further comprising: In response to identifying a first query pixel, identifying pixels of a first layer adjacent to the first query pixel in the occupied image frame; determining whether any of the pixels in the first layer are invalid based on the values ​​of the pixels in the first layer; as well as In response to determining that at least one of the pixels in the first layer is invalid, a point of the reconstructed 3D point cloud represented by a pixel in the first frame at the same location as the first query pixel in the occupancy map frame is identified as a boundary point.

11. The method of claim 10, further comprising: In response to determining that all pixels in the first layer are valid, identifying pixels in the second layer that are adjacent to pixels in the first layer in the occupation map frame; determining whether any of the pixels in the second layer are invalid based on the values ​​of the pixels in the second layer; In response to determining that at least one of the pixels in the second layer is invalid, identifying a point of the reconstructed 3D point cloud represented by a pixel in the first frame at the same location as the first query pixel in the occupancy map frame as a boundary point; as well as In response to determining that all pixels in the second layer are valid, or identifying a point of the reconstructed 3D point cloud as a boundary point, subsequent query pixels corresponding to boundary points of the reconstructed 3D point cloud occupying the image frame are identified.

12. The method of claim 9, further comprising: In response to identifying the first query pixel, identifying a first tile in the first frame to which a geometry pixel belongs, wherein the geometry pixel is located at the same position as the first query pixel in the occupancy frame; identifying pixels of the first layer adjacent to the geometric structure pixel in the first frame; determining (i) whether any of the pixels in the first layer belong to the second tile and (ii) whether any of the pixels in the first layer are invalid, wherein the first tile and the second tile correspond to different point clusters of the 3D point cloud; In response to determining that at least one of the pixels in the first layer belongs to the second tile or at least one of the pixels in the first layer is invalid, identifying a point of the reconstructed 3D point cloud represented by the geometric structure pixel as a boundary point; In response to determining that all pixels in the first layer belong to the first tile and all pixels in the first layer are valid, identifying pixels of the second layer adjacent to the pixels of the first layer in the first frame; determining whether (i) any of the pixels in the second layer belongs to the second tile and (ii) any of the pixels in the second layer is invalid; and In response to determining that at least one of the pixels in the second layer belongs to the second tile or at least one of the pixels in the second layer is invalid, identifying a point of the reconstructed 3D point cloud represented by the geometry pixel as a boundary point.

13. The method of claim 9, further comprising: In response to identifying the first query pixel, determining whether the first query pixel is within a proximity threshold from an edge of the occupancy map frame, and In response to determining that the first query pixel is within the proximity threshold from an edge of the occupancy map frame, a point of the reconstructed 3D point cloud corresponding to the first query pixel is identified as a boundary point.

14. A coding device for point cloud coding, the coding device comprising: The processor is configured to: generating a first frame comprising pixels for a three-dimensional 3D point cloud, wherein a portion of the pixels of the first frame represent geometric positions of points of the 3D point cloud organized into tiles corresponding to point clusters of the 3D point cloud, generating a second frame including pixels, wherein a portion of the pixels of the second frame represent attribute information of points of the 3D point cloud, and positions of the pixels in the second frame correspond to respective positions of the pixels in the first frame, generating an occupancy map frame, wherein the occupancy map frame comprises pixels identifying the portion of pixels in the first frame representing geometric locations of points of the 3D point cloud, encoding the first frame to generate a geometry substream, encoding the second frame to generate an attribute substream, and encoding the occupancy map frame to generate an occupancy map substream; In response to decoding the geometry substream, the attribute substream, and the occupancy map substream, reconstructing the 3D point cloud using the first frame, the second frame, and the occupancy map frame, performing a single pass of the occupancy map frame to identify points of a reconstructed 3D point cloud based on values ​​of pixels within the occupancy map frame, wherein the points of the reconstructed 3D point cloud correspond to a subset of pixels in the first frame and the second frame that are within a proximity threshold from a boundary of any of the tiles, and combining the geometry substream, the attribute substream, and the occupancy map substream to generate a bitstream; and a communication interface operatively coupled to the processor, wherein the communication interface is configured to transmit the bit stream, In order to identify the points of the reconstructed 3D point cloud corresponding to the subset of pixels, the processor is configured to: Select the query pixel that occupies the image frame, determining whether the query pixel is valid or invalid based on the value of the query pixel, In response to determining that the query pixel is valid, identifying the query pixel as a first query pixel, The valid pixel indicates that a pixel in the first frame located at the same position as a valid pixel in the occupancy map frame represents a geometric structure position of a point in the 3D point cloud.

15. A method for point cloud encoding, the method comprising: generating a first frame comprising pixels for a three-dimensional 3D point cloud, wherein a portion of the pixels of the first frame represent geometric positions of points of the 3D point cloud organized into tiles corresponding to point clusters of the 3D point cloud; generating a second frame including pixels, wherein a portion of the pixels of the second frame represent attribute information of points of the 3D point cloud, and positions of the pixels in the second frame correspond to respective positions of the pixels in the first frame, generating an occupancy map frame, wherein the occupancy map frame comprises pixels identifying the portion of pixels in the first frame representing geometric locations of points of the 3D point cloud; encoding the first frame to generate a geometry substream, encoding the second frame to generate an attribute substream, and encoding the occupancy map frame to generate an occupancy map substream; In response to decoding the geometry substream, the attribute substream, and the occupancy map substream, reconstructing the 3D point cloud using the first frame, the second frame, and the occupancy map frame, performing a single pass of the occupancy map frame to identify points of a reconstructed 3D point cloud based on values ​​of pixels within the occupancy map frame, wherein the points of the reconstructed 3D point cloud correspond to a subset of pixels in the first frame and the second frame that are within a proximity threshold from a boundary of any of the tiles; combining the geometry substream, the attribute substream, and the occupancy map substream to generate a bitstream; and sending the bitstream, The step of identifying points of the reconstructed 3D point cloud corresponding to the pixel subset comprises: Select the query pixel that occupies the image frame, determining whether the query pixel is valid or invalid based on the value of the query pixel, In response to determining that the query pixel is valid, identifying the query pixel as a first query pixel, The valid pixel indicates that a pixel in the first frame located at the same position as a valid pixel in the occupancy map frame represents a geometric structure position of a point in the 3D point cloud.

Images