Point cloud encoding and decoding method, device, electronic device and storage medium

Abstract

Aspects of the present application provide point cloud encoding and decoding methods, devices, electronic devices, and storage media. The method includes determining one or more original points in a point cloud associated with a reconstructed position. The positions of the one or more original points are reconstructed to the reconstructed position based on the geometric structure quantization result. Then, based on the attribute information of the one or more original points, an attribute value of the reconstructed position is determined, and the texture of the point cloud is encoded by the reconstructed position having the determined attribute value.

Description

[0001] Reference Addition

[0002] This application claims priority to U.S. Patent Application No. 17 / 064,029, filed on October 6, 2020, entitled "METHOD AND APPARATUS FOR POINT CLOUD CODING," which claims priority to U.S. Provisional Patent Application No. 62 / 942,536, filed on December 2, 2019, entitled "HYBRID CODING ORDER FOR POINT CLOUD CODING." The entire contents of the two prior applications are incorporated herein by reference. Technical Field

[0003] The present application relates to point cloud encoding and decoding, and in particular to a point cloud encoding and decoding method, device, electronic device and storage medium. Background Art

[0004] The background description provided herein is intended to generally present the context of the present application. To the extent described in this background section, the work of the inventors currently named, as well as various aspects of the description that may not have the qualification of being prior art at the time of filing this application, are neither explicitly nor implicitly considered to be prior art of the present application.

[0005] Various technologies have been developed to capture the world and represent it in three-dimensional (3D) space, e.g., objects in the world, environments in the world, etc. 3D representations of the world can enable more immersive forms of interaction and communication. Point clouds can be used as 3D representations of the world. A point cloud is a set of points in 3D space, each with associated attributes, e.g., color, material attributes, texture information, intensity attributes, reflectivity attributes, motion-related attributes, morphology attributes, and various other attributes. Such point clouds may include a large amount of data, which may be costly and time-consuming to store and transmit.

[0006] In some point cloud encoding and decoding related technologies, a recoloring process is applied on the encoder side when quantizing the geometric structure of points in the point cloud and merging repeated positions. However, this recoloring process is relatively complex, making the point cloud encoding and decoding efficiency not high. Summary of the invention

[0007] Various aspects of the present application provide point cloud compression and decompression methods and devices. In some examples, a point cloud compression / decompression device includes a processing circuit. In some embodiments, a processor determines one or more original points in a point cloud associated with a reconstructed position. The positions of the one or more original points can be reconstructed to the reconstructed position based on the geometric structure quantization result. Then, the processor determines the attribute value of the reconstructed position based on the attribute information of the one or more original points, and encodes the texture of the point cloud through the reconstructed position with the determined attribute value.

[0008] In some embodiments, the apparatus includes a memory storing a data structure associating one or more original points with the reconstructed position. The data structure is accessed based on the reconstructed position to retrieve the one or more original points associated with the reconstructed position.

[0009] In some embodiments, the processing circuit performs octree partitioning to divide the space of the point cloud into voxels, and associates one or more original points located in a voxel with a reconstructed position representing the voxel. In one example, the association relationship is stored using an appropriate data result.

[0010] In some embodiments, the processing circuit calculates an average of the attribute values ​​of the plurality of original points as the determined attribute value of the reconstructed position in response to the plurality of original points being associated with the reconstructed position. In some examples, the processing circuit calculates a weighted average of the attribute values ​​of the plurality of original points as the determined attribute value of the reconstructed position. In one example, the processing circuit weights the attribute values ​​of the original points in the plurality of original points based on the inverse of the distance between the original points in the plurality of original points and the reconstructed position.

[0011] In some embodiments, the processing circuitry, in response to the multiple original points being associated with the reconstructed location, assigns a particular attribute value of the closest point among the multiple original points as the determined attribute value of the reconstructed location. In one example, the processing circuitry, in response to the multiple closest points among the multiple original points having the same shortest distance to the reconstructed location, selects a median attribute value among the attribute values ​​of the multiple closest points. In another example, the processing circuitry, in response to the multiple closest points among the multiple original points having the same shortest distance to the reconstructed location, calculates an average attribute value of the attribute values ​​of the multiple closest points.

[0012] Various aspects of the present application provide a point cloud encoding and decoding method, including: determining one or more original points in a point cloud that are associated with a reconstructed position, the position of the one or more original points being reconstructed to the reconstructed position based on a geometric structure quantization result; determining an attribute value of the reconstructed position based on attribute information of the one or more original points; and encoding the texture of the point cloud through the reconstructed position having the determined attribute value.

[0013] Aspects of the present application also provide a point cloud encoding and decoding device, including: an original point determination unit, used to determine one or more original points associated with a reconstructed position in a point cloud, the positions of the one or more original points being reconstructed to the reconstructed position based on a geometric structure quantization result; an attribute value determination unit, used to determine an attribute value of the reconstructed position based on attribute information of the one or more original points; and an encoding unit, used to encode the texture of the point cloud through the reconstructed position having the determined attribute value.

[0014] Aspects of the present application also provide an electronic device, including a memory for storing computer-readable instructions; a processor for reading the computer-readable instructions and executing any one or a combination of the point cloud encoding and decoding methods as instructed by the computer-readable instructions.

[0015] Various aspects of the present application also provide a non-volatile computer-readable storage medium storing instructions, which, when executed by a point cloud encoding / decoding computer, causes the computer to perform any one or a combination of the point cloud encoding / decoding methods.

[0016] Various aspects of the present application provide a recoloring technology that can be used in a point cloud encoding and decoding solution, which can improve the encoding and decoding speed of the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, nature and various advantages of the subject matter of the present application will become more apparent from the following detailed description and accompanying drawings, in which:

[0018] Figure 1 is a simplified block diagram schematic diagram of a communication system according to an embodiment;

[0019] Figure 2 is a simplified block diagram schematic diagram of a streaming system according to an embodiment;

[0020] Figure 3 shows a block diagram of an encoder for encoding a point cloud frame according to some embodiments;

[0021] Figure 4 A decoder block diagram for decoding a compressed code stream corresponding to a point cloud frame according to some embodiments is shown;

[0022] Figure 5 is a simplified block diagram of a video decoder according to an embodiment;

[0023] Figure 6 is a simplified block diagram of a video encoder according to an embodiment;

[0024] Figure 7shows a block diagram of an encoder for encoding a point cloud frame according to some embodiments;

[0025] Figure 8 A block diagram of a decoder for decoding a compressed code stream corresponding to a point cloud frame according to some embodiments is shown;

[0026] Fig. 9 A diagram showing cube partitioning based on octree partitioning technology according to some embodiments of the present application is shown.

[0027] Fig.10 Examples of octree partitioning and an octree structure corresponding to the octree partitioning according to some embodiments of the present application are shown.

[0028] Fig.11 A diagram illustrating a derivation technique according to some embodiments of the present application is shown.

[0029] Fig.12 A flow chart outlining an example of a method according to some embodiments is shown.

[0030] Fig.13 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION

[0031] Various aspects of the present application provide point cloud codec (PCC) technology. PCC can be performed according to various schemes, for example, a geometry-based scheme (referred to as G-PCC), a video codec-based scheme (referred to as V-PCC), and the like. According to some aspects of the present application, G-PCC directly encodes 3D geometric structures, and is a purely geometry-based method that does not share much with video codecs, while V-PCC is mainly based on video codecs. For example, V-PCC can map points of a 3D cloud to pixels of a 2D grid (image). The V-PCC scheme can use a general video codec for point cloud compression. The Moving Picture Experts Group (MPEG) is studying the G-PCC standard and the V-PCC standard that use the G-PCC scheme and the V-PCC scheme, respectively.

[0032] Various aspects of the present application provide a recoloring technology for PCC schemes such as the G-PCC scheme and the V-PCC scheme. The recoloring technology can improve the encoding and decoding speed of point clouds.

[0033] Point clouds can be used in a wide range of applications. For example, point clouds can be used for object detection and localization in autonomous vehicles; point clouds can be used for mapping in geographic information systems (GIS), and can be used in cultural heritage to visualize and archive cultural heritage objects and collections.

[0034] Hereinafter, a point cloud may generally refer to a set of points in 3D space, each of which may be defined by location information with associated attributes (e.g., color, material, texture information, intensity attributes, reflectivity attributes, motion-related attributes, modal attributes, and various other attributes). A point cloud may be used to reconstruct an object or scene as a combination of these points. These points may be collected using multiple cameras, depth sensors, or lidars in various settings, and may consist of thousands to billions of points in order to realistically represent the reconstructed scene. A patch may generally refer to a contiguous subset of a surface described by a point cloud. In one example, a patch includes points whose surface normal vectors deviate from each other by less than a threshold amount.

[0035] Compression techniques can reduce the amount of data required to represent a point cloud for faster transmission or reduced storage. Thus, techniques for lossy compression of point clouds for real-time communications and six-degree-of-freedom (6DoF) virtual reality are needed. Additionally, techniques for lossless point cloud compression in dynamic mapping contexts such as autonomous driving and cultural heritage applications are also being sought.

[0036] According to one aspect of the present application, the main idea behind V-PCC is to compress the geometry, occupancy and texture of the dynamic point cloud into three separate video sequences using existing video codecs. The additional metadata required to interpret the three video sequences is compressed separately. A small part of the entire bitstream is metadata, which can be efficiently encoded / decoded using software implementation. Most of the information is handled by the video codec.

[0037] Figure 1 A simplified block diagram of a communication system (100) according to an embodiment of the present application is shown. The communication system (100) includes a plurality of terminal devices that can communicate with each other via, for example, a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) interconnected via the network (150). Figure 1 In the example of , a first pair of terminal devices (110) and (120) can perform a one-way transmission of point cloud data. For example, the terminal device (110) can compress a point cloud (e.g., points representing a structure) collected by a sensor (105) connected to the terminal device (110). The compressed point cloud can be sent to another terminal device (120) via a network (150), for example, in the form of a code stream. The terminal device (120) can receive the compressed point cloud from the network (150), decompress the code stream to reconstruct the point cloud, and appropriately display the reconstructed point cloud. One-way data transmission may be common in media service applications, etc.

[0038] exist Figure 1In the example of, the terminal device (110) and the terminal device (120) may be illustrated as a server and a personal computer, but the principles of the present application may not be limited to this. Embodiments of the present application are applicable to laptop computers, tablet computers, smart phones, game consoles, media players and / or dedicated three-dimensional (3D) devices. The network (150) represents any number of networks capable of transmitting compressed point clouds between the terminal device (110) and the terminal device (120). The network (150) may include, for example, a wired and / or wireless communication network. The network (150) can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purposes of discussion of the present application, the architecture and topology of the network (150) may not be important for the operation of the present application unless otherwise explained below.

[0039] Figure 2 A simplified block diagram of a streaming system (200) according to an embodiment is illustrated. Figure 2 An example is an application of the subject matter disclosed in this application for point clouds. The subject matter disclosed in this application is also applicable to other applications implemented with point clouds, such as 3D telescopic applications, virtual reality applications, etc.

[0040] The streaming system (200) may include an acquisition subsystem (213). The acquisition subsystem (213) may include a point cloud source (201), such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component that generates an uncompressed point cloud in software, etc., to generate, for example, an uncompressed point cloud (202). In an example, the point cloud (202) includes points acquired by a 3D camera. The point cloud (202) is depicted as a thick line to emphasize that it has a higher data volume compared to a compressed point cloud (204) (a bitstream of the compressed point cloud). The compressed point cloud (204) may be generated by an electronic device (220), which includes an encoder (203) coupled to the point cloud source (201). The encoder (203) may include hardware, software, or a combination thereof to implement or implement various aspects of the subject matter disclosed in the present application as described in more detail below. The compressed point cloud (204) (or the compressed point cloud bitstream (204)) is depicted as a thin line to emphasize that it has a lower data volume compared to the point cloud stream (202), which can be stored on the streaming server (205) for future use. One or more streaming client subsystems, such as Figure 2The client subsystem (206) and the client subsystem (208) in the video processing apparatus can access the streaming server (205) to retrieve the copy (207) and the copy (209) of the compressed point cloud (204). The client subsystem (206) can include a decoder (210), such as in the electronic device (230). The decoder (210) decodes the incoming copy (207) of the compressed point cloud and produces an output stream (211) of a reconstructed point cloud that can be presented on a presentation device (212).

[0041] Please note that the electronic devices (220) and (230) may include other components (not shown). For example, the electronic device (220) may include a decoder (not shown), and the electronic device (230) may also include an encoder (not shown).

[0042] In some streaming systems, the compressed point cloud (204), the compressed point cloud (207), and the compressed point cloud (209) (e.g., a code stream of the compressed point cloud) can be compressed according to a specific standard. In some examples, a video codec standard is used for compression of the point cloud. Examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and the like.

[0043] Figure 3 A block diagram of a V-PCC encoder (300) for encoding a point cloud frame according to some embodiments is shown. In some embodiments, the V-PCC encoder (300) can be used in a communication system (100) and a streaming system (200). For example, the encoder (203) can be configured and operated in a manner similar to the V-PCC encoder (300).

[0044] The V-PCC encoder (300) receives a point cloud frame as an uncompressed input and generates a code stream corresponding to the compressed point cloud frame. In some embodiments, the V-PCC encoder (300) may receive the point cloud frame from a point cloud source such as the point cloud source (201).

[0045] exist Figure 3 In the example, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometric image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image filling modules (316) and (318), a group dilation module (320), video compression modules (322), (323) and (332), an auxiliary patch information compression module (338), an entropy compression module (334) and a multiplexer (324).

[0046] According to one aspect of the present application, a V-PCC encoder (300) converts a 3D point cloud frame into an image-based representation together with some metadata (e.g., occupancy map and patch information) used to convert a compressed point cloud back into a decompressed point cloud. In some examples, the V-PCC encoder (300) can convert a 3D point cloud frame into a geometric image, a texture image, and an occupancy map, and then encode the geometric image, texture image, and occupancy map into a bitstream using video coding techniques. Typically, a geometric image is a 2D image whose pixels are filled with geometric values ​​associated with points projected to the pixels, and pixels filled with geometric values ​​can be referred to as geometric samples. A texture image is a 2D image whose pixels are filled with texture values ​​associated with points projected to the pixels, and pixels filled with texture values ​​can be referred to as texture samples. An occupancy map is a 2D image whose pixels are filled with values ​​indicating that the patch is occupied or not occupied by the patch.

[0047] The patch generation module (306) segments the point cloud into a set of patches (e.g., a patch is defined as a contiguous subset of the surface described by the point cloud), which may or may not overlap, such that each patch can be described by a depth field relative to a plane in 2D space. In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while also minimizing reconstruction errors.

[0048] The patch information module (304) may collect patch information indicating the size and shape of the patch. In some examples, the patch information may be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.

[0049] The patch packing module (308) is configured to map the extracted patches onto a 2-dimensional (2D) grid while minimizing unused space and ensuring that each M×M (e.g., 16×16) block of the grid is associated with a unique patch. Effective patch packing can directly affect compression efficiency by minimizing unused space or ensuring temporal consistency.

[0050] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) store the geometry and texture of the point cloud as images using the 3D to 2D mapping calculated during the packing process. To better handle the situation where multiple points are projected onto the same sample, each patch is projected onto two images, called layers. In one example, the geometry image is represented by a W×H monochrome frame in a YUV420-8 bit format. To generate the texture image, the texture generation step uses the reconstructed / smoothed geometry to calculate the colors to be associated with the resampled points.

[0051] The occupancy map module (314) can generate an occupancy map describing fill information at each cell. For example, the occupancy map includes a binary map that indicates, for each cell of the grid, whether the cell belongs to empty space or to a point cloud. In one example, the occupancy map uses binary information for each pixel describing whether the pixel is filled. In another example, the occupancy map uses binary information for each pixel block describing whether the pixel block is filled.

[0052] The occupancy map generated by the occupancy map module (314) can be compressed using lossless encoding or lossy encoding. When lossless encoding is used, the occupancy map is compressed using the entropy compression module (334). When lossy encoding is used, the occupancy map is compressed using the video compression module (332).

[0053] Note that the patch packing module (308) may leave some empty space between the 2D patches packed into the image frame. The image padding modules (316) and (318) may fill the empty space (referred to as padding) to generate an image frame that may be suitable for use with 2D video and image codecs. Image padding, also known as background padding, may fill unused space with redundant information. In some examples, good background padding may minimally increase the bit rate without introducing significant coding artifacts around patch boundaries.

[0054] The video compression modules (322), (323) and (332) can encode 2D images, such as filled geometry images, filled texture images and occupancy maps, based on appropriate video coding standards, such as HEVC, VVC, etc. In one example, the video compression modules (322), (323) and (332) are separate components that operate separately. Note that in another example, the video compression modules (322), (323) and (332) can be implemented as a single component.

[0055] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometric image. The smoothed image can be provided to the texture image generation (312). The texture image generation (312) can then adjust the generation of the texture image based on the reconstructed geometric image. For example, when the patch shape (e.g., the geometric structure) is slightly distorted during encoding and decoding, the distortion can be taken into account when generating the texture image to correct for the distortion of the patch shape.

[0056] In some embodiments, group dilation (320) is configured to fill pixels around object boundaries with redundant low-frequency content to improve codec gain and visual quality of the reconstructed point cloud.

[0057] The multiplexer (324) can multiplex the compressed geometry image, the compressed texture image, the compressed occupancy map, and the compressed auxiliary patch information into a compressed bitstream.

[0058] Figure 4 A block diagram of a V-PCC decoder (400) for decoding a compressed code stream corresponding to a point cloud frame according to some embodiments is shown. In some embodiments, the V-PCC decoder (400) can be used in a communication system (100) and a streaming system (200). For example, the decoder (210) can be configured to operate in a similar manner to the V-PCC decoder (400). The V-PCC decoder (400) receives a compressed code stream and generates a reconstructed point cloud based on the compressed code stream.

[0059] exist Figure 4 In the example of , the V-PCC decoder (400) includes a demultiplexer (432), video decompression modules (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).

[0060] The demultiplexer (432) may receive the compressed code stream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.

[0061] The video decompression modules (434) and (436) can decode the compressed image according to an appropriate standard (e.g., HEVC, VVC, etc.) and output the decompressed image. For example, the video decompression module (434) decodes the compressed texture image and outputs the decompressed texture image; the video decompression module (436) decodes the compressed geometric image and outputs the decompressed geometric image.

[0062] The occupancy map decompression module (438) can decode the compressed occupancy map according to an appropriate standard (eg, HEVC, VVC, etc.) and output a decompressed occupancy map.

[0063] The auxiliary patch-information decompression module (442) can decode the compressed auxiliary patch information according to an appropriate standard (eg, HEVC, VVC, etc.) and output decompressed auxiliary patch information.

[0064] The geometry reconstruction module (444) can receive the decompressed geometry image and generate a reconstructed point cloud geometry structure based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0065] The smoothing module (446) can smooth inconsistencies at patch edges. The smoothing process is intended to mitigate discontinuities that may appear at patch boundaries due to compression artifacts. In some embodiments, a smoothing filter can be applied to pixels located on patch boundaries to mitigate distortion that may be caused by compression / decompression.

[0066] The texture reconstruction module (448) can determine texture information of points in the point cloud based on the decompressed texture image and the smoothed geometry.

[0067] The color smoothing module (452) can smooth out inconsistencies in shading. Non-adjacent patches in 3D space are typically packed adjacent to each other in 2D video. In some examples, pixel values ​​from non-adjacent patches can be blended by a block-based video codec. The goal of color smoothing is to reduce visible artifacts that appear at patch boundaries.

[0068] Figure 5 A block diagram of a video decoder (510) according to an embodiment of the present application is shown. The video decoder (510) can be used in the V-PCC decoder (400). For example, the video decompression modules (434) and (436), and the occupancy map decompression module (438) can be configured in a similar manner to the video decoder (510).

[0069] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from compressed images such as an encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510). The parser (520) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may be performed according to a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (Group of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0070] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory, thereby creating symbols (521).

[0071] Depending on the type of coded video picture or coded video picture portion (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser (520) and the multiple units below is not described.

[0072] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0073] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as symbols (521) from the parser (520) and control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) can output a block including sample values, which can be input into an aggregator (555).

[0074] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding reconstructed information extracted from the current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a portion of the reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0075] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (521) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signals) by the aggregator (555) to generate output sample information. The acquisition of the predicted samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), for example, including X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0076] The output samples of the aggregator (555) may be used by various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques that are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0077] The output of the loop filter unit (556) may be a sample stream that may be output to a display device and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0078] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.

[0079] The video decoder (510) may perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata of the HRD buffer management signaled in the encoded video sequence.

[0080] Figure 6 A block diagram of a video encoder (603) according to an embodiment of the present application is shown. The video encoder (603) can be used to compress point clouds in the V-PCC encoder (300). In one example, the video compression modules (322) and (323) and the video compression module (332) are configured in a similar manner to the encoder (603).

[0081] The video encoder (603) may receive an image, such as a filled geometry image, a filled texture image, etc., and generate a compressed image.

[0082] According to an embodiment, the video encoder (603) may encode and compress pictures (images) of a source video sequence into an encoded video sequence (compressed images) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of brevity, coupling is not indicated in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, picture group (group of pictures, GOP) layout, maximum motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0083] In some embodiments, the video encoder (603) operates in an encoding loop. As a simplified description, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, e.g., a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered by the subject matter disclosed in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0084] The operation of the "local" decoder (633) can be combined with the above Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into the coded video sequence, the entropy decoding portion of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).

[0085] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operation. The description of encoder technology can be simplified because encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only needed in certain areas and is provided below.

[0086] During operation, in some examples, the source encoder (630) may perform motion compensated predictive coding to predictively encode an input picture with reference to one or more previously encoded pictures from a video sequence designated as “reference pictures.” In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

[0087] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 6 When the video encoder (603) is decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0088] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (634).

[0089] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0090] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0091] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned any of the following picture types:

[0092] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0093] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0094] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0095] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or the blocks may be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0096] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0097] The video may be in the form of multiple source pictures (images) in a temporal sequence. Intra-picture prediction (often shortened to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an embodiment, a particular picture being encoded / decoded is partitioned into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0098] In some embodiments, bidirectional prediction techniques may be used in inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be predicted by a combination of the first reference block and the second reference block.

[0099] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0100] According to some embodiments disclosed in the present application, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type for the CU, for example, an inter-prediction type or an intra-prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Taking the luminance prediction block as a prediction block as an example, the prediction block includes a matrix of pixel values ​​(e.g., luminance values), for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0101] Figure 7 A block diagram of a G-PPC encoder (700) according to an embodiment is shown. The encoder (700) may be configured to receive point cloud data and compress the point cloud data to generate a code stream carrying the compressed point cloud data. In an embodiment, the encoder (700) may include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic coding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing reconstructed attribute values.

[0102] like Figure 7As shown, an input point cloud (701) can be received at an encoder (700). The positions (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. The duplicate point removal module (712) is configured to receive the quantized positions and perform a filtering process to identify and remove duplicate points. The octree encoding module (730) is configured to receive the filtered positions from the duplicate point removal module (712) and perform an octree-based encoding process to generate an occupancy code sequence for describing a 3D grid of voxels. The occupancy code is provided to an arithmetic coding module (770).

[0103] The attribute transfer module (720) is configured to receive attributes of the input point cloud, and when multiple attribute values ​​are associated with each voxel, perform an attribute transfer process to determine the attribute value of each voxel. The attribute transfer process can be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output by the octree encoding module (730) and reorganize the points into different LODs. The LOD information is provided to the attribute prediction module (750).

[0104] The attribute prediction module (750) processes the points according to the LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction value for the current point based on the reconstructed attributes of a set of neighboring points of the current point stored in the memory (790). A prediction residual may then be obtained based on the original attribute value received from the attribute transfer module (720) and the locally generated attribute prediction value. When candidate indices are used in various attribute prediction processes, the index corresponding to the selected prediction candidate may be provided to the arithmetic coding module (770).

[0105] The residual quantization module (760) is configured to receive the prediction residual from the attribute prediction module (750) and perform quantization to generate a quantized residual. The quantized residual is provided to the arithmetic coding module (770).

[0106] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by an inverse operation of the quantization operation performed at the residual quantization module (760). The addition module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and receive the respective attribute prediction values ​​from the attribute prediction module (750). By combining the reconstructed prediction residual and the attribute prediction value, a reconstructed attribute value is generated and stored in the memory (790).

[0107] The arithmetic coding module (770) is configured to receive the occupancy code, the candidate index (if used), the quantized residual (if generated), and other information, and perform entropy coding to further compress the received values ​​or information. As a result, a compressed code stream (702) carrying compressed information can be generated. The code stream (702) can be transmitted or otherwise provided to a decoder that decodes the compressed code stream, or can be stored in a storage device.

[0108] Figure 8 A block diagram of a G-PCC decoder (800) according to an embodiment is shown. The decoder (800) may be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream to generate decoded point cloud data. In an embodiment, the decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.

[0109] like Figure 8 As shown, a compressed bitstream (801) can be received at an arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain a quantized residual (if generated) and an occupancy code of the point cloud. The octree decoding module (830) is configured to determine a reconstructed position of a point in the point cloud based on the occupancy code. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstructed position and determine an order based on the LOD. The inverse residual quantization module (820) is configured to generate a reconstructed residual based on the quantized residual received from the arithmetic decoding module (810).

[0110] The attribute prediction module (850) is configured to perform an attribute prediction process to determine an attribute prediction for a point according to an LOD-based order. For example, the attribute prediction for the current point may be determined based on the reconstructed attribute values ​​of neighboring points of the current point stored in the memory (860). The attribute prediction module (850) may combine the attribute prediction with each reconstructed residual to generate a reconstructed attribute for the current point.

[0111] In one example, the reconstructed attribute sequence generated from the attribute prediction module (850) together with the reconstructed positions generated from the octree decoding module (830) corresponds to the decoded point cloud (802) output from the decoder (800). In addition, the reconstructed attributes are also stored in the memory (860) and can be subsequently used to derive attribute predictions for subsequent points.

[0112] In various embodiments, the encoder (300), decoder (400), encoder (700) and / or decoder (800) may be implemented in hardware, software or a combination thereof. For example, the encoder (300), decoder (400), encoder (700) and / or decoder (800) may be implemented in processing circuits such as one or more integrated circuits (ICs), which operate with or without software such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. In another example, the encoder (300), decoder (400), encoder (700) and / or decoder (800) may be implemented as software or firmware, including instructions stored in a non-volatile (or non-volatile) computer-readable storage medium. These instructions, when executed by the processing circuits such as one or more processors, cause the processing circuits to perform the functions of the encoder (300), decoder (400), encoder (700) and / or decoder (800).

[0113] Note that the attribute prediction module (750) or (850) configured to implement the attribute prediction techniques disclosed herein may be included in other Figure 7 and Figure 8 In a decoder or encoder with a structure similar or different to that shown in . In addition, the encoder (700) and the decoder (800) may be included in the same device, or in separate devices in various examples.

[0114] According to some aspects of the present application, in some related examples (e.g., the current version of the TMC13 model), a recoloring process is applied on the encoder side when the geometry is quantized and repeated positions are merged. The recoloring process is relatively complex. For example, the recoloring process relies on a k-dimensional tree (e.g., a three-dimensional tree) data structure to search for one or more nearest neighbors. The search based on the k-dimensional tree may be complex and time consuming.

[0115] In a related example (e.g., a version of the TMC13 model), the input to the recoloring algorithm may include the original positions of the original points in the point cloud, the attributes associated with the original positions, and the reconstructed positions (e.g., the result of geometry quantization and repeated position merging). The recoloring algorithm may transfer the attributes associated with the original positions as the attributes associated with the reconstructed positions, for example, with minimized attribute distortion. The attributes associated with the reconstructed positions may form the reconstructed points in the reconstructed point cloud. It should be noted that the recoloring algorithm may transfer any suitable attributes, such as color, material properties, texture information, intensity attributes, reflectivity attributes, motion-related attributes, morphological attributes, and various other attributes. In an example, the recoloring algorithm includes five steps described in detail in the following specification.

[0116] In the first step, the original position of a point in the point cloud, the attributes associated with the original position, and the reconstructed position are received. i ) i＝0...N-1 represents the original position of the original point in the original point cloud, N represents the number of points in the original point cloud, Indicates that the position has been rebuilt, N rec Indicates the number of reconstructed positions. If there are duplicate points and they are merged, then N rec <N, otherwise N rec =N.

[0117] In the second step, for each position in the reconstructed point cloud Perform a search process (e.g., based on a k-dimensional tree search) to determine the nearest neighbors in the original point cloud Represents the attribute value associated with the nearest neighbor in the original point cloud.

[0118] In the third step, for each position in the reconstructed point cloud A search process (e.g., based on a k-dimensional tree search) is performed to determine a set of original positions in the original point cloud, represented as The group's original location is shared as their nearest neighbors in the reconstructed point cloud. H(i) represents The number of elements in express One of the elements. It should be noted that Can be empty or can have one or more elements.

[0119] In the fourth step, Check. When empty, it will be the same as the position Associated attribute values Set to

[0120] In the fifth step, if is not empty, then the position can be calculated based on, for example, (Equation 1) Associated attribute values

[0121]

[0122] In a related example, a search is used, such as a k-dimensional tree (e.g., a three-dimensional tree) search, to search for one or more nearest neighbors. Searches based on k-dimensional trees can be complex and time consuming. The present application provides a recoloring technique to reduce complexity and increase processing speed. For example, the search for the nearest neighbor can be determined based on information stored during geometry quantization and duplicate point merging.

[0123] The methods proposed in this application can be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder and decoder can be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-volatile computer-readable storage medium.

[0124] According to some aspects of the present application, geometric information and associated attributes of a point cloud, such as color, reflectivity, etc., may be compressed separately (for example, in a test model 13 (TMC13) model). In some embodiments, geometric information of a point cloud, including 3D coordinates of points in the point cloud, may be processed according to a quantization and repeated point merging process. In some examples, the quantization and repeated point merging process may be based on octree partitioning, and the geometric information may be encoded with occupancy information of the partitions through the octree partitioning. Attributes may be compressed based on the reconstructed geometry using, for example, prediction, lifting, and region-adaptive hierarchical transformation techniques. For example, the encoded geometric information may be dequantized to reconstruct geometric information of the point cloud, such as a reconstructed position. The attributes of the original point are transferred to the reconstructed position and then encoded.

[0125] According to some aspects of the present application, octree partitioning can be used to partition three-dimensional space. An octree is a three-dimensional simulation of a quadtree in a two-dimensional space. Octree partitioning technology refers to a partitioning technology that recursively subdivides a three-dimensional space into eight octants, and an octree structure refers to a tree structure representing these partitions. In one example, each node in the octree structure corresponds to a three-dimensional space, and the node can be an end node (no longer divided, also referred to as a leaf node in some examples) or a node with further partitioning. The partitioning at the node can divide the three-dimensional space represented by the node into eight octants. In some examples, the node corresponding to the partition of a particular node can be referred to as a child node of the particular node.

[0126] Fig. 9 FIG. 1 shows a partition diagram of a 3D cube (900) (corresponding to a node) based on an octree partitioning technique according to some embodiments of the present application. The partitioning can divide the 3D cube (900) into 8 smaller cubes 0-7 of equal size, such as Fig. 9 shown.

[0127] The octree partitioning technique (e.g., in TMC13) can recursively partition the original 3D space into smaller units, and can encode the occupancy information of each subspace to represent the geometric position.

[0128] In some embodiments (e.g., in TMC13), an octree geometry codec is used. The octree geometry codec can perform geometry encoding. In some examples, geometry encoding is performed on a cube box. For example, a cube box can be a box consisting of two points (0, 0, 0) and (2 M-1 , 2 M-1 , 2 M-1 ) defines an axis-aligned bounding box B, where 2 M-1 Defines the size of the bounding box B, M can be specified in the codestream.

[0129] Then, the octree structure is constructed by recursively subdividing the cube box. For example, the two points (0, 0, 0) and (2 M-1 , 2 M-1 , 2 M-1 ) is divided into 8 sub-cube boxes, generating an 8-bit code, called an occupancy code. Each bit in the occupancy code is associated with a sub-cube box, and the value of the bit is used to indicate whether the associated sub-cube box contains any point of the point cloud. For example, a bit value of 1 indicates that the sub-cube box associated with the bit contains one or more points of the point cloud; a bit value of 0 indicates that the sub-cube box associated with the bit does not contain any point of the point cloud.

[0130] In addition, for an empty sub-cube box (e.g., the value of the bit associated with the sub-cube box is 0), the sub-cube box is no longer segmented. When the sub-cube box has one or more points of the point cloud (e.g., the value of the bit associated with the sub-cube box is 1), the sub-cube box is further divided into 8 smaller sub-cube boxes, and an occupancy code can be generated for the sub-cube box to indicate the occupancy of the smaller sub-cube box. In some examples, the subdivision operation can be repeatedly performed on non-empty sub-cube boxes until the size of the sub-cube box is equal to a predetermined threshold, such as a size of 1. In some examples, a sub-cube box with a size of 1 is called a voxel, and a sub-cube box with a size greater than a voxel can be called a non-voxel.

[0131] Fig.10 An example of an octree partition (1010) and an octree structure (1020) corresponding to the octree partition (1010) according to some embodiments of the present application is shown. Fig.10 Two levels of partitioning in the octree partition (1010) are shown. The octree structure (1020) includes nodes (N0) corresponding to the cube boxes of the octree partition (1010). At the first level, according to Fig. 9The numbering technique shown divides the cube box into 8 sub-cube boxes numbered from 0 to 7. The occupancy code of the division of node N0 is binary "10000001", which means that the first sub-cube box represented by node N0-0 and the eighth sub-cube box represented by node N0-7 include points in the point cloud, and the other sub-cube boxes are empty.

[0132] Then, in the second level of partitioning, the first sub-cube box (represented by node N0-0) and the eighth sub-cube box (represented by node N0-7) are further subdivided into eight octants respectively. Fig. 9 The numbering technique shown divides the first sub-cube box (represented by node N0-0) into 8 smaller sub-cube boxes numbered 0 to 7. The occupancy code of the division of node N0-0 is binary "00011000", which indicates that the fourth smaller sub-cube box (represented by node N0-0-3) and the fifth smaller sub-cube box (represented by node N0-0-4) include points in the point cloud, and the other smaller sub-cube boxes are empty. At the second level, the seventh sub-cube box (represented by node N0-7) is divided into 8 smaller sub-cube boxes in a similar manner, as shown in FIG. Fig.10 shown.

[0133] exist Fig.10 In the example of , nodes corresponding to nodes of non-empty cubic spaces (eg, cubic boxes, sub-cube boxes, smaller sub-cube boxes, etc.) are colored gray and are referred to as shadow nodes.

[0134] In some embodiments, for a voxel of a unit-sized cube box, one or more points in the unit-sized cube box may be quantized to, for example, the center position of the unit-sized cube box. When there are multiple points in the unit-sized cube box, these points are merged into the center position of the unit-sized cube box. The center position of the voxel may form the reconstructed position of the reconstructed point in the reconstructed point cloud.

[0135] According to some aspects of the present application, a suitable data structure (such as a lookup table, a list, etc.) can be used to associate the original point in the voxel with the reconstructed position (e.g., the center position) of the voxel. For example, an entry in the lookup table can associate the index of the voxel with the index of the original point in the voxel. The data structure can be stored. Therefore, for each position in the reconstructed point cloud (which is the center point of the voxel), the original point in the voxel that is merged into the center position can be determined based on the data structure.

[0136] According to one aspect of the present application, the geometry of the point cloud is quantized by scalar quantization, which in some examples is the only source of geometric distortion. In some embodiments, when octree partitioning is used for geometry quantization, a voxel is a unit-sized cubic box, each voxel includes one or more original points of the point cloud, and the one or more original points are quantized to the center point of the voxel. Therefore, the one or more original points in the voxel share the same center point of the voxel (in the reconstructed point cloud) with the nearest neighbor, and the set of each position in the reconstructed point cloud is Not null. Therefore, the fourth step in the recoloring algorithm is not required.

[0137] Fig.11 A one-dimensional (1D) diagram is shown to illustrate a derivation technique that can be used in three dimensions (3D) to determine a set of points that share a reconstructed position during geometry quantization. Diamond dots 1-8 are original dots, and circular black dots R1-R3 are reconstructed positions of reconstructed dots (also called quantized points in some examples).

[0138] In the example, the quantization step size is q, so the range All points in can be quantized to positions nq, where n = 0, ±1, ±2, ... For example, original points 1-3 are in the range of [-3q / 2, -q / 2) and are quantized to reconstructed positions R1, original points 4-5 are in the range of [-q / 2, q / 2) and are quantized to reconstructed positions R2, and original points 6-8 are in the range of [q / 2, 3q / 2) and are quantized to reconstructed positions R3. Therefore, the nearest neighbor of the original point in the reconstructed point cloud is the reconstructed position. In other words, the set of each reconstructed position is the original point quantized to the reconstructed position. Therefore, in some embodiments, a suitable data structure is used to store the set associated with the reconstructed position. Then, based on this data structure, the set can be quickly determined for the reconstructed position No additional k-dimensional tree search is required.

[0139] Based on the quantization results stored in a suitable data structure, the recoloring algorithm can be simplified. For example, the attribute value of each quantized position is assigned based on the attribute value at the original position quantized to the quantized position. Specifically, in some examples, the recoloring algorithm may include 3 steps described in detail in the following specification.

[0140] In the first step, the original position of a point in the point cloud, the attributes associated with the original position, and the reconstructed position are received. i ) i＝0...N-1 represents the original position of the point in the original point cloud, N represents the number of points in the original point cloud, represents the reconstructed position of the reconstructed point cloud, N rec Represents the number of reconstructed locations in the reconstructed point cloud. If there are duplicate points and they are merged, then N rec <N, otherwise N rec =N.

[0141] In the second step, for each reconstructed position in the reconstructed point cloud A data structure storing a set of original positions associated with each reconstructed position may be accessed based on the reconstructed position to determine the set of original positions The original positions in the group are quantized and dequantized as H(i) represents The number of elements in X i (h) indicates One of the elements in . Can be obtained during the geometry quantization process.

[0142] In the third step, we can The attribute value associated with the original position in the reconstructed position is obtained Associated attribute values

[0143] Property values ​​can be calculated using a variety of techniques

[0144] In an embodiment, an average value of multiple attribute values ​​is calculated as the attribute value of the reconstructed position. In the example, a i (h) indicates the same as X i (h) Attribute values ​​associated with the reconstructed points Associated attribute values It can be calculated using (Equation 2):

[0145]

[0146] In another embodiment, the weighted average of multiple attributes is calculated. Such as using (Equation 3):

[0147]

[0148] in, w i (h)≥0 and represents the attribute weight at the corresponding position, which can be compared with the original position X i (h) and reconstructed position The original position X is inversely proportional to the distance between them. i (h) and reconstructed position The distance between can be evaluated by any suitable distance metric. In an example, the distance is a spatial distance, such as Euclidean distance.

[0149] In another embodiment, according to The attribute value of the original position closest to Assign to rebuilt location When multiple original positions and reconstructed positions When the distances are the same, the median attribute value or the average attribute value of the attribute values ​​of the multiple original positions may be assigned to Original Position X i (h) and reconstructed position The distance between can be evaluated by any other distance metric.In an example, the distance is a spatial distance, such as Euclidean distance.

[0150] Fig.12 A flowchart outlining a method (1200) according to an embodiment of the present application is shown. The method (1200) may be used during a point cloud encoding and decoding process. In various embodiments, the method (1200) is performed by a processing circuit, such as a processing circuit in a terminal device (110), a processing circuit that performs the functions of an encoder (203) and / or a decoder (201), a processing circuit that performs the functions of an encoder (300), a decoder (400), an encoder (700) and / or a decoder (800), etc. In some embodiments, the method (1200) is implemented in software instructions so that when the processing circuit executes these software instructions, the processing circuit performs the method (1200). The method starts at (S1201) and proceeds to (S1210).

[0151] At (S1210), one or more original points in the point cloud associated with the reconstructed position are determined. The positions of the one or more original points are reconstructed to the reconstructed position according to the geometry quantization result.

[0152] In some examples, octree partitioning may be performed to encode and decode geometric information. Octree partitioning may divide the space of the point cloud into voxels. The one or more original points are in a voxel and are associated with a reconstructed position representing the voxel. The geometric information of the one or more original points may be reconstructed (e.g., quantized and dequantized) to the reconstructed position. In an embodiment, a data structure associating the one or more original points with the reconstructed position is stored. The data structure may be accessed based on the reconstructed position to retrieve one or more original points associated with the reconstructed position.

[0153] At (S1220), based on the attribute information of the one or more original points, an attribute value of the reconstructed position is determined.

[0154] In an embodiment, when a plurality of original points are associated with the reconstructed position, an average value of the attribute values ​​of the plurality of original points is calculated and used as the determined attribute value of the reconstructed position.

[0155] In some examples, the attribute value of the original point in the plurality of original points may be weighted based on the inverse of the distance between the original point and the reconstructed position.

[0156] In another embodiment, a weighted average of the attribute values ​​of the plurality of original points is calculated as the determined attribute value of the reconstructed position. In an example, the attribute value of an original point in the plurality of original points is weighted based on the inverse of the distance between the original point and the reconstructed position.

[0157] In another embodiment, in response to multiple original points being associated with the reconstructed location, a particular attribute value of a closest point among the multiple original points is assigned as the determined attribute value of the reconstructed location. In one example, in response to multiple closest points having the same shortest distance to the reconstructed location, a median attribute value among the attribute values ​​of the multiple closest points is selected. In another example, in response to multiple closest points having the same shortest distance to the reconstructed location, an average attribute value of the attribute values ​​of the multiple closest points is calculated.

[0158] At (S1230), the texture of the point cloud is encoded by the reconstructed position with the determined attribute value. In an example, the attribute value of the reconstructed position is compressed and included in the encoded bitstream of the point cloud as texture information of the point cloud. Then, the method proceeds to (S1299) and terminates.

[0159] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable storage media. Fig.13 A computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0160] The computer software may be encoded using any suitable machine code or computer language, which may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0161] The instructions may be executed on various types of computers or computer components, including, for example, personal computers, tablets, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0162] Fig.13 The components for computer system (1300) shown in the example are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present application. Nor should the configuration of components be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of computer system (1300).

[0163] The computer system (1300) may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input from one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, taps), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that may not be directly related to a person's conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0164] Input human interface devices may include one or more of the following (only one of each is depicted): keyboard (1301), mouse (1302), track pad (1303), touch screen (1310), data gloves (not shown), joystick (1305), microphone (1306), scanner (1307), camera (1308).

[0165] The computer system (1300) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (1310), a data glove (not shown), or a joystick (1305), although there may also be tactile feedback devices that do not serve as input devices), audio output devices (e.g., speakers (1309), headphones (not depicted)), visual output devices (e.g., screens (1310), including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light emitting diode (OLED) screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which are capable of outputting two-dimensional visual output or output greater than three dimensions through, for example, stereographic output; virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted).

[0166] The computer system (1300) may also include human-accessible storage devices and associated media for the storage devices, such as optical media including CD / DVD ROM / RW (1320) with CD / DVD etc. media (1321), thumb drives (1322), removable hard drives or solid-state drives (1323), legacy magnetic media such as tapes and floppy disks (not depicted), ROM / application specific integrated circuit (ASIC) / programmable logic device (PLD) based special devices such as security protection devices (not depicted), and the like.

[0167] Those skilled in the art will also appreciate that the term "computer-readable storage medium" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transient signals.

[0168] The computer system (1300) may also include an interface to one or more communication networks. The network may be, for example, wireless, wired, optical. The network may also be local, wide area, metropolitan, vehicle-mounted and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including Global System for Mobile Communications (GSM), third generation (3G), fourth generation (4G), fifth generation (5G), long term evolution (LTE), etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle-mounted networks and industrial networks including controller area network bus (CANBus), and the like. Some networks typically require an external network interface adapter attached to some general data port or peripheral bus (1349) (e.g., a universal serial bus (USB) port of the computer system (1300)); other networks are typically integrated into the core of the computer system (1300) by attaching to a system bus as described below (e.g., integrated into a PC computer system through an Ethernet interface, or integrated into a smart phone computer system through a cellular network interface). Using any of these networks, the computer system (1300) can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast TV), one-way send only (e.g., CANBus connected to some CANBus devices), or bidirectional, for example, using a local area digital network or a wide area digital network to connect to other computer systems. Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0169] The above-mentioned human-machine interface device, human-accessible storage device, and network interface may be attached to the core (1340) of the computer system (1300).

[0170] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1343), hardware accelerators for certain tasks (1344), and the like. These devices, along with read-only memory (ROM) (1345), random access memory (1346), and internal mass storage devices (1347) such as internal non-user accessible hard drives, solid-state drives (SSDs), etc., may be connected via a system bus (1348). In some computer systems, the system bus (1348) may be accessible in the form of one or more physical plugs to enable expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached to the core's system bus (1348) directly or via a peripheral bus (1349). Architectures for peripheral buses include peripheral component interconnect (PCI), USB, and the like.

[0171] The CPU (1341), GPU (1342), FPGA (1343), and accelerator (1344) may execute certain instructions, which in combination may constitute the above-mentioned computer code. The computer code may be stored in ROM (1345) or RAM (1346). Transient data may also be stored in RAM (1346), while permanent data may be stored, for example, in an internal mass storage device (1347). Fast storage and retrieval of any memory device may be achieved by using a cache memory, which may be closely associated with one or more CPUs (1341), GPUs (1342), mass storage devices (1347), ROMs (1345), RAMs (1346), etc.

[0172] The computer readable storage medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be media and computer codes designed and constructed specifically for the purpose of this application, or may be of a type well known and available to technicians in the computer software field.

[0173] By way of example and not limitation, a computer system having the architecture (1300) and in particular the core (1340) may provide functionality resulting from the execution of software embodied in one or more tangible computer-readable storage media by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable storage media may be media associated with the user-accessible mass storage device introduced above and certain storage devices of the non-temporary nature of the core (1340) (e.g., a core internal mass storage device (1347) or ROM (1345)). Software implementing various embodiments of the present application may be stored in such devices and executed by the core (1340). Depending on specific needs, the computer-readable storage medium may include one or more memory devices or chips. The software may enable the core (1340) and, in particular, the processor therein (including a CPU, GPU, FPGA, etc.) to perform specific processes or specific portions of specific processes described herein, including defining data structures stored in the RAM (1346) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality produced by logic hardwired or otherwise embodied in circuits (e.g., accelerators (1344)) that may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable storage media may encompass circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both, where appropriate. The present application encompasses any suitable combination of hardware and software.

[0174] Although the present application describes several exemplary embodiments, various modifications, permutations and combinations, and various alternative equivalents are possible within the scope of the present application. Therefore, it should be understood that within the spirit and scope of the application, those skilled in the art can design various systems and methods that are not explicitly shown or described herein but can embody the principles of the present application.

Claims

1. A point cloud encoding method, characterized in that: include: determining one or more original points in the point cloud associated with a reconstructed position, the positions of the one or more original points being reconstructed to the reconstructed position based on the geometric structure quantization result; Determining attribute values ​​of the reconstructed position based on the attribute information of the one or more original points; encoding a texture of the point cloud by means of the reconstructed positions having the determined attribute values; storing a data structure that associates the one or more original points with the reconstructed position based on the geometric structure quantization result; Performing octree partitioning to divide the space of the point cloud into voxels, wherein a voxel is a cube of a unit size, each voxel contains one or more points in the point cloud, and the one or more points are quantized to the center point of the voxel; and associating the one or more original points located in a voxel with the reconstructed position representing the voxel; The data structure is accessed based on the reconstructed position to retrieve the one or more original points associated with the reconstructed position; wherein, for each position in the reconstructed point cloud that is a voxel center point, the original point in the voxel that is merged into the center position is determined based on the data structure.

2. The method according to claim 1, characterized in that Also includes: In response to a plurality of original points being associated with the reconstructed position, an average of the attribute values ​​of the plurality of original points is calculated as the determined attribute value of the reconstructed position.

3. The method according to claim 2, characterized in that Also includes: A weighted average of the attribute values ​​of the plurality of original points is calculated and used as the determined attribute value of the reconstructed position.

4. The method according to claim 3, characterized in that Also includes: The attribute value of the original point among the plurality of original points is weighted based on the inverse of the distance between the original point among the plurality of original points and the reconstructed position.

5. The method according to claim 1, characterized in that: Also includes: In response to multiple original points being associated with the reconstructed position, a specific attribute value of a point closest to the multiple original points is assigned as the determined attribute value of the reconstructed position, and the specific attribute value is: a median attribute value among the multiple attribute values ​​closest to the points or an average attribute value of the multiple attribute values ​​closest to the points.

6. The method according to claim 5, characterized in that Also includes: In response to a plurality of closest points among the plurality of original points having the same shortest distance to the reconstructed position, a median attribute value among the attribute values ​​of the plurality of closest points is selected.

7. The method according to claim 5, characterized in that Also includes: In response to a plurality of closest points among the plurality of original points having the same shortest distance to the reconstructed position, an average attribute value of the attribute values ​​of the plurality of closest points is calculated.

8. A point cloud encoding and decoding device, characterized in that: include: A processing circuit configured to execute the method as claimed in any one of claims 1 to 7.

9. A point cloud encoding and decoding device, characterized in that: include: an original point determination unit, configured to determine one or more original points in the point cloud associated with a reconstructed position, wherein the positions of the one or more original points are reconstructed to the reconstructed position according to the geometric structure quantization result; an attribute value determining unit, configured to determine an attribute value of the reconstructed position based on the attribute information of the one or more original points; as well as an encoding unit, configured to encode the texture of the point cloud by means of the reconstructed positions having the determined attribute values; a memory configured to store a data structure associating the one or more original points with the reconstructed position based on the geometric structure quantization result; A partitioning unit, configured to perform octree partitioning to partition the space of the point cloud into voxels, wherein a voxel is a cube of a unit size, each voxel contains one or more points in the point cloud, and the one or more points are quantized to a center point of the voxel; an associating unit for associating the one or more original points located in a voxel with the reconstructed position representing the voxel; A retrieval unit is used to access the data structure based on the reconstructed position to retrieve the one or more original points associated with the reconstructed position; wherein, for each position in the reconstructed point cloud that is a voxel center point, the original point in the voxel that is merged into the center position is determined based on the data structure.

10. An electronic device, characterized in that: The invention comprises a memory for storing computer-readable instructions; a processor for reading the computer-readable instructions and executing the method according to any one of claims 1 to 7 as instructed by the computer-readable instructions.

11. A computer-readable storage medium having instructions stored thereon, wherein when the instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

12. A method for processing a code stream, characterized in that: The code stream is generated according to the encoding method according to any one of claims 1 to 7.

Images