Point cloud compression using video coding with temporally consistent patches

Abstract

A system comprising an encoder configured to compress attribute information and / or spatial information of a point cloud, and / or a decoder configured to decompress the compressed attribute information and / or spatial information of the point cloud. To compress the attribute information and / or spatial information, the encoder is configured to convert the point cloud into an image-based representation. Additionally, the decoder is configured to generate a decompressed point cloud based on the image-based representation of the point cloud. In some embodiments, the encoder generates temporally consistent patches for multiple versions of the point cloud at multiple time instants, and uses these temporally consistent patches to generate image-based representations of the point cloud at the multiple time instants.

Description

Background Art Technical Field

[0002] The present disclosure generally relates to compression and decompression of point clouds comprising a plurality of points, each point having associated spatial information and attribute information.

[0003] Related technical description

[0004] Various types of sensors (such as light detection and ranging (LIDAR) systems, 3D cameras, 3D scanners, etc.) can capture data indicating the location of points in three-dimensional space (e.g., positions in the X, Y, and Z planes). Furthermore, such systems can capture attribute information, such as color information (e.g., RGB values), texture attributes, intensity attributes, reflectivity attributes, motion-related attributes, modality attributes, or various other attributes, in addition to spatial information about the corresponding points. In some cases, additional attributes can be assigned to the corresponding points, such as a timestamp when the point was captured. The points captured by such sensors can constitute a "point cloud," which includes a set of points, each with associated spatial information and one or more associated attributes. In some cases, a point cloud can include thousands, hundreds of thousands, millions, or even more points. Furthermore, in some cases, a point cloud can be generated, for example, in software, separate from the point cloud being captured by one or more sensors. In either case, such a point cloud can include a large amount of data, and storing and transmitting these point clouds can be costly and time-consuming. Summary of the Invention

[0005] In some embodiments, a system includes one or more sensors configured to capture points that collectively constitute a point cloud, wherein each point includes spatial information identifying a spatial location of the respective point and attribute information defining one or more attributes associated with the respective point.

[0006] In some embodiments, a system further includes an encoder configured to compress attribute information and / or spatial information of a point. To compress the attribute and / or spatial information, the encoder is configured to determine a plurality of patches for a point cloud, each patch corresponding to a portion of the point cloud. The encoder is further configured to, for each patch: generate a patch image comprising a set of points corresponding to the patch projected onto a patch plane; and generate another patch image comprising geometric information, such as depth information, of the set of points corresponding to the patch, wherein the geometric information comprises a depth of the point in a direction perpendicular to the patch plane.

[0007] For example, a geometric patch image corresponding to a patch projected onto a patch plane can depict points of a point cloud included in the patch in two directions (such as the X and Y directions). The points of the point cloud can be projected onto the patch plane approximately perpendicular to a normal vector that is perpendicular to the surface of the point cloud at the location of the patch. Furthermore, for example, a geometric patch image that includes depth information for a set of points included in the patch can depict depth information, such as depth distance in the Z direction. To depict depth information, the geometric patch image can include a parameter that varies in intensity based on the depth of a point in the point cloud at a particular location in the patch image. For example, a geometric patch image depicting depth information can have the same shape as an attribute patch image representing attributes of the points projected onto the patch plane. However, the geometric information patch image can be an image that includes image attributes (such as one or more colors) that vary in intensity based on depth, where the intensity of one or more image attributes corresponds to the depth of a corresponding point in the point cloud at a location in the geometric patch image where the image attribute is displayed in the geometric patch image depicting depth. For example, points closer to the patch plane may be encoded as darker values ​​in the patch image depicting depth, and points further from the patch plane may be encoded as brighter values ​​in the patch image depicting depth, such as in a monochrome patch image depicting depth. Thus, the depth information patch image, when aligned with other patch images representing attribute values ​​of points projected onto the patch plane, may indicate the relative depth of a point projected onto the patch plane based on the intensity of the corresponding image attributes at locations in the geometric patch image corresponding to the locations of the points in the other patch images including the point cloud points projected onto the patch plane.

[0008] In some embodiments, the encoder can independently generate a patch for a time-varying point cloud at each of multiple time instants. For example, for a point cloud representing a walking person, the encoder can independently generate a patch that includes the portion of the person projected onto the patch plane determined at different time instants, without considering patches determined at other time instants.

[0009] Alternatively, in some embodiments, the encoder can generate patches for a time-varying cloud based on a mapping of points as they move from one moment in time to another. In such embodiments, a time-consistent patch can be generated, where the same time-consistent patch can be used for multiple versions of the point cloud at different moments in time. Additionally, the time-consistent patch can maintain the same patch size and orientation relative to the point cloud across multiple moments in time.

[0010] The encoder is further configured to pack the generated patch images (including the geometric patch images and one or more additional patch images for one or more other attributes (such as color, texture, reflectivity, etc.)) of each of the determined patches into one or more image frames. For example, the encoder may utilize a video encoding component according to the High Efficiency Video Coding (HEVC / H.265) standard or other suitable standards (such as the Advanced Video Coding (AVC / H.264) standard, the AOMedia Video 1 (AV1) video coding format produced by the Alliance for Open Media (AOM), etc.). In some embodiments, the encoder may utilize an image encoder according to the Moving Picture Experts Group (MPEG), the Joint Photographic Experts Group (JPEG) standard, the International Telecommunication Union telecommunication standard (e.g., the ITU-T standard), etc.

[0011] In some embodiments, a decoder is configured to receive one or more encoded image frames, the one or more encoded image frames comprising patch images for a plurality of patches of a compressed point cloud, wherein for each patch, the one or more encoded image frames comprise: a patch image comprising a set of points of the patch projected onto a patch plane, and a patch image comprising depth information for the set of points of the patch, wherein the depth information indicates the depth of the points of the patch in a direction perpendicular to the patch plane. In some embodiments, the depth patch image can be packed into the image frame along with other attribute patch images. For example, the decoder can receive one or more image frames comprising packed patch images generated by the above-described encoder. In some embodiments, the one or more image frames can comprise temporally consistent patches. In some embodiments, a common occupancy map can be signaled for a sequence of patch images using the same temporally consistent patch. In some embodiments, at least one of the image frames can be downscaled at the encoder, and the decoder can upscale the downscaled image frame in the 2D video domain.

[0012] The decoder is further configured to perform video decoding on one or more video-encoded image frames including patch images. In some embodiments, the decoder may utilize a video decoder according to the High Efficiency Video Coding (HEVC) standard or other suitable standards (such as the Advanced Video Coding (AVC) standard, the AOMedia Video 1 (AV1) video coding format, etc.). In some embodiments, the decoder may utilize an image decoder according to the Moving Picture Experts Group (MPEG) or Joint Photographic Experts Group (JPEG) standards, etc.

[0013] The decoder is further configured to receive or determine relationship information indicating a relationship between corresponding attribute patch images, depth patch images, and / or image frames. For example, the decoder may receive relationship information in a compressed point cloud file, where the relationship information is determined by the encoder. Additionally, in some embodiments, the decoder may determine the relationship information based on information included in the compressed point cloud file (such as an occupancy map and / or auxiliary information of the compressed point cloud). In some embodiments, the decoder may utilize the received or determined relationship information to adjust one or more parameters used for video decoding of the video-encoded image frames.

[0014] The decoder is further configured to: for each patch, determine spatial information of the set of points of the patch at least in part based on an attribute patch image including the set of points of the patch projected onto the patch plane and a geometric patch image including depth information of the set of points of the patch, and generate a reconstructed version of the compressed point cloud based at least in part on the determined spatial information of multiple patches and the attribute information included in the patches.

[0015] In some embodiments, a non-transitory computer-readable medium stores program instructions that, when executed by one or more processors, cause the one or more processors to implement an encoder as described herein to compress geometric information and attribute information of a point cloud.

[0016] In some embodiments, a non-transitory computer-readable medium stores program instructions that, when executed by one or more processors, cause the one or more processors to implement a decoder as described herein to decompress geometric information and attribute information of a point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A system according to some embodiments is shown, the system including a sensor that captures information about points of a point cloud and an encoder that compresses spatial information and attribute information of the point cloud, where the compressed spatial information and attribute information is sent to a decoder.

[0018] Figure 2A Components of an encoder for intra-encoding a point cloud frame are shown according to some embodiments.

[0019] Figure 2B Components of a decoder for decoding a point cloud frame are shown according to some embodiments.

[0020] Figure 2C Components of an encoder for inter-encoding a point cloud are shown according to some embodiments.

[0021] Figure 2D Components of a decoder for decoding point cloud inter frames are shown according to some embodiments.

[0022] Figure 3A An exemplary patch segmentation process is shown according to some embodiments.

[0023] Figure 3B An exemplary image frame including a packed patch image and a padding portion is shown according to some embodiments.

[0024] Figure 4 An exemplary image frame including a patch portion and a fill portion is shown according to some embodiments.

[0025] Figure 5 An exemplary patch segmentation process for generating a time-consistent patch is shown in accordance with some embodiments.

[0026] Figure 6 An exemplary sequence of image frames including time-aligned patches is shown in accordance with some embodiments.

[0027] Figure 7 Shown are examples of versions of a point cloud at different times with different numbers of points and mappings between the points at different times, according to some embodiments.

[0028] Figure 8 A process for determining a time-consistent patch is shown according to some embodiments.

[0029] Figure 9 A process for determining segment size for motion estimation is shown according to some embodiments.

[0030] Figure 10A Steps for removing outliers in determining a time-consistent patch are shown, according to some embodiments.

[0031] Figure 10B Steps for applying a noise reduction filter when determining a time-consistent patch are shown according to some embodiments.

[0032] Figure 10C Steps for resampling one or more versions of a point cloud at different time instances in determining a time-consistent patch are shown, according to some embodiments.

[0033] Figure 11 A bitstream structure for compressing a point cloud is shown according to some embodiments.

[0034] Figure 12A A process for compressing attribute information and spatial information of a point cloud is shown according to some embodiments.

[0035] Figure 12BA process for decompressing attribute information and spatial information of a point cloud is shown according to some embodiments.

[0036] Figure 12C Patch images are shown being generated and packed into image frames to compress attribute information and spatial information of a point cloud, according to some embodiments.

[0037] Figure 12D Patch images are shown being generated and packed into image frames to compress attribute information and spatial information of a moving or changing point cloud, according to some embodiments.

[0038] Figure 12E A decoder according to some embodiments is shown that receives an image frame including patch images, patch information, and an occupancy map and generates a decompressed representation of a point cloud.

[0039] Figure 13 Compressed point cloud information is shown being used in a 3D application according to some embodiments.

[0040] Figure 14 Compressed point cloud information is shown being used in a virtual reality application according to some embodiments.

[0041] Figure 15 An exemplary computer system is shown that may implement an encoder or decoder according to some embodiments.

[0042] This specification includes references to "one embodiment" or "an embodiment." The appearance of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. The particular features, structures, or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0043] The term "comprising" is open-ended. As used in the appended claims, the term does not exclude additional structures or steps. Consider the following recited claim: "An apparatus comprising one or more processor units..." Such a claim does not exclude the apparatus from including additional components (e.g., a network interface unit, graphics circuitry, etc.).

[0044] Various units, circuits, or other components may be described or recited as being “configured to” perform one or more tasks. In such contexts, “configured to” is used to imply a structure (e.g., circuitry) that performs the one or more tasks during operation by indicating that the unit / circuit / component includes the structure. Thus, the unit / circuit / component is said to be configured to perform the task even when the specified unit / circuit / component is not currently operational (e.g., not turned on). Units / circuits / components used with the “configured to” language include hardware, such as circuitry, memory storing program instructions executable to implement the operation, and the like. Reference to a unit / circuit / component being “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. §112(f) with respect to that unit / circuit / component. Furthermore, “configured to” may include a general structure (e.g., general circuitry) manipulated by software and / or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in a manner capable of performing the one or more tasks to be solved. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor fabrication facility) to produce a device (eg, an integrated circuit) suitable for implementing or performing one or more tasks.

[0045] "First," "Second," etc. As used herein, these terms act as labels for the nouns that precede them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing a write operation for a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0046] "Based on." As used herein, this term is used to describe one or more factors that influence a determination. This term does not exclude additional factors that influence the determination. That is, the determination may be based solely on these factors, or at least in part on these factors. Consider the phrase "A is determined based on B." In this case, B is a factor that influences the determination of A, and such a phrase does not exclude that the determination of A may also be based on C. In other examples, A may be determined based solely on B. DETAILED DESCRIPTION

[0047] As data acquisition and display technologies become more advanced, the ability to capture point clouds comprising tens of thousands of points in 2D or 3D space has increased (such as via LIDAR systems). Furthermore, the development of advanced display technologies (such as virtual reality or augmented reality systems) has increased the potential uses of point clouds. However, point cloud files are typically very large, and storing and transmitting these point cloud files can be costly and time consuming. For example, communication of a point cloud over a private network or a public network (such as the Internet) can require a considerable amount of time and / or network resources, such that some uses of the point cloud data (such as real-time uses) may be limited. Additionally, the storage requirements of the point cloud files may consume a significant amount of storage capacity of the device storing the point cloud files, which may also limit potential applications using the point cloud data.

[0048] In some embodiments, an encoder can be used to generate a compressed point cloud to reduce the cost and time associated with storing and transmitting large point cloud files. In some embodiments, a system can include an encoder that compresses attribute information and / or spatial information of a point cloud file, allowing the point cloud file to be stored and transmitted more quickly and in a manner that occupies less storage space than an uncompressed point cloud. In some embodiments, the compression of attributes of points in a point cloud can be transmitted over a network in real time or near real time. For example, a system can include a sensor that captures attribute information about points in an environment in which the sensor is located, where the captured points and corresponding attributes constitute a point cloud. The system can also include an encoder that compresses the attribute information of the captured point cloud. The compressed attribute information of the point cloud can be transmitted over a network in real time or near real time to a decoder that decompresses the compressed attribute information of the point cloud. The decompressed point cloud can be further processed, for example, to make control decisions based on the surrounding environment at the sensor location. The control decisions can then be transmitted back to a device at or near the sensor location, where the device receiving the control decisions implements the control decisions in real time or near real time. In some embodiments, the decoder can be associated with an augmented reality system, and the decompressed attribute information can be displayed or otherwise used by the augmented reality system. In some embodiments, compressed attribute information about the point cloud can be sent along with compressed spatial information about the points in the point cloud. In other embodiments, the spatial information and attribute information can be encoded separately and / or sent separately to the decoder.

[0049] In some embodiments, a system may include a decoder that receives one or more point cloud data sets including compressed attribute information from a remote server or other storage device storing one or more point cloud files via a network. For example, a 3D display, a holographic display, or a head-mounted display may be manipulated in real time or near real time to display different portions of a virtual world represented by a point cloud. To update the 3D display, holographic display, or head-mounted display, a system associated with the decoder may request point cloud data from a remote server based on user manipulation of the display, and the point cloud data may be transmitted from the remote server to the decoder and decoded by the decoder in real time or near real time. The display may then be updated with updated point cloud data (such as updated point attributes) responsive to the user manipulation.

[0050] In some embodiments, a system may include one or more LIDAR systems, 3D cameras, 3D scanners, etc., and such sensor devices may capture spatial information, such as the X, Y, and Z coordinates of points in the sensor device's field of view. In some embodiments, the spatial information may be relative to a local coordinate system or may be relative to a global coordinate system (e.g., a Cartesian coordinate system may have fixed reference points, such as fixed points on the Earth, or may have non-fixed local reference points, such as sensor locations).

[0051] In some embodiments, such sensors may also capture attribute information of one or more points, such as color attributes, texture attributes, reflectivity attributes, velocity attributes, acceleration attributes, time attributes, modality, and / or various other attributes. In some embodiments, in addition to LIDAR systems, 3D cameras, 3D scanners, etc., other sensors may capture attribute information to be included in the point cloud. For example, in some embodiments, a gyroscope or accelerometer may capture motion information to be included in the point cloud as an attribute associated with one or more points of the point cloud. For example, a vehicle equipped with a LIDAR system, a 3D camera, or a 3D scanner may include the vehicle's direction and velocity in the point cloud captured by the LIDAR system, 3D camera, or 3D scanner. For example, when points in the vehicle's field of view are captured, these points may be included in a point cloud, wherein the point cloud includes the captured points and associated motion information corresponding to the state of the vehicle at the time the points were captured.

[0052] Example system layout

[0053] Figure 1 A system according to some embodiments is shown, the system including a sensor that captures information about points of a point cloud and an encoder that compresses attribute information of the point cloud, where the compressed attribute information is sent to a decoder.

[0054] System 100 includes a sensor 102 and an encoder 104. Sensor 102 captures a point cloud 110, which includes points representing structures 106 in a view 108 of sensor 102. For example, in some embodiments, structure 106 may be a mountain, a building, a sign, the surroundings of a street, or any other type of structure. In some embodiments, a captured point cloud, such as captured point cloud 110, may include spatial information and attribute information about the points included in the point cloud. For example, point A in captured point cloud 110 includes X, Y, and Z coordinates, as well as attributes 1, 2, and 3. In some embodiments, the attributes of a point may include attributes such as R, G, and B color values, velocity at the point, acceleration at the point, reflectivity of the structure at the point, a timestamp indicating when the point was captured, a string value indicating the modality at the time the point was captured, such as "walking," or other attributes. Captured point cloud 110 may be provided to encoder 104, which generates a compressed version of the point cloud (compressed attribute information 112), which is transmitted to decoder 116 via network 114. In some embodiments, a compressed version of a point cloud (such as compressed attribute information 112) may be included in a common compressed point cloud that also includes compressed spatial information about the points of the point cloud, or in some embodiments, the compressed spatial information and compressed attribute information may be transmitted as separate data sets.

[0055] In some embodiments, encoder 104 can be integral to sensor 102. For example, encoder 104 can be implemented in hardware or software included in a sensor device, such as sensor 102. In other embodiments, encoder 104 can be implemented on a separate computing device adjacent to sensor 102.

[0056] Exemplary 3D Intra Encoder

[0057] Figure 2A Components of an encoder for intra-encoding a 3D point cloud frame according to some embodiments are shown. Figure 1 The described encoder can be used with Figure 2A The encoder 200 described in Figure 2C The encoder 250 described in operates in a similar manner.

[0058] The encoder 200 receives the uncompressed point cloud 202 and generates compressed point cloud information 204. In some embodiments, an encoder such as the encoder 200 may receive the uncompressed point cloud 202 from a sensor such as Figure 1 The uncompressed point cloud 202 may be received by the sensor 102 shown, or in some embodiments, the uncompressed point cloud 202 may be received from another source, such as a graphics generation component that generates the uncompressed point cloud in software, for example.

[0059] In some embodiments, an encoder (such as encoder 200) includes a decomposition into patches module 206, a packing module 208, a spatial image generation module 210, a texture image generation module 212, and an attribute information generation module 214. In some embodiments, an encoder (such as encoder 200) also includes an image frame padding module 216, a video compression module 218, and a multiplexer 224. Additionally, in some embodiments, an encoder (such as encoder 200) may include an occupancy map compression module (such as occupancy map compression module 220) and an auxiliary patch information compression module (such as auxiliary patch information compression module 222). In some embodiments, an encoder (such as encoder 200) converts a 3D point cloud into an image-based representation and some metadata (e.g., occupancy map and patch information) required to convert a compressed point cloud back into a decompressed point cloud.

[0060] In some embodiments, the conversion process decomposes the point cloud into a set of patches (e.g., a patch is defined as a contiguous subset of the surface described by the point cloud), which may or may not overlap, such that each patch can be described by a depth field relative to a plane in 2D space. Figure 3A More details on the patch dissolution process are provided.

[0061] After determining the patches for the point cloud being compressed or in conjunction with these patches, a 2D sampling process is performed in the planes associated with these patches. The 2D sampling process can be applied to approximate each patch with a uniformly sampled point cloud, which can be stored as a set of 2D patch images describing the geometry / texture / attributes of the point cloud at the patch location. A "packing" module 208 can store the 2D patch images associated with the patches in a single (or multiple) 2D image, which are referred to herein as "image frames" or "video image frames." In some embodiments, a packing module (such as packing module 208) can pack the 2D patch images so that the packed 2D patch images do not overlap (even though the outer bounding box of one patch image may overlap the outer bounding box of another patch image). In addition, the packing module can pack the 2D patch images in a manner that minimizes unused image pixels of the image frame.

[0062] In some embodiments, the "Geometry / Texture / Attribute Generation" modules (such as modules 210, 212, and 214) each generate a 2D patch image associated with the geometry / texture / attribute of the point cloud at a given patch location. As previously described, the packing process (such as the process performed by the packing module 208) can leave some empty space between the packed 2D patch images in the image frame. Additionally, a padding module (such as the image frame padding module 216) can fill in such areas to generate an image frame that is suitable for 2D video and image codecs.

[0063] In some embodiments, an occupancy map (e.g., for each pixel or pixel block, binary information describing whether the pixel or pixel block is filled) may be generated and compressed, for example, by the occupancy map compression module 220. The occupancy map may be sent to a decoder to enable the decoder to distinguish filled pixels from unfilled pixels of an image frame.

[0064] Note that other metadata associated with the patch may also be sent to the decoder for use in the decompression process. For example, patch information indicating the size and shape of the patches determined for the point cloud and packed into the image frame may be generated and / or encoded by an auxiliary patch information compression module (such as auxiliary patch information compression module 222). In some embodiments, one or more image frames may be encoded by a video encoder (such as video compression module 218). In some embodiments, the video encoder (such as video compression module 218) may operate according to the High Efficiency Video Coding (HEVC) standard or other suitable video coding standard. In some embodiments, the encoded video image, the encoded occupancy map information, and the encoded auxiliary patch information may be multiplexed by a multiplexer (such as multiplexer 224) and provided to a recipient as compressed point cloud information (such as compressed point cloud information 204).

[0065] In some embodiments, the occupancy map may be encoded and decoded by a video compression module, such as video compression module 218. This may be done at an encoder, such as encoder 200, so that the encoder has an accurate representation of the occupancy map when it is decoded by a decoder. Additionally, when determining an occupancy map for an image frame, variations in the image frame due to lossy compression and decompression may be accounted for by an occupancy map compression module, such as occupancy map compression module 220.

[0066] Exemplary 3D Intra Decoder

[0067] Figure 2B Components of a decoder for decoding a point cloud frame according to some embodiments are shown. The decoder 230 receives compressed point cloud information 204, which may be the same compressed point cloud information 204 generated by the encoder 200. The decoder 230 generates a reconstructed point cloud 246 based on the received compressed point cloud information 204.

[0068] In some embodiments, a decoder (such as decoder 230) includes a demultiplexer 232, a video decompression module 234, an occupancy map decompression module 236, and an auxiliary patch information decompression module 238. In addition, a decoder (such as decoder 230) includes a point cloud generation module 240 that reconstructs a point cloud based on patch images included in one or more image frames included in the received compressed point cloud information (such as compressed point cloud information 204). In some embodiments, a decoder (such as decoder 203) also includes a smoothing filter (such as smoothing filter 244). In some embodiments, the smoothing filter can smooth out inconsistencies at the edges of patches where the point cloud generation module has used data included in the patch images of the patches to reconstruct the point cloud from the patch images for the patches. In some embodiments, the smoothing filter can be applied to pixels located on the patch boundaries to mitigate distortion that may be caused by the compression / decompression process.

[0069] Exemplary 3D Interframe Encoder

[0070] Figure 2C Components of an encoder for inter-coding 3D point cloud frames are shown in accordance with some embodiments. A 3D point cloud inter-encoder, such as inter-point cloud encoder 250, can encode a 3D version of a point cloud at a particular time instant (e.g., an image frame) while taking into account one or more previously encoded / decoded versions of the point cloud at other time instants (e.g., other image frames) as reference frames.

[0071] In some embodiments, an encoder (such as encoder 250) includes a point cloud resampling module 252, a 3D motion compensation and delta vector prediction module 254, a spatial image generation module 256, a texture image generation module 258, and an attribute image generation module 260. In some embodiments, an encoder (such as encoder 250) may also include an image padding module 262 and a video compression module 264. An encoder (such as encoder 250) may generate compressed point cloud information (such as compressed point cloud information 266). In some embodiments, the compressed point cloud information may be reference point cloud information previously encoded by the encoder, such as information from one or more reference image frames or derived from one or more reference image frames. In this way, an encoder (such as encoder 250) for point cloud inter-frames may generate more compact compressed point cloud information by not repeating information included in a reference image frame, but instead conveying the difference between the reference frame and the current state of the point cloud.

[0072] In some embodiments, an encoder (such as encoder 250) can be combined with or share modules with a point cloud intra encoder (such as encoder 200). In some embodiments, a point cloud resampling module (such as point cloud resampling module 252) can resample points in an input point cloud image frame to determine a one-to-one mapping between points in a patch of a current image frame and points in a patch of a reference image frame for the point cloud. In some embodiments, a 3D motion compensation and delta vector prediction module (such as 3D motion compensation and delta vector prediction module 254) can apply temporal prediction to the geometry / texture / attributes of the resampled points of the patch. The prediction residual can be stored in the image, which can then be padded and compressed using a video / image codec. For spatial changes in the points of the patch between the reference frame and the current frame, the 3D motion compensation and delta vector prediction module 254 can determine a corresponding vector for each point or segment of the point cloud, where the corresponding vectors indicate how the points or segments move from the reference frame to the target frame (e.g., the current frame). The 3D motion compensation and delta vector prediction module 254 can then use different image parameters to encode the motion vector. For example, a change in the X direction of a point can be represented by the amount of red included at that point in the patch image that includes the point. In a similar manner, a change in the Y direction of a point can be represented by the amount of blue included at that point in the patch image that includes the point. Additionally, in a similar manner, a change in the Z direction of a point can be represented by the amount of green included at that point in the patch image that includes the point. In some embodiments, other characteristics of the image included in the patch image can be adjusted to indicate the motion of a point included in the patch between a reference frame for the patch and a current frame for the patch.

[0073] Exemplary 3D Interframe Decoder

[0074] Figure 2D Components of a decoder for decoding cloud frames according to some embodiments are shown. In some embodiments, a decoder (such as decoder 280) includes a video decompression module 270, an inverse 3D motion compensation and inverse delta prediction module 272, a point cloud generation module 274, and a smoothing filter 276. In some embodiments, a decoder (such as decoder 280) can be combined with a decoder (such as decoder 230) or can share some components with the decoder, such as the video decompression module and / or the smoothing filter. In decoder 280, the video / image stream is first decoded, and then inverse motion compensation and delta prediction procedures can be applied. The resulting image is then used to reconstruct a point cloud, which can be smoothed as previously described to generate a reconstructed point cloud 282.

[0075] Segmentation process

[0076] Figure 3A An exemplary segmentation process for determining patches for a point cloud according to some embodiments is shown. Figure 3A The segmentation process described in . The segmentation process can decompose the point cloud into the minimum number of patches (e.g., continuous subsets of the surface described by the point cloud) while ensuring that the corresponding patches can be represented by a depth field relative to the patch plane. This can be done without significantly losing shape information.

[0077] In some embodiments, the segmentation process includes:

[0078] Assume that point cloud PC is the input point cloud to be divided into patches, and {P(0), P(1)…, P(N-1)} are the positions of the points of point cloud PC.

[0079] In some embodiments, a fixed set of K 3D orientations D = {D(0), D(1), ..., D(K-1)} is predefined. For example, D can be selected as follows: D = {(1.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0), (-1.0, 0.0, 0.0), (0.0, -1.0, 0.0), (0.0, 0.0, -1.0)}

[0080] In some embodiments, the surface normal at each point P(i) is estimated. Any suitable algorithm may be used to determine the surface normal. For example, one technique may include obtaining a set H of the "N" closest points to P(i) and fitting a plane Π(i) to H(i) using principal component analysis techniques. The normal to P(i) can be determined by fitting the normal to Let Π(i) be used to estimate Π(i). Note that "N" can be a user-defined parameter or can be found by applying an optimization procedure. "N" can also be fixed or adaptive. The normal values ​​can then be consistently oriented by using a minimum spanning tree method.

[0081] Normal-based segmentation: At 302, the points of the point cloud PC can be segmented by using the normal-based segmentation algorithm. The maximized direction D(k) is associated to obtain the initial segmentation S0 of the corresponding point, where <.│.> is the normalized dot product of R3. The pseudo code is provided below.

[0082]

[0083] o Iterative segmentation refinement: At 302, an iterative segmentation refinement process may be performed. Note that

[0084] The partition S0 associates the corresponding points with the plane Π(i) that best preserves the geometry of its neighborhood (e.g., the neighborhood of the segment). In some cases, the partition S0 may generate too many small connected components with irregular boundaries, which may lead to poor compression performance. To avoid such problems, the following iterative partition refinement procedure can be applied:

[0085] 1. An adjacency graph A can be constructed by associating a vertex V(i) with the corresponding point P(i) of the point cloud PC and adding R edges {E(i,j(0)),…,E(i,j(R-1)} connecting the vertex V(i) to its nearest neighbors {V(j(0)),V(j(1)),…,V(j(R-1))}. More precisely,

[0086] {V(j(0)),V(j(1)),…,V(j(R-1))} can be the vertices associated with the points {P(j(0)),P(j(1)),…,P(j(R-1))}, which can be the nearest neighbors of P(i). Note that "R" can be a user-defined parameter or can be found by applying an optimization procedure. This parameter can also be fixed or adaptive.

[0087] 2. In each iteration, the points of the point cloud PC can be traversed and each vertex can be compared with the The direction of maximization D(k) is associated with

[0088] |ζ(i)| is the number of R nearest neighbors of V(i) that belong to the same cluster, and

[0089] is a parameter that controls the regularity of the generated patch. Note that the parameters λ and R can be user-defined or determined by applying an optimization procedure. These two parameters can also be fixed or adaptive. In some embodiments, the "user" referred to herein may be an engineer who configures the point cloud compression techniques described herein for one or more applications.

[0090] 3. The pseudo code example is provided below

[0091]

[0092]

[0093] *In some embodiments, the pseudocode shown above may also include an early termination step. For example, the search may terminate early if a certain score is reached, or if the difference between the achieved score and the best score changes by a certain amount or less. Additionally, the search may terminate if the cluster index (clusterindex) does not change after a certain number of iterations (l=m).

[0094] o Patch Segmentation: In some embodiments, the patch segmentation procedure 306 also segments the clusters detected in the previous step into patches, which can be represented by a depth field relative to the projection plane. According to some embodiments, the method proceeds as follows:

[0095] 1. First, construct a cluster-based adjacency graph with R' number of neighbors, while only considering points belonging to the same cluster as neighbors. Note that R' may be different from the number of neighbors R used in the previous step.

[0096] 2. Next, extract the different connected components of the cluster-based adjacency graph. Only consider connected components with a number of points greater than the parameter α. Suppose CC = {CC(0), CC(1), …, CC(M-1)} is the set of extracted connected components.

[0097] 3. The corresponding connected component CC(m) inherits the orientation D(m) of the cluster to which it belongs. Then, the points of CC(m) are projected onto the projection plane with the orientation D(m) as the normal, and the depth map is updated at the same time. The depth map records the depth of the point closest to the projection plane for each pixel.

[0098] 4. Then, an approximate version of CC(m) (denoted as C'(m)) is constructed by associating the corresponding updated pixels of the depth map with the 3D points with the same depth. Let PC' be the point cloud obtained by union of the reconstructed connected components {CC'(0), CC'(1), ..., CC'(M-1)}.

[0099] 5. Note that the projective reconstruction process can be lossy and may miss some points. To detect such points, each point P(i) of the point cloud PC can be checked to ensure that its distance from a point in PC' is less than a parameter δ. If this is not the case, P(i) can be marked as a missing point and added to the set of missing points denoted as MP.

[0100] 6. Then, steps 2 to 5 are applied to the missing point MP. This process is repeated until the MP is empty or the CC is empty. Note that the parameters δ and α can be defined by the user or determined by applying an optimization procedure. These two parameters can also be fixed or adaptive.

[0101] 7. Filters may be applied to the detected patches to make them more suitable for compression. Example filters may include:

[0102] a. A smoothing filter (eg, median filtering) based on the geometry / texture / attributes of the points of the patch, which takes both spatial and temporal aspects into account.

[0103] b. Discard small and isolated patches.

[0104] c. User-directed filtering.

[0105] d. Other suitable smoothing filtering techniques.

[0106] Pack

[0107] In some embodiments, a depth map associated with a patch (also referred to herein as a "depth patch image" or "geometry patch image," such as those described above) can be packed into a 2D image frame. Additionally, attribute patch images and / or texture patch images can be packed into a 2D image frame. For example, a packing module (such as packing module 208) can: pack a depth patch image generated by a spatial image generation module (such as spatial image generation module 210) into a 2D image frame; pack a texture patch image generated by a texture image generation module (such as texture image generation module 212) into a 2D image frame; and pack an attribute patch image generated by an attribute image generation module (such as attribute image generation module 214) into a 2D image frame. The maps or patch images can be packed such that (A) non-overlapping blocks of T×T pixels do not contain patch information from two different patches, and (B) the size of the resulting image frame is minimized.

[0108] In some embodiments, packaging includes the following steps:

[0109] a. Sort the patches by height and then by width. Then insert the patches one by one into image frame (I) in this order. At each step, traverse the pixels of image frame (I) in raster order while checking whether the current patch can be inserted under the two conditions (A) and (B) above. If not, double the height of (I).

[0110] b. Repeat this process until all patches are inserted.

[0111] In some embodiments, the above-described packing process can be applied to pack subsets of patches within multiple tiles of a single image frame or multiple image frames. This can allow patches with similar / close orientations based on visibility from the rendering camera position to be stored in the same image frame / tile to enable view-dependent streaming and / or decoding. This can also allow for parallel encoding / decoding.

[0112] In some embodiments, the packing process can be considered as a bin packing problem, and the first fit decreasing strategy described above can be applied to solve the bin packing problem. In other embodiments, other methods can be applied in the packing process, such as a modified first fit decreasing (MFFD) strategy.

[0113] In some embodiments, if temporal prediction is used (such as described with respect to inter-frame compression encoder 250), such optimization can be performed using temporal prediction / coding in addition to spatial prediction / coding. Such considerations can be made for the entire video sequence or per group of pictures (GOP). In the latter case, additional constraints can be specified. For example, a constraint can be that the resolution of the image frame should not exceed a threshold amount. In some embodiments, even if temporal prediction is not used, additional temporal constraints can be imposed, such as ensuring that the patch corresponding to a particular object view does not move more than x pixels from the previous instance.

[0114] Figure 3B An exemplary image frame including a packed patch image and a padding portion according to some embodiments is shown. An image frame 300 includes a patch image 308 packed into the image frame 300, and also includes padding 310 in the space of the image frame 300 not occupied by the patch image. In some embodiments, padding (such as padding 310) can be determined to minimize the inconsistency between the patch image and the padding. For example, in some embodiments, padding can construct new pixel blocks that are replicas of or similar to the pixel blocks on the edge of the patch image to some extent. In some embodiments, because an image and / or video encoder can encode based on the difference between adjacent pixels, such a method can reduce the number of bytes required to encode an image frame including a patch image and padding.

[0115] In some embodiments, the patch information can be stored in the same order as used during packing, which allows for handling overlapping 2D bounding boxes of patches. Thus, a decoder receiving the patch information can extract the patch images from the image frame in the same order in which they were packed into the image frame. Additionally, because the decoder knows this order, it can resolve overlapping patch image bounding boxes.

[0116] Figure 4 An exemplary image frame 402 with overlapping patches is shown according to some embodiments. Figure 4An exemplary image frame 402 is shown with two patches (patch image 1 and patch image 2) having overlapping 2D bounding boxes 404 and 406 that overlap at region 408. To determine to which patch a T×T block in region 408 belongs, the order of the patches can be considered. For example, T×T block 404 may belong to the last patch encoded. This may be because, in the case of overlapping patches, a later-placed patch is placed so that it overlaps a previously placed patch. By knowing the placement order, the problem of the region of overlapping bounding boxes being with the most recently placed patch can be resolved. In some embodiments, the patch information is predicted and encoded (e.g., using an entropy / arithmetic encoder). Additionally, in some embodiments, the bounding box sizes U0, V0, DU0, and DV0 of the patches are encoded as multiples of T, where T is the block size used during the padding phase.

[0117] Figure 4 Also shown are blocks of image frame 402, which can be further divided into sub-blocks. For example, blocks A1, B1, C1, A2, etc. can be divided into multiple sub-blocks, and in some embodiments, the sub-blocks can be further divided into smaller blocks. In some embodiments, the video compression module of the encoder (such as video compression module 218 or video compression module 264) can determine whether the block includes active pixels, inactive pixels, or a mixture of active and inactive pixels. Compared to the amount of resources reserved for encoding blocks including active pixels, the video compression module can reserve fewer resources to compress blocks including inactive pixels. In some embodiments, active pixels can be pixels that include data about the patch image, and inactive pixels can be pixels that include padding. In some embodiments, the video compression module can subdivide blocks that include both active and inactive pixels and reserve resources based on whether the sub-blocks of the block include active pixels or inactive pixels. For example, blocks A1, B1, C1, A2 can include inactive pixels. As another example, block E3 can include active pixels, and, for example, block B6 can include a mixture of active and inactive pixels.

[0118] Exemplary segmentation process for temporally consistent patches

[0119] In some embodiments, as described above, an encoder can independently determine patches for a point cloud, where the patches are determined for each moment in time, independent of patches determined for the point cloud at other moments in time. This approach can make the patch segmentation process simpler and less computationally intensive. However, when using such an approach, a video encoder encoding packed images may not be able to take advantage of the inter-frame prediction efficiency of the video encoder's video codec. For example, by encoding motion vectors for blocks of an image (which are located in a target frame at a target moment in time relative to a reference frame at a reference moment in time), a video encoder can utilize a 2D video inter-frame prediction process to encode data in subsequent image frames relative to a reference frame. However, when using a time-independent patch segmentation process, different patches (e.g., patches of different sizes and / or shapes) may be generated for each different moment in time. Furthermore, patches may be packed into corresponding image frames in different orders or positions. This variability may hinder the video encoder's 2D video inter-frame prediction process and may result in large motion vectors and associated bit consumption, or may otherwise reduce the video encoder's compression efficiency.

[0120] In some embodiments, to address various issues associated with time-independent patch segmentation, the encoder can alternatively utilize a time-consistent patch segmentation process that ensures that patches packed into a sequence of image frames are time-consistent patches (e.g., patches have the same (or similar) size and shape for different versions of a point cloud at different time instants). The time-consistent patches can further be packed into image frames in a time-consistent order such that the time-consistent patches are placed at consistent locations in corresponding image frames of the sequence of image frames.

[0121] For example, Figure 3B Patch images 308 are shown placed in a single image frame 300, where some portions of the image frame 300 are filled with fill 310 and other portions of the image are filled with patch images 308. In such an embodiment, the encoder can determine patches for each image frame 300 independently for the same point cloud that is slightly modified at different moments in time. This can result in patches of various sizes and shapes being located at various locations in different ones of the image frames 300. In contrast, an encoder that implements temporally consistent patches can determine temporally consistent patches and pack the temporally consistent patches into the same or similar locations in multiple image frames of a sequence of image frames, where the patches have the same or similar size and shape across the image frames of the sequence. For example, as Figure 6As shown, the time-consistent patches can be packed in a time-consistent order and can be included in a sequence 600 of image frames. The sequence 600 of image frames can include patch images of the same (or similar) size and shape that are packed in the same (or similar) position. For example, fill 604 and the time-consistent patches (such as time-consistent patches 601, 602, and 603) can be located in the same (or similar) position in each image frame of the sequence 600 and can have the same (or similar) size and shape in each image frame of the sequence 600. However, the pixel values ​​of the patches that indicate geometric information or attribute information of the point cloud can vary across the time-consistent patches of the sequence 600, even if the patches have the same size and shape. As discussed in more detail below, Figure 3B The previous one of the patches 302 shown in FIG may be replaced with multiple smaller patches, such as time-consistent patches 601 and 603. In some cases, using multiple smaller patches instead of a larger patch may allow for a consistent patch across multiple moments in time.

[0122] Figure 5 An exemplary patch segmentation process for generating a time-consistent patch is shown in accordance with some embodiments.

[0123] At 510, an encoder (such as any of the encoders described herein (e.g., encoders 200, 250, etc.)) can receive a set of versions of a point cloud at multiple time instants, such as set 562 showing versions of the point cloud at times 1, 2, and 3. The encoder can then perform steps 510 through 560 to generate temporally consistent patches for the versions of the point cloud at multiple time instants. In some embodiments, these steps can constitute a decomposition into patches process, such as the decomposition into patches process 206 of encoder 200.

[0124] To generate a temporally consistent patch, at 510, the encoder may determine motion functions that model the corresponding motion of corresponding portions of a point cloud from a first version of the point cloud at a first time instant to a second (or additional) version of the point cloud at a second (or additional) time instant. For example, a motion function may be determined to model the motion of a segment of the point cloud from time 1 to time 2 and from time 2 to time 3. Additionally, at 520, the encoder may determine a mapping of points across multiple time instants based on applying a motion function to a reference version of the point cloud to generate a motion-compensated version of the point cloud; and then comparing the motion-compensated version of the point cloud to another version of the point cloud at a next time instant. For example, a motion function may be applied to a version of the point cloud at time 1 to determine a motion-compensated version compensated from time 1. This motion-compensated version may then be compared to a time 2 version of the point cloud.

[0125] In some embodiments, the encoder may overlay or otherwise compare a motion-compensated version of the point cloud from a first time instant (e.g., the version motion-compensated from time 1) with a version of the point cloud at a next time instant (e.g., the version at time 2), and identify the closest matching points between the two versions of the point cloud. For example, a nearest neighbor algorithm may be used to identify the closest point in the version at time 2 that most closely matches the motion-compensated point from the version at time 1. Based on the proximity of these points, the nearest neighbor algorithm may determine that the motion-compensated point from time 1 is actually the same point as the nearest neighbor identified in the version at time 2. This nearest neighbor matching process may be used to determine corresponding point mappings across time instants. A similar process may be applied to subsequent versions of the point cloud at subsequent time instants. For example, a similar process may be used to map points from the version at time 2 to the version at time 3, and so on. In some embodiments, a unidirectional mapping as described above may be applied (e.g., points may be mapped from time 1 to time 2 and from time 2 to time 3). In some embodiments, a bidirectional mapping may be applied. For example, points may be mapped between time 1 and time 2 such that points are mapped from time 1 to time 2 and also from time 2 to time 1. In some embodiments, a similar bidirectional mapping may be performed between time 2 and time 3.

[0126] In some embodiments, a segmented motion compensation procedure can be used to determine a motion function for segments of a point cloud between time instances. For example, the person shown in set 562 can be segmented into segments, and a motion function can be selected to describe the motion of each segment in the segments. In some embodiments, a rate-distortion optimization (RDO) process can be used to determine the degree of segmentation to be applied to the point cloud. For example, the RDO process can weigh the computational and encoding costs associated with further segmentation of the point cloud against improvements in distortion to determine the degree to which the point cloud should be segmented.

[0127] Additionally, different motion transformation functions may be applied to different segments of a point cloud to model the motion of the corresponding segments. For example, in some embodiments, some segments of a point cloud may move in a manner that is best modeled using a rigid body motion model (e.g., translation and / or rotation), while other portions of the point cloud may be best modeled using an affine motion transformation or an elastic motion transformation.

[0128] In some embodiments, a rigid-body motion transformation can model portions of a point cloud that translate or rotate but maintain their shape while moving. In some embodiments, an affine motion transformation can model portions of a point cloud that contract, expand, inflate, rotate, shear, translate, or some combination thereof, or undergo affine-like motion. For example, an affine motion transformation can be used to model a person's legs as they walk, capturing the contraction and expansion of muscles and the translation and rotation of the person's foot, calf, thigh, etc. An affine motion transformation can model motion between affine spaces that maintains points, lines, and planes, where a set of parallel lines remains parallel after applying an affine motion transformation. An affine motion transformation does not necessarily maintain angles between lines or distances between points (as in the case of rigid-body motion), but may otherwise maintain the ratio of distances between points on a line. In some embodiments, an elastic motion transformation can model the motion of portions of a point cloud that deform within certain continuity constraints. For example, when a person wearing a dress walks within certain continuity constraints (such as the elasticity of the fabric, how the fabric folds over itself, etc.), the person may cause the dress's shape to deform.

[0129] In some embodiments, a three-dimensional (3D) segmentation method (such as an octree segmentation process) can be used to determine the portion of the point cloud for which a motion model is to be selected. In some embodiments, various motion functions can be selected from a set of supported motion functions to optimally model the motion of a given segment of the point cloud. For example, the motion of some segments can be modeled using a rigid body motion transformation function that accounts for translations and rotations between versions of the segment of the point cloud at different times. While other segments can be modeled using more complex motion functions (such as an affine motion transformation or an elastic motion transformation). In some embodiments, motion functions can be determined for multiple segments of the point cloud simultaneously (e.g., in parallel).

[0130] For example, in set 562, the person is moving his left arm so that it bends at the elbow. In such cases, the encoder can determine that the arm needs to be segmented into forearm segments and elbow segments because the larger segments covering both the elbow and the forearm do not have a consistent size or shape across multiple moments. In addition, the forearm can be bent and best modeled by affine motion transformation, while the motion of the elbow can be best modeled by rigid body motion transformation. In some embodiments, the encoder can iteratively test motion transformation to select a motion transformation for modeling the motion of the segment of the point cloud. For example, the encoder can first use rigid body motion transformation to model motion and calculate the level of distortion. If the distortion is greater than a threshold value, the encoder can then use a more complex motion transformation (such as affine motion transformation or elastic motion transformation) to model motion. If the distortion increases by more than a threshold amount, the encoder can choose to model the motion of the segment via a more complex motion transformation. In some embodiments, various other motion transformations can be used.

[0131] As described above, at 520, the encoder may utilize a nearest neighbor search or other suitable algorithm to match motion-compensated points in the point cloud at one time instant to points in the point cloud at another time instant. In some embodiments, the nearest neighbor matching algorithm may utilize spatial information to map matching points. For example, the algorithm may identify points in the motion-compensated version from the previous time instant that have the shortest Euclidean distance to corresponding matching points in the next time instant version of the point cloud. In some embodiments, the nearest neighbor matching algorithm may also consider properties of these points in determining matching points between the motion-compensated version and the next time instant version. For example, the matching algorithm may identify matching points between the motion-compensated version and the next time instant version that share the same or closely related attribute values, such as the same or similar color. In some embodiments, the point matching algorithm may simultaneously consider the spatial similarity between the motion-compensated points and the next time instant points, as well as the similarity of attribute values ​​(such as the same or similar color). Additionally, in some embodiments, neighborhood characteristics of the set of points in the motion-compensated version and the next time instant version may be considered simultaneously. For example, if changes in lighting alter the property values ​​of a set of neighboring points in a similar manner, this correlated change in property values ​​can be taken into account when determining matching points between the motion compensated version and the next-time-step version.

[0132] In some embodiments, a reduced-resolution version of the point cloud may be used to perform motion estimation / modeling and point mapping for determining a temporally consistent patch. This may reduce the number of points to be matched and therefore reduce computational complexity.

[0133] At 530, the same method as above may be used to determine the Figure 3A and Figure 3AThe normal-based segmentation process is performed in a similar manner as described for element 302 of . However, instead of constraining the set of points to be included in the patch based on the normal vectors of a single version of the point cloud, the same or similar constraints may be applied to multiple versions of the points across the point cloud at multiple time instants, where the mapping determined at 520 is used to associate matching points with the various versions. For example, in a time-consistent patch segmentation process, in order to qualify for inclusion in a given patch, a point may need to have a normal vector that does not deviate from the normal vectors of the set of other points to be included in the patch by more than a threshold amount at time 1, time 2, and time 3. Because the points now have to satisfy additional constraints (e.g., constraints at more than one time instant), the patch is likely to be smaller because fewer points will satisfy the constraints at all time instants in the sequence. This can be achieved by performing a time-consistent patch segmentation on the points. Figure 3B and Figure 6 The patch shown in the figure is compared to see that Figure 6 In, from Figure 3B The patch associated with the person's left arm is replaced with two different time-consistent patches 601 and 603, where one time-consistent patch represents the elbow and the other time-consistent patch represents the forearm.

[0134] Figure 5 The set 564 shown in FIG shows an initial segmentation of the point cloud considering multiple time instants. At 550, a method similar to that described with respect to Figure 3A The iterative segmentation refinement procedure of 304 is described. However, the refinement procedure may consider multiple versions of the point cloud at multiple time instants (eg, time 1, time 2, and time 3) to generate a sequence 566 of point cloud versions having refined segments.

[0135] At 560, a final segmentation may be performed to generate a time-consistent patch for the point cloud version of the sequence 568. For example, the method described above in Figure 3A A similar process as described at 306 of FIG. Additionally, a similar packing process as described above can be applied to pack the final patches into corresponding image frames for different time instants. In addition to the temporally consistent patches, the final patches of each version of the point cloud at different time instants in the sequence can be packed into different corresponding image frames in the sequence at the same or similar positions in the corresponding image frames. Furthermore, as described above, even if the patch size, shape, and position are temporally consistent across different image frames, the points included in the temporally consistent patches can have different attribute values ​​in different image frames corresponding to different versions of the point cloud at different time instants.

[0136] Figure 6 An exemplary sequence of image frames including time-aligned patches is shown in accordance with some embodiments.

[0137] As described above, the patches of the image frames of the sequence 600 can be consistent in size, shape, and position, even though the attribute values ​​of the points included in the patches (or the depth values ​​of the geometric patches) can change. Additionally, in some embodiments, a single occupancy map can be generated and signaled for the sequence. For example, a single occupancy map can correspond to multiple image frames in the sequence of image frames that have temporally consistent patches. For example, a single occupancy map can be signaled for the image frames included in the sequence 600. In some embodiments, the occupancy map to frame sequence relationship (e.g., which image frames correspond to which occupancy map) can be signaled in a sequence header, such as can be included in a PCCNAL unit, as described below with respect to Figure 11 Described in more detail.

[0138] Figure 7 Shown are examples of versions of a point cloud at different times with different numbers of points and mappings between the points at different times, according to some embodiments.

[0139] In some embodiments, a version of a point cloud at a first moment in time may include more or fewer points than another version of the point cloud at another moment in time. Figure 7 , includes more points than the target frame 750. In such cases, the point mapping process may map more than one point in a particular version of the point cloud to a single point in a subsequent or previous version of the point cloud. For example, points 702 and 704 are both mapped to point 752. Additionally, points 706 and 708 are both mapped to point 754. In some embodiments, the encoder may alternatively or additionally resample one or more versions of the point cloud so that different versions have the same number of points or have a smaller difference in point counts.

[0140] Figure 8 A process for determining a time-consistent patch is shown according to some embodiments.

[0141] At 802 , an encoder or a decomposition-to-patch module of an encoder receives or selects a sequence of versions of a point cloud at different time instances for which a time-consistent patch is to be generated.

[0142] At 804, a motion function is determined that models the motion of a segment of the point cloud across multiple moments in time. For example, the point cloud can be segmented into octaves of an octree, and a motion function can be determined for each lowest level octave. In some embodiments, various motion transforms / functions can be used to model the motion of the octaves, such as rigid body motion transforms, affine motion transforms, elastic motion transforms, etc. In some embodiments, rate-distortion optimization or a similar process can be used to select the degree to which the point cloud is to be segmented, such as whether to segment the lowest level octave into even smaller octaves. Rate-distortion optimization can also be used to select the motion transform function to be applied to the lowest level octave. In some embodiments, a single RDO (rate-distortion optimization) process can take into account both segment size selection and segment motion transform function selection.

[0143] At 806 , the determined motion function / motion transform may be used to determine a mapping between points of the point cloud included in different versions of the point cloud at different times. For example, a nearest neighbor matching process as described above may be used.

[0144] At 808, a time-consistent patch is determined to apply a segmentation constraint to the mapped points across multiple time instants.

[0145] At 810, temporally consistent patches are packed into a set of image frames of a sequence, wherein the temporally consistent patches are packed into temporally consistent locations in the set of image frames of the sequence.

[0146] Figure 9 A process for determining segment size for motion estimation is shown according to some embodiments.

[0147] At 902, a point cloud for which a time-consistent patch is to be determined is segmented into a plurality of segments. For example, the point cloud may be segmented into octants of a first level of an octree.

[0148] At 904 , a motion function / motion transform is selected to model the motion of each of the segments of the point cloud from a reference frame (eg, a reference time instant) to a target frame (eg, a target time instant).

[0149] At 906, a distortion measure is calculated for each segment in the segment to determine the distortion of the point cloud when the motion function is applied to the determined segment, compared to the point cloud at the target time instant. In some embodiments, the distortion measure can be an overall distortion measure for the point cloud that measures the amount of distortion introduced by comparing the motion-compensated reference frame version of the point cloud with the target frame version of the point cloud. In some embodiments, the distortion can be determined / calculated at the segment level.

[0150] At 908, a determination is made as to whether the calculated distortion is greater than a distortion threshold and / or whether the rate-distortion optimization analysis recommends further segmentation. If so, then at 910, the segments of the point cloud that exceed the distortion threshold or for which further segmentation is warranted by the RDO are further segmented. For example, a quadrant of the point cloud may be further segmented into eight lower level quadrants. Each of the further segmented segments is then processed through steps 904 and 906 until the distortion threshold of 908 is met. In some embodiments, different segments of the point cloud may be segmented into smaller segments than other segments of the point cloud. For example, a threshold or RDO process may indicate that further segmentation is warranted for some segments, while further segmentation is not warranted for other segments.

[0151] At 912 , the determined motion function is applied to the determined segment of the point cloud at the reference frame to generate a motion compensated version of the reference frame point cloud.

[0152] At 914, the points of the motion compensated segment of the reference frame are mapped to points in the target frame using a nearest neighbor mapping technique. The nearest neighbor matching technique maps the points of the motion compensated version of the reference frame to the nearest neighboring points in the target frame that have the shortest Euclidean distance between the motion compensated points and the target frame points. Additionally, in some embodiments, attribute values ​​of the points may also be considered to map points between the reference frame and the target frame.

[0153] At 916, points having normal vectors that do not deviate from each other across multiple mapped versions of the point cloud at different times are grouped together to form patches.The patches may be further refined as described herein to determine a final set of patches.

[0154] Figure 10A Steps for removing outliers in determining a time-consistent patch are shown, according to some embodiments.

[0155] In some embodiments, outliers having motion across multiple moments in the sequence that varies by more than a threshold amount due to motion of surrounding points can be removed from consideration when determining a temporally consistent patch. For example, in some embodiments, an outlier can be identified at 1002 between steps 804 and 806, and the identified outlier can be removed from consideration when determining a temporally consistent patch at 1004.

[0156] Figure 10B Steps for applying a noise reduction filter when determining a time-consistent patch are shown according to some embodiments.

[0157] In some embodiments, a noise reduction filter may be applied to the point cloud between step 802 and step 804. For example, at 1006, a noise reduction filter is applied to the point cloud version of the sequence.

[0158] Figure 10C Steps for resampling one or more versions of a point cloud at different time instances in determining a time-consistent patch are shown, according to some embodiments.

[0159] In some embodiments, point resampling may be applied to the point cloud between step 802 and step 804. For example, one or more versions of the point cloud included in the sequence are resampled at 1008. The resampling may cause different versions of the point cloud in the sequence to have the same number of points or to have an even closer number of points.

[0160] filling

[0161] In some embodiments, padding may be performed to fill unoccupied pixels with values ​​so that the resulting image is suitable for video / image compression. For example, the image frame padding module 216 or the image padding module 262 may perform padding as described below.

[0162] In some embodiments, padding is applied to pixel blocks while favoring intra-prediction modes used by existing video codecs. More precisely, for each block of size B×B to be padded, the intra-prediction modes available at the video encoder side are evaluated, and the intra-prediction mode that produces the lowest prediction error on occupied pixels is retained. This can take advantage of the fact that video / image codecs typically operate on pixel blocks of predefined sizes (e.g., 64×64, 32×32, 16×16, etc.). In some embodiments, other padding techniques may include linear extrapolation, inpainting techniques, or other suitable techniques.

[0163] Video Compression

[0164] In some embodiments, a video compression module (such as video compression module 218 or video compression module 264) may perform video compression as described below.

[0165] In some embodiments, a video encoder can utilize an occupancy map, which describes, for each pixel of an image, whether the pixel stores information belonging to a point cloud or a filler pixel. In some embodiments, such information can allow for the adaptive enabling of various features, such as deblocking, adaptive loop filtering (ALF), or shape adaptive offset (SAO) filtering. Furthermore, such information can allow a rate control module to adapt to blocks containing occupancy map edges and assign different (e.g., lower) quantization parameters (QPs) to these blocks, which are essentially different numbers of bits. Coding parameters such as Lagrange multipliers, quantization thresholding, and quantization matrices can also be adjusted based on the characteristics of the point cloud projection blocks. In some embodiments, such information can also enable rate-distortion optimization (RDO) and rate control / allocation to utilize the occupancy map to account for distortion based on unfilled pixels. In a more general form, the weighting of distortion can be based on the "importance" of each pixel to the point cloud geometry. Importance can be based on a variety of aspects, such as proximity to other point cloud samples, sample directionality / orientation / position, and so on. For example, forward-facing samples may have a higher weight in the distortion calculation than backward-facing samples. Distortion may be calculated using metrics such as mean squared or absolute error, but different distortion metrics may also be considered, such as SSIM, VQM, VDP, Hausdorff distance, etc.

[0166] Additionally, in some embodiments, such encoders and decoders as described herein or other encoders and decoders may be applicable to other visual volumetric content, including meshes, three degrees of freedom plus (3DOF+) scenes or alternatively referred to in some contexts as MPEG MIV material, light fields, or other types of six degrees of freedom (6DOF) content.

[0167] Example bitstream structure for compressed point cloud data

[0168] In some embodiments, the relationship information of the patch images in the image frame may be included in or derived from the bitstream of the compressed point cloud. Figure 11 A bitstream structure for compressing a point cloud according to some embodiments is shown. In some embodiments, the bitstream structure may include a stream header and multiple groups of frame streams. In some embodiments, each group of frame streams may include a header for a group of frames and a group of frames of geometry image frames, texture / attribute image frames, and auxiliary information / occupancy maps. In some embodiments, a time-consistent patch may be used so that the occupancy map applies to multiple geometry / attribute / texture image frames describing a point cloud at multiple moments in time (e.g., 3D point cloud frames). In some embodiments, a header for a group of frames and / or a header for a stream may indicate which occupancy map applies to which set of 3D point cloud frames represented by the geometry and attribute / texture 2D image frames.

[0169] In some embodiments, the auxiliary information may include relationship information for the patch images. Additionally, in some embodiments, an occupancy map may include relationship information for the patches of an image frame. For example, an occupancy map may indicate which portions of an image frame are occupied or unoccupied. Additionally, the auxiliary information may indicate which blocks of an image frame correspond to which patches. This information may be used to determine portions of an image frame that correspond to the same patch. Additionally, depth information included in geometric information (e.g., depth patch images) may be used to identify portions of an image frame that have points in a point cloud that share a common depth. Additionally, attribute / texture information included in a texture video stream may be used to identify patches in an image frame that have similar textures or attribute values.

[0170] Example methods for compressing and decompressing point clouds

[0171] Figure 12A A process for compressing attribute information and spatial information of a point cloud is shown according to some embodiments.

[0172] At 1202, a point cloud is received by an encoder. The point cloud may be captured, for example, by one or more sensors, or may be generated, for example, in software.

[0173] At 1204 , compressed point cloud information is determined using any technique described herein or using one or more combinations of the techniques described herein.

[0174] At 1206, the compressed point cloud is encoded using the compressed point cloud information determined at 1204. The point cloud may be compressed using any of the techniques described herein.

[0175] Figure 12B A process for decompressing attribute information and spatial information of a point cloud is shown according to some embodiments.

[0176] At 1203, an encoded point cloud is received. The point cloud may have been encoded using any of the encoding techniques described herein, such as patch images that are packed into image frames and then encoded by a video encoder. In some embodiments, the encoded point cloud may include a projection of the point cloud, such as onto a cube, cylinder, sphere, etc., which is then encoded via the video encoder.

[0177] At 1205, spatial and attribute information for the encoded point cloud is determined. For example, a video decoder may be used to decode a packed image or projection of the video encoding. Spatial information may then be determined based on the packed image or projection, and the spatial information may be combined to determine spatial information for the points of the point cloud. For example, depth information for a point in a patch may be matched with X and Y information for the point in the patch to determine spatial information for the point in the patch in 3D space. In a similar manner, other attributes included in the patch image (such as color attributes, texture attributes, etc.) may be matched with corresponding points to determine attribute values ​​for the point. Additionally, in the case of multiple projections, the same point may be identified in more than one projection to determine spatial information for the point in 3D space.

[0178] At 1207 , the decompressed point cloud may be provided to a recipient device or module.

[0179] Figure 12C Patch images are shown being generated and packed into image frames to compress attribute information and spatial information of a point cloud, according to some embodiments.

[0180] At 1210, a patch is determined for a portion of a point cloud. For example, the patch can be determined as described above. At 1225, patch information for the patch can be generated, and at 1226, the patch information can be encoded to be sent to a decoder. In some embodiments, the encoded patch information can be encoded separately from one or more image frames comprising the packed patch image.

[0181] At 1211, a first patch (or a next patch is selected). At 1212, a color (e.g., attribute) patch image is generated for the points of the point cloud included in the patch. At 1214, one or more additional attribute images, such as texture attribute images, are generated for the points of the point cloud included in the patch.

[0182] At 1213, a spatial information image is generated for the points of the point cloud included in the patch. In some embodiments, to generate the spatial information image, at 1215, the points of the point cloud are projected onto a patch plane that is perpendicular to a normal vector that is orthogonal to the surface of the point cloud at the patch location. At 1217, a first spatial image is generated for the patch based on the projection of the points onto the patch plane at 1215. Additionally, at 1216, depth information for the points of the patch relative to the patch plane is determined, and at 1218, a depth patch image is generated based on the depth information determined at 1216.

[0183] At 1219, a determination is made as to whether additional patches are to be generated for patch images. If so, the process returns to 1211 for the next patch. If not, at 1220, the patch images for the patch are packed into one or more image frames. In some embodiments, the patch images for the corresponding patch may be packed before patch images for other patches are determined. At 1221, an occupancy map is generated based on the positions of the patch images when they are packed into the one or more image frames. At 1224, the occupancy map is encoded. As described above, in some embodiments, the occupancy map may be encoded using an arithmetic encoder, an entropy encoder, or the like. Furthermore, in some embodiments, the occupancy map may be encoded using a video encoder, where the occupancy map is organized as an additional image frame that corresponds to the patch image frame and represents the portion of the patch image frame occupied by the patch image (e.g., occupied pixels) and the portion of the patch image frame that is filled (e.g., unoccupied pixels).

[0184] At 1222, space in one or more image frames not occupied by patch images is filled. In some embodiments, an occupancy map for the patch image frames may be generated before or after filling the patch image frames at 1222.

[0185] At 1223 , the one or more image frames are video encoded, such as according to the High Efficiency Video Coding (HEVC) standard. In some embodiments where the occupancy map is represented by an occupancy map video image frame, the occupancy map video image frame may be video encoded at 1223 .

[0186] Figure 12D Patch images are shown being generated and packed into image frames to compress attribute information and spatial information of a moving or changing point cloud, according to some embodiments.

[0187] At 1230, point cloud information for a previously encoded point cloud is received, wherein the point cloud information represents a subsequent version of the previously encoded point cloud. For example, the subsequent version may be a representation of the point cloud at a subsequent time, wherein the point cloud has moved or changed over time.

[0188] At 1231, a determination is made as to whether any new patches need to be determined for the point cloud. For example, an object that is not currently in a previously encoded point cloud may have been added to the point cloud. For example, the point cloud may be that of a road, and a ball may have entered the road. If a new patch needs to be added, at 1233, the occupancy map is updated to include the new patch, and at 1234, the occupancy map is encoded. Additionally, at 1232, a patch image is generated for the new patch in a manner similar to that described in steps 1212 through 1214. The generated patch image is included in the package at 1243.

[0189] At 1235, the first or next patch of the patches generated for the reference (previous) point cloud is selected. At 1236, the points of the patch are resampled as described herein. At 1237, motion vectors for points in the selected patch between the reference point cloud and the current point cloud are determined. At 1240, a relative motion patch image is generated using the motion vectors. For example, in some embodiments, generating the relative motion patch image may include encoding vector motion in different directions using different image characteristics at 1241, as described herein. At 1238, an updated color patch image is generated. In some embodiments, the updated color patch image may encode residual values ​​that indicate the difference in color of the points in the patch between the reference point cloud and the current point cloud. In a similar manner, other attribute update patch images may be generated at 1239.

[0190] At 1242, a determination is made as to whether there are additional patches to be evaluated. If so, the process returns to 1235 for the next patch. If not, at 1243, the patch images for the patch are packed into one or more image frames. In some embodiments, the patch images for the respective patches may be packed before determining the patch images for other patches.

[0191] At 1244 , space in one or more image frames not occupied by patch images is filled.

[0192] At 1245, the one or more image frames are video encoded, such as according to the High Efficiency Video Coding (HEVC) standard.

[0193] Figure 12E A decoder according to some embodiments is shown that receives an image frame including patch images, patch information, and an occupancy map and generates a decompressed representation of a point cloud.

[0194] At 1250, an occupancy map is received by the decoder, and at 1251, patch information is received by the decoder. In some embodiments, the occupancy map and patch information may be encoded, and the decoder may decode the occupancy map and patch information (not shown). At 1252, the decoder receives one or more encoded video image frames. At 1252, the decoder identifies patch images in the one or more encoded video image frames, and at 1254, the decoder decodes the encoded video image frames. In some embodiments, the decoder may use the occupancy map and patch information to identify active and inactive portions of the one or more encoded video images, and may adjust one or more decoding parameters used to decode the encoded video images based on whether the portion (e.g., block, sub-block, pixel, etc.) includes activity information or inactivity information.

[0195] At 1255 , the decoder determines spatial information and / or attribute information for points of the corresponding patch and, at 1256 , generates a decompressed representation of the point cloud encoded in the one or more encoded video images.

[0196] Sample application using point cloud encoder and decoder

[0197] Figure 13 Illustration of a compressed point cloud being used in a 3D application according to some embodiments.

[0198] In some embodiments, a sensor (such as sensor 102), an encoder (such as encoder 104 or any other encoder described herein), and a decoder (such as decoder 116 or any decoder described herein) may be used to transmit a point cloud in a 3D application. For example, at 1302, a sensor (such as sensor 102) may capture a 3D image, and at 1304, the sensor or a processor associated with the sensor may perform 3D reconstruction based on the sensed data to generate a point cloud.

[0199] At 1306, an encoder (such as encoder 104) may compress the point cloud, and at 1308, the encoder or post-processor may package the compressed point cloud and transmit it via network 1310. At 1312, the data packet may be received at a destination location including a decoder (such as decoder 116). At 1314, the decoder may decompress the point cloud, and at 1316, the decompressed point cloud may be rendered. In some embodiments, the 3D application may transmit the point cloud data in real time, so that the display at 1316 may represent the image being observed at 1302. For example, at 1316, a camera in a canyon may allow a remote user to experience walking through a virtual canyon.

[0200] Figure 14 Illustration of a compressed point cloud being used in a virtual reality (VR) or augmented reality (AR) application, according to some embodiments.

[0201] In some embodiments, the point cloud may be generated in software (e.g., as opposed to being captured by a sensor). For example, at 1402, virtual reality or augmented reality content is generated. The virtual reality or augmented reality content may include point cloud data and non-point cloud data. For example, as an example, a non-point cloud character may traverse terrain represented by a point cloud. At 1404, the point cloud data may be compressed, and at 1406, the compressed point cloud data and the non-point cloud data may be packaged and transmitted via a network 1408. For example, the virtual reality or augmented reality content generated at 1402 may be generated at a remote server and transmitted to a VR or AR content consumer via the network 1408. At 1410, the data packets may be received and synchronized at the VR or AR consumer's device. At 1412, a decoder operating at the VR or AR consumer's device may decompress the compressed point cloud, and the point cloud and non-point cloud data may be rendered in real time, for example, in a head-mounted display of the VR or AR consumer's device. In some embodiments, point cloud data may be generated, compressed, decompressed, and rendered in response to a VR or AR consumer manipulating a head-mounted display to look in different directions.

[0202] In some embodiments, point cloud compression as described herein can be used in various other applications such as geographic information systems, live sports broadcasts, museum displays, autonomous navigation, etc.

[0203] Exemplary Computer System

[0204] Figure 15 An exemplary computer system 1500 is shown that can implement an encoder or decoder or any other of the components described herein (e.g., as described above with reference to FIG. 1 ). Figures 1 to 14 ). Computer system 1500 can be configured to perform any or all of the embodiments described above. In various embodiments, computer system 1500 can be any of various types of devices, including, but not limited to, a personal computer system, a desktop computer, a laptop computer, a notebook computer, a tablet computer, an all-in-one computer, a tablet or netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a camera, a set-top box, a mobile device, a consumer device, a video game controller, a handheld video game device, an application server, a storage device, a television, a video recording device, a peripheral device (such as a switch, a modem, a router), or generally any type of computing or electronic device.

[0205] The various embodiments of the point cloud encoder or decoder described herein may be executed on one or more computer systems 1500, which may interact with various other devices. Figures 1 to 14Any component, action or function described may be configured as Figure 15 1500. In the illustrated embodiment, the computer system 1500 includes one or more processors 1510 coupled to a system memory 1520 via an input / output (I / O) interface 1530. The computer system 1500 also includes a network interface 1540 coupled to the I / O interface 1530, and one or more input / output devices 1550, such as a cursor control device 1560, a keyboard 1570, and a display 1580. In some cases, it is contemplated that the embodiments may be implemented using a single instance of the computer system 1500, while in other embodiments, multiple such systems or multiple nodes comprising the computer system 1500 may be configured to host different portions or instances of the embodiments. For example, in one embodiment, some elements may be implemented via one or more nodes of the computer system 1500 that are different from those that implement other elements.

[0206] In various embodiments, computer system 1500 can be a uniprocessor system including one processor 1510, or a multiprocessor system including several processors 1510 (e.g., two, four, eight, or another suitable number). Processor 1510 can be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1510 can be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each of processors 1510 can typically, but not necessarily, implement the same ISA.

[0207] System memory 1520 can be configured to store point cloud compression or point cloud decompression program instructions 1522 and / or sensor data accessible by processor 1510. In various embodiments, system memory 1520 can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions 1522 can be configured to implement an image sensor control application that incorporates any of the functionality described above. In some embodiments, program instructions and / or data can be received, sent, or stored on different types of computer-accessible media separate from system memory 1520 or computer system 1500, or on similar media. Although computer system 1500 is described as implementing the functionality of the functional blocks of the preceding figures, any functionality described herein can be implemented by such a computer system.

[0208] In one embodiment, I / O interface 1530 may be configured to coordinate I / O communications between processor 1510, system memory 1520, and any peripheral devices in the device (including network interface 1540 or other peripheral device interfaces, such as input / output devices 1550). In some embodiments, I / O interface 1530 may perform any necessary protocol, timing, or other data conversion to convert data signals from one component (e.g., system memory 1520) into a format suitable for use by another component (e.g., processor 1510). In some embodiments, I / O interface 1530 may include support for devices attached, for example, via various types of peripheral buses (such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, the functionality of I / O interface 1530 may be divided into two or more separate components, such as a northbridge and a southbridge, for example. Furthermore, in some embodiments, some or all of the functionality of I / O interface 1530 (such as the interface to system memory 1520) may be incorporated directly into processor 1510.

[0209] The network interface 1540 can be configured to allow data to be exchanged between the computer system 1500 and other devices (e.g., carriers or proxy devices) attached to the network 1585, or between nodes of the computer system 1500. In various embodiments, the network 1585 can include one or more networks, including but not limited to a local area network (LAN) (e.g., an Ethernet or an intranet), a wide area network (WAN) (e.g., the Internet), a wireless data network, some other electronic data network, or some combination thereof. In various embodiments, the network interface 1540 can support communication over a wired or wireless general data network (such as any suitable type of Ethernet network), for example; communication via a telecommunications / telephone network (such as an analog voice network or a digital fiber optic communication network); communication via a storage area network (such as a Fibre Channel SAN), or communication via any other suitable type of network and / or protocol.

[0210] In some embodiments, input / output devices 1550 may include one or more display terminals, keyboards, keypads, trackpads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or accessing data by one or more computer systems 1500. Multiple input / output devices 1550 may be present in computer system 1500 or may be distributed across various nodes of computer system 1500. In some embodiments, similar input / output devices may be separate from computer system 1500 and may interact with one or more nodes of computer system 1500 through a wired or wireless connection, such as through network interface 1540.

[0211] like Figure 15 As shown, memory 1520 may include program instructions 1522, which may be executable by the processor to implement any of the elements or actions described above. In one embodiment, the program instructions may execute the method described above. In other embodiments, different elements and data may be included. It should be noted that the data may include any of the data or information described above.

[0212] Those skilled in the art will appreciate that computer system 1500 is merely illustrative, and is not intended to limit the scope of the embodiments. Specifically, computer system and equipment can include any combination of hardware or software that can perform the functions indicated, including computers, network equipment, internet equipment, personal digital assistants, wireless phones, pagers, etc. Computer system 1500 can also be connected to other devices not shown, or can be operated as an independent system on the contrary. In addition, the functions provided by the shown components can be combined in fewer components or be distributed in additional components in some embodiments. Similarly, in some embodiments, the functions of some components in the shown components may not be provided, and / or other additional functions may be available.

[0213] Those skilled in the art will also recognize that, although various items are shown as being stored in memory or on storage devices during use, for the purpose of memory management and data integrity, these items or parts thereof can be transmitted between memory and other storage devices. Alternatively, in other embodiments, some or all of these software components can be executed in the memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures can also be stored on a computer-accessible medium or portable article (for example, as instructions or structured data) to be read by a suitable driver, and its various examples are described above. In some embodiments, the instructions stored on the computer-accessible medium separated from the computer system 1500 can be transferred to the computer system 1500 via a transmission medium or signal (such as an electrical signal, an electromagnetic signal or a digital signal transmitted via a communication medium such as a network and / or a wireless link). Various embodiments may also include receiving, sending or storing instructions and / or data implemented according to the above description on a computer-accessible medium. Generally speaking, computer-accessible media may include non-transitory computer-readable storage media or memory media, such as magnetic or optical media, for example, disks or DVD / CD-ROMs, volatile or non-volatile media, such as RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc. In some embodiments, computer-accessible media may include transmission media or signals, such as electrical signals, electromagnetic signals, or digital signals transmitted via a communication medium, such as a network and / or a wireless link.

[0214] In different embodiments, the methods described herein can be implemented in software, hardware, or a combination thereof. In addition, the order of the method blocks can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For those skilled in the art who benefit from this disclosure, it is obvious that various modifications and changes can be made. The various embodiments described herein are intended to be illustrative and not restrictive. There can be many variations, modifications, additions, and improvements. Therefore, multiple examples can be provided for the components described as a single example in this article. There are slightly arbitrary boundaries between the various components, operations, and data storage devices, and specific operations are shown in the context of a specific exemplary configuration. Other allocations of functions are contemplated and may fall within the scope of the appended claims. Finally, the structure and function presented as discrete components in the exemplary configuration may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of the embodiments as defined in the following claims.

Claims

1. A non-transitory computer-readable medium storing program instructions that, when executed by one or more processors, cause the one or more processors to: determining a plurality of patches for a set of visual volumetric content comprising a point cloud, wherein each patch comprises a point of the point cloud projected onto a patch plane, wherein to determine the plurality of patches, the program instructions cause the one or more processors to: determining one or more motion functions that model motion of one or more segments of the point cloud across a plurality of moments in time; determining a mapping between points of the point cloud across the plurality of time instants based on the one or more motion functions; as well as determining one or more patches comprising points of the point cloud having surface normal vectors that deviate from one another by less than a threshold amount at each of the plurality of moments in time; as well as Patch images of the determined one or more patches are packed into a plurality of image frames, wherein the plurality of image frames include consistent patches of the point cloud across the plurality of time instants.

2. The non-transitory computer-readable medium of claim 1 , wherein the program instructions cause the one or more processors to: The patch images at consistent locations are packed into the plurality of image frames for the plurality of time instants.

3. The non-transitory computer-readable medium of claim 1 , wherein to determine the mapping between points of the point cloud across the plurality of time instants, the program instructions cause the one or more processors to: applying the determined one or more motion functions to a first set of points at a first time instant to determine estimated positions of the points at one or more other time instants in the plurality of time instants starting from the first time instant; and For each of the one or more other moments in time, a nearest neighbor point of the point cloud at the one or more other moments in time is identified, the nearest neighbor point corresponding to the estimated position of the point of the point cloud estimated from the first moment in time. 4 . The non-transitory computer-readable medium of claim 1 , wherein the program instructions cause the one or more processors to determine different motion functions for different ones of the segments of the point cloud.

5. The non-transitory computer-readable medium of claim 1 , wherein the one or more motion functions include one or more of: Rigid body motion transformation function; Affine motion transformation function; or Elastic motion transformation function.

6. The non-transitory computer-readable medium of claim 1, wherein the one or more motion functions are determined based on spatial coordinates of the points of the point cloud across the multiple moments in time and attribute values ​​of the points of the point cloud across the multiple moments in time.

7. The non-transitory computer-readable medium of claim 1 , wherein the program instructions further cause the one or more processors to: Prior to determining the one or more motion functions, a noise reduction filter is applied to data representing the point cloud across the plurality of time instants.

8. The non-transitory computer-readable medium of claim 1 , wherein the program instructions further cause the one or more processors to: Prior to determining the one or more patches: identifying one or more outlier points of the point cloud having motion that deviates from surrounding points across the plurality of time instants; and The one or more outliers are removed from the point cloud data used to determine the one or more patches.

9. The non-transitory computer-readable medium of claim 1 , wherein the program instructions further cause the one or more processors to: An occupancy map is generated that indicates locations of the patch images in the image frames for the plurality of time instances.

10. The non-transitory computer-readable medium of claim 1 , wherein the program instructions further cause the one or more processors to: In a sequence header of a sequence of image frames, an occupancy map to be applied to the image frames comprised in the sequence of image frames is indicated.

11. An apparatus for determining a time-consistent patch, the apparatus comprising: a memory storing program instructions for compressing three-dimensional (3D) point cloud data using two-dimensional (2D) image frames; and one or more processors, wherein the program instructions, when executed by the one or more processors, cause the one or more processors to: determining a plurality of patches for a set of visual volumetric content comprising a point cloud, wherein each patch comprises a point of the point cloud projected onto a 2D patch plane, wherein to determine the plurality of patches, the program instructions cause the one or more processors to: determining one or more motion functions that model motion of one or more segments of the point cloud across a plurality of moments in time; determining a mapping between points of the point cloud across the plurality of time instants based on the one or more motion functions; as well as generating one or more patches comprising points of the point cloud whose normal vectors deviate from each other by less than a threshold amount at each of the plurality of time instants; as well as The patch images of the determined one or more patches are packed into a plurality of 2D image frames.

12. The device according to claim 11, further comprising: One or more sensors configured to capture spatial information and attribute information of the points of the point cloud.

13. The apparatus of claim 12, wherein the patch images packed into the plurality of 2D image frames comprise: Patch image of geometric information of each patch; as well as One or more attribute information patch images for each patch.

14. The apparatus of claim 13 , wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: The plurality of 2D image frames are video encoded to generate a compressed video representation of the captured point cloud.

15. The apparatus of claim 11 , wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: Prior to determining the one or more motion functions, a noise reduction filter is applied to data representing the point cloud across the plurality of time instants.

16. The apparatus of claim 11 , wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: Prior to determining the one or more patches: identifying one or more outlier points of the point cloud having motion that deviates from surrounding points by an amount greater than an outlier threshold across the plurality of time instants; and The one or more outliers are removed from the point cloud data used to determine the one or more patches.

17. The apparatus of claim 11, wherein the one or more patches are determined based on identifying points having surface normal vectors that deviate from each other by less than the threshold amount at each of the plurality of moments in time.

18. A system for determining a time-consistent patch, the system comprising: One or more computers, the one or more computers configured to: generating a visual volumetric content comprising a point cloud, the point cloud comprising a plurality of points located in a three-dimensional (3D) space, wherein corresponding points among the points comprise spatial information of the point and attribute information of the point; as well as Compressing the generated point cloud into a plurality of video images including packed patch images, wherein to determine patches used to generate the patch images, the one or more computers are configured to: determining one or more motion functions that model motion of one or more segments of the point cloud across a plurality of moments in time; determining a mapping between points of the point cloud across the plurality of time instants based on the one or more motion functions; as well as The patch is generated, wherein the patch includes points of the point cloud whose normal vectors at each of the plurality of moments in time deviate from each other by less than a threshold amount.

19. The system of claim 18, wherein to determine the one or more motion functions, the one or more computers are configured to: Segmenting the point cloud generated at the first moment into a plurality of segments, each segment including one or more points of the point cloud in 3D space; determining a motion compensation function to apply to the segment at the first time instant to model the motion of the point included in the segment at the first time instant to a target time instant; determining a position difference between the position of the point determined using the motion compensation function and the actual position of the point of the point cloud at the target time; and In response to the determined positional differences exceeding one or more thresholds, further segmenting one or more of the segments, determining a motion compensation function for the further segmented one or more segments, and determining positional differences for points of the further segmented one or more segments, wherein the one or more computers are configured to further segment segments of the point cloud until the positional differences are less than the one or more thresholds.

20. The system of claim 18, wherein prior to generating the patch for the point cloud, the one or more computers are configured to: identifying one or more versions of the point cloud at one or more respective moments in time, the one or more versions comprising a greater or lesser number of points than other versions of the point cloud at other moments in time; and At least one of the versions of the point cloud is resampled such that the at least one resampled version of the point cloud includes the same number of points as the other versions of the point cloud.

Images