Point cloud encoding using homography

Abstract

Point cloud encoding methods using homography transforms send the homography transform of 3D patches instead of explicit projection values ​​(such as bounding boxes and patch orientation and rotation). This method has a more compact representation, is more efficient in transmission, and allows faster decoding, especially in cases where 3D points will be reprojected.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority under 35 USC §119(e) to U.S. Provisional Patent Application Serial No. 62 / 794,091, filed on January 18, 2019, entitled "POINT CLOUD CODING USING HOMOGRAPHY TRANSFORM," which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present invention relates to three-dimensional graphics. More particularly, the present invention relates to encoding of three-dimensional graphics. Background Art

[0004] Point clouds have been considered as a candidate transmission format for 3D data either captured by 3D scanners, LIDAR sensors, or used in popular applications such as virtual reality / augmented reality (VR / AR). A point cloud is a set of points in 3D space. In addition to the spatial position (X, Y, Z), each point typically has associated attributes such as color (R, G, B), or even reflectivity and a timestamp (for example, in LIDAR images). In order to obtain a high-fidelity representation of the target 3D object, the device captures a point cloud of about thousands or even millions of points. In addition, for dynamic 3D scenes used in VR / AR applications, each frame typically has a unique dense point cloud, which results in the transmission of millions of point clouds per second. For the feasible transmission of such large amounts of data, compression is typically used.

[0005] In 2017, MPEG published a call for proposals (CfP) for compression of point clouds. After evaluating several proposals, MPEG is considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods), or 3D to 2D projection followed by traditional video coding. In the case of dynamic 3D scenes, MPEG is using a test model software (TMC2) based on patch surface modeling, projection of patches from 3D images to 2D images, and encoding of 2D images with video codecs such as HEVC. This approach has been shown to be more efficient than native 3D coding and is able to achieve competitive bitrates at acceptable quality.

[0006] When encoding a point cloud, TMC2 encodes auxiliary information related to the patch projection, such as the patch position and bounding box size in the 2D canvas image. For the temporal encoding of the auxiliary information, patch matches between patches from the current point cloud and patches from the immediately decoded point cloud are used for prediction. The process is limited to adjacent frames and involves delta encoding all frames in the sequence. Summary of the invention

[0007] Point cloud encoding methods using homography transforms send the homography transform of 3D patches instead of explicit projection values ​​(e.g., bounding box and patch orientation, rotation). This method has a more compact representation, is more efficient in transmission, and allows faster decoding, especially when the 3D points will be reprojected.

[0008] In one aspect, a method programmed in a non-transitory memory of a device includes acquiring three-dimensional content and encoding the three-dimensional content using a homography. Encoding the three-dimensional content using a homography includes converting the three-dimensional content to two-dimensional content. Encoding the three-dimensional content using a homography includes transmitting homography coefficients of a three-dimensional to two-dimensional transformation. Encoding the three-dimensional content using a homography includes calculating a homography matrix. The homography matrix is ​​calculated from three matrices. The three matrices include an observation matrix, a projection matrix, and a viewport matrix. The observation matrix represents a camera, the projection matrix represents pixels in a sensor of the camera, and the viewport matrix represents a transformation of pixels in the camera to pixels in an image.

[0009] In another aspect, a device includes: a non-temporary memory for storing an application, the application for: acquiring three-dimensional content and encoding the three-dimensional content using a homography; and a processor coupled to the memory, the processor configured to process the application. Encoding the three-dimensional content using a homography includes converting the three-dimensional content into two-dimensional content. Encoding the three-dimensional content using a homography includes transmitting homography coefficients of a three-dimensional to two-dimensional transformation. Encoding the three-dimensional content using a homography includes calculating a homography matrix. The homography matrix is ​​calculated from three matrices. The three matrices include an observation matrix, a projection matrix, and a viewport matrix. The observation matrix represents a camera, the projection matrix represents pixels in a sensor of the camera, and the viewport matrix represents a transformation of pixels in the camera to pixels in an image.

[0010] In another aspect, a system includes one or more cameras for acquiring three-dimensional content, and an encoder for encoding the three-dimensional content using a homography. Encoding the three-dimensional content using a homography includes converting the three-dimensional content into two-dimensional content. Encoding the three-dimensional content using a homography includes transmitting homography coefficients of the three-dimensional to two-dimensional transformation. Encoding the three-dimensional content using a homography includes calculating a homography matrix. The homography matrix is ​​calculated from three matrices. The three matrices include an observation matrix, a projection matrix, and a viewport matrix. The observation matrix represents the camera, the projection matrix represents the pixels in the sensor of the camera, and the viewport matrix represents the transformation of pixels in the camera to pixels in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flow chart illustrating a method of point cloud mapping in accordance with some embodiments.

[0012] Figure 2 A diagram illustrating projections used for segmentation in accordance with some embodiments.

[0013] Figure 3 Illustrated is a representation of encoding data in accordance with some embodiments.

[0014] Figure 4 A diagram illustrating patch generation based on camera projection in accordance with some embodiments.

[0015] Figure 5 A diagram illustrating how a homography transformation is constructed in accordance with some embodiments.

[0016] Figure 6 A flow chart illustrating a graphics pipeline implementing point cloud encoding using a homography transformation method in accordance with some embodiments.

[0017] Figure 7 An exemplary V-PCC bitstream structure is illustrated in accordance with some embodiments.

[0018] Figure 8 Illustrate an example patch frame layer unit syntax in accordance with some embodiments.

[0019] Fig. 9 Illustrate an example patch frame header syntax in accordance with some embodiments.

[0020] Fig.10 Illustrate an example patch frame data unit syntax in accordance with some embodiments.

[0021] Fig.11 Illustrate an example patch information data syntax in accordance with some embodiments.

[0022] Fig.12Illustrate an example patch data unit syntax in accordance with some embodiments.

[0023] Fig.13 Illustrate delta patch data unit syntax in accordance with some embodiments.

[0024] Fig.14 Illustrate an example PCM patch data unit syntax in accordance with some embodiments.

[0025] Fig.15 Illustrate an example homography patch data unit syntax in accordance with some embodiments.

[0026] Fig.16 Illustrate a matrix used to reconstruct a point cloud from homography patches in accordance with some embodiments.

[0027] Fig.17 Illustrated is a block diagram of an example computing device configured to implement point cloud encoding using homography transformations in accordance with some embodiments. DETAILED DESCRIPTION

[0028] Existing techniques for point cloud compression using video encoders represent point clouds as 3D patches and encode a 2D image formed by the projection of geometry and attributes into a 2D canvas. The mapping of 3D points to 2D points is performed taking into account several different options, such as direction planes and patch orientation. Currently, point cloud compression standards encode all these options in an auxiliary data structure and send it to the decoder, which reconstructs the 3D points. Since these points will be used for rendering, another transformation will be applied to these points to reproject them. The method described in this paper directly transmits the homography transformation, which can achieve fast rendering of point clouds. In addition, this method is more compact in terms of compression efficiency.

[0029] Compared to previous point cloud compression schemes, the method described in this paper sends the homography transformation of the 3D patch instead of explicit projection values ​​(e.g., bounding box and patch orientation, rotation). This method has a more compact representation, is more efficient in transmission, and allows faster decoding, especially in cases where the 3D points will be reprojected (e.g., when generating video in a mobile phone).

[0030] In this paper, we describe a new method for encoding 3D point cloud data into 2D surfaces. The method uses 3D surface patches to represent the point cloud and transfers a series of homography transformations of 3D points to 2D patch surfaces.

[0031] In 3D point cloud encoding using a video encoder, projections from 3D to 2D are used to generate a video that will represent the point cloud. The most efficient way to generate these videos is to use 3D patches that segment the surface of the object and use orthogonal projections to generate segmented depth images that are bundled together and used as input to the video encoder. However, some auxiliary information that indicates how to reconstruct the projected points in 3D is transmitted along with the video images. Currently, values ​​such as bounding boxes and patch orientation and displacement are explicitly sent in the bitstream.

[0032] As described herein, auxiliary data used to reproject points from 2D to 3D is represented as a homography. Homography is widely used in projective geometry to define projections between 3D points and 2D faces. By sending a projection matrix, the auxiliary data representation can be encoded more efficiently. In addition, the homography can be used in conjunction with other homographies to reproject points directly from 2D patches to observation projections, avoiding the high cost operations associated with current 2D to 3D projections using exact values.

[0033] Figure 1 A flowchart illustrating a method of point cloud mapping according to some embodiments is illustrated. A 3D point cloud 100 is obtained (e.g., captured by a camera / camcorder, received from another device, downloaded). In step 110, normals are estimated for the 3D point cloud. Estimating normals includes establishing a kd tree, checking the local neighborhood (e.g., k nearest neighbors), calculating the best approximate tangent plane (using covariance analysis), achieving normal smoothing, and determining the normal orientation (minimum spanning tree propagation). In step 112, initial segmentation is achieved. Initial segmentation includes classifying points according to 6 orientations (+X, -X, +Y, -Y, +Z, -Z). Initial segmentation classifies points according to a fixed orientation. After classification, the orientation is projected onto a plane. Once a cluster of points is determined, the points are projected onto the same plane. Projecting these points defines a bounding box of these points. In step 114, the segmentation is refined. Refining the segmentation includes using the neighborhood to smooth the pixel classification (e.g., maximizing a specified score). In step 116, the patch is segmented. Segmenting the patch includes calculating adjacency information, extracting patches, implementing depth filtering and locating missing points. 3D point cloud mapping results in a 2D image 102. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is changed.

[0034] Figure 2 A diagram illustrating projections for segmentation in accordance with some embodiments. Unlike previous implementations that only allow projections on surfaces parallel to the x, y, z axes, the methods described herein are able to project onto a plane at any position by using a homography. A homography (or transformation) maps a 3D point to 2D (or vice versa).

[0035] The matrix for orthographic projection can be defined by a 6-tuple (left, right, bottom, top, near, far) that defines the clipping planes. These planes form a box with the smallest corner at (left, bottom, -near) and the largest corner at (right, top, -far).

[0036] Figure 3 The diagram illustrates representations for encoding data in accordance with some embodiments. Representation 300 shows how previous implementations represent fixed projection information, which includes information such as U0, V0, size U0, and size V0 for each patch. Representation 302 is a representation that includes homography transformation information for each patch (e.g., Figure 2 By using a homography, the system does not care on which plane the bounding box is projected.

[0037] Figure 4 404。

[0038] Figure 5 A diagram illustrating how a homography is constructed in accordance with some embodiments. A homography is three matrices (e.g., observation matrix, projection matrix, and viewport matrix) that are multiplied together. The matrices are based on the camera matrices associated with each camera (e.g., R 0，0 ). Operations (e.g., the projection matrix) can move the projection by values ​​such as U1, V1, and D1. One of the matrices (e.g., the viewport matrix) determines the surface and maps it to the 2D canvas.

[0039] The observation matrix (ViewMat) represents the camera, the projection matrix (ProjMat) represents the pixels in the camera sensor, and the viewport matrix (ViewportMat) represents the transformation from pixels in the camera to pixels in the image. Homography can be applied to 3D pixels to map pixels to a 2D surface. Operations such as scaling, shearing, and perspective projection can also be implemented.

[0040] Figure 6The diagram illustrates a flow chart of a graphics pipeline for implementing point cloud encoding using a homography transformation method in accordance with some embodiments. At step 600, model (x, y, z) vertices are input. At step 602, a model view transform is applied. At step 604, a projection transform is applied. At step 606, normalization and clipping transforms are applied. At step 608, a viewport transform is applied. At step 610, rasterization is implemented. At step 612, shading is applied. At step 614, compositing is applied to generate a 2D representation. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is changed.

[0041] In some embodiments, encoding the homography transform comprises encoding each coefficient (floating point) individually using an arithmetic encoder.

[0042] Figure 7 An exemplary V-PCC bitstream structure is illustrated in accordance with some embodiments. Figure 8 Illustrate an example patch frame layer unit syntax in accordance with some embodiments. Fig. 9 Illustrate an example patch frame header syntax in accordance with some embodiments.

[0043] Fig.10 The diagram illustrates an example patch frame data unit syntax according to some embodiments. pfdu_patch_mode[frmIdx][p] indicates the patch mode for the patch with index p in the patch frame with index frmIdx. For patch frames with pfh_type[frmIdx]=I, the allowed values ​​of pfdu_patch_mode[frmIdx][p] are specified in 1000, while for patch frames with pfh_type[frmIdx]=P, the allowed values ​​of pfdu_patch_mode[frmIdx][p] are specified in 1002.

[0044] Fig.11 Illustrate an example patch information data syntax in accordance with some embodiments.

[0045] Fig.12 Illustrated is an example patch data unit syntax according to some embodiments. pdu_lod[frmIdx][p] specifies the LOD scaling factor to be applied to the patch with index p in the patch frame frmIdx. The reconstructed point 3D positions of patch p in frame frmIdx will be scaled by 2D after they are projected from 2D and before any further transformations are applied. pdu_lod[frmIdx][p] If pdu_lod[frmIdx][p] is not present, its value shall be inferred to be equal to 0.

[0046] pdu_projection_mode[frmIdx][p] equal to 0 specifies that the patch with index p in patch frame frmIdx is projected onto the near projection plane. pdu_projection_mode[frmIdx][p] equal to 1 specifies that the patch is projected onto the far projection plane. When pdu_projection_mode[frmIdx][p] is not present, its value shall be inferred to be equal to 0.

[0047] Fig.13 Illustrate delta patch data unit syntax in accordance with some embodiments.

[0048] Fig.14 The diagram illustrates an exemplary PCM patch data unit syntax according to some embodiments. ppdu_patch_in_pcm_video_flag[frmIdx][p] specifies whether the geometry and attribute data associated with the PCM coded patch p in the patch frame indexed by frmIdx are encoded in a separate video compared to the geometry and attribute data of intra- and inter-coded patches. If ppdu_patch_in_pcm_video_flag[frmIdx][p] is equal to 0, then the geometry and attribute data associated with the PCM coded patch p in the patch frame indexed by frmIdx are encoded in the same video as the geometry and attribute data of the intra- and inter-coded patches. If ppdu_patch_in_pcm_video_flag[frmIdx][p] is equal to 1, then the geometry and attribute data associated with the PCM coded patch p in the patch frame indexed by frmIdx are encoded in a separate video from the geometry and attribute data of the intra- and inter-coded patches. ppdu_pcm_points[frmIdx][p] specifies the number of pcm points present in the PCM coded patch p in the patch frame with index frmIdx. The value of ppdu_pcm_points[frmIdx][p] shall be in the range of 0 to (Patch2dSizeU[frmIdx][p]*Patch2dSizeU[frmIdx][p] / 3), inclusive.

[0049] Fig.15Illustrated is an example homography patch data unit syntax in accordance with some embodiments. hpdu_2d_shift_u[frmIdx][p] specifies the x coordinate of the upper left corner of the patch bounding box size expressed as a multiple of ops_occupancy_packing_block_size for patch p in the patch frame indexed by frmIdx. The value of hpdu_2d_shift_u[frmIdx][p] should be in the range of 0 to Min(2pfh_2d_shift_u_bit_count_minus1[frmIdx]+1-1,sps_frame_width / ops_occupancy_packing_block_size-1), inclusive. hpdu_2d_shift_v[frmIdx][p] specifies the y coordinate of the upper left corner of the patch bounding box size expressed as a multiple of ops_occupancy_packing_block_size for patch p in the patch frame indexed by frmIdx. The value of hpdu_2d_shift_v[frmIdx][p] should be in the range of 0 to Min(2pfh_2d_shift_v_bit_count_minus1[frmIdx]+1-1,sps_frame_height / ops_occupancy_packing_block_size-1) (inclusive). When p is equal to 0, hpdu_2d_delta_size_u[frmIdx][p] specifies the width of the patch with index 0 in the patch frame with index frmIdx. When p is greater than 0, hpdu_2d_delta_size_u[frmIdx][p] specifies the difference between the width of the patch with index p and the width of the patch with index (p-1). When p is equal to 0, hpdu_2d_delta_size_v[frmIdx][p] specifies the height of the patch with index 0 in the patch frame with index frmIdx. When p is greater than 0, hpdu_2d_delta_size_v[frmIdx][p] specifies the difference in height between the patch with index p and the patch with index (p-1). hpdu_orientation_swap_flag[frmIdx][p] indicates whether the patch orientation of the patch with index p of the patch frame frmIdx is used according to the table. If pdu_orientation_swap_flag[frmIdx][p] is not present, its value shall be inferred to be equal to 0. hpdu_homography_transform[frmIdx][patchIndex][i][j] specifies the coefficients [i][j] of the 4×4 homography matrix.The homography transformation allows for transformations of the patch, such as different patch orientations, different camera directions, etc.

[0050] Implements the decoding process for patch units encoded in homography mode. The input to this process is the current patch frame index frmIdx and the current patch index p. Given the parsed element in the patch data unit, first assign the following patch-related variables:

[0051] Patch2dShiftU[frmIdx][p]=pdu_2d_shift_u[frmIdx][p]

[0052] *ops_occupancy_packing_block_size (8-6)

[0053] Patch2dShiftV[frmIdx][p]=pdu_2d_shift_v[frmIdx][p]

[0054] *ops_occupancy_packing_block_size (8-7)

[0055] The variables Patch2dSizeU[frmIdx][p] and Patch2dSizeV[frmIdx][p] are then derived as follows:

[0056] If p is equal to 0, then:

[0057] Patch2dSizeU[frmIdx][p]=pdu_2d_delta_size_u[frmIdx][p]*ops_occupancy_packing_block_size (8-8)

[0058] Patch2dSizeV[frmIdx][p]=pdu_2d_delta_size_v[frmIdx][p]*ops_occupancy_packing_block_size (8-9)

[0059] Otherwise, if (p>0), then:

[0060] Patch2dSizeU[frmIdx][p]=Patch2dSizeU[frmIdx][p-1]+pdu_2d_delta_size_u[frmIdx][p]*ops_occupancy_packing_block_size (8-10)

[0061] Patch2dSizeV[frmIdx][p]=Patch2dSizeV[frmIdx][p-1]+pdu_2d_delta_size_v[frmIdx][p]*ops_occupancy_packing_block_size (8-11)

[0062]

[0063] Fig.16 Illustrate matrices used to reconstruct a point cloud from homography patches in accordance with some embodiments.The output point cloud is simply generated by multiplying the homography transformation by a vector in homogeneous coordinate representation formed by the position and depth values ​​in the canvas.

[0064] Another way to signal the homography transform is to add a flag to the patch data unit and signal that more information will be sent, namely the homography transform coefficients.

[0065] Fig.17 A block diagram of an exemplary computing device configured to implement point cloud coding using homography transformations according to some embodiments is illustrated. The computing device 1700 can be used to acquire, store, calculate, process, communicate and / or display information such as images and videos including 3D content. The computing device 1700 can implement any aspect of point cloud coding. Typically, a hardware structure suitable for implementing the computing device 1700 includes a network interface 1702, a memory 1704, a processor 1706, one or more I / O devices 1708, a bus 1710, and a storage device 1712. The choice of processor is not important, as long as an appropriate processor with sufficient speed is selected. The memory 1704 can be any conventional computer memory known in the art. The storage device 1712 can include a hard drive, a CDROM, a CDRW, a DVD, a DVDRW, a high-definition optical disk / drive, an ultra-high-definition drive, a flash memory card, or any other storage device. The computing device 1700 can include one or more network interfaces 1702. Examples of network interfaces include network cards connected to Ethernet or other types of LANs. I / O devices 1708 can include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. Point cloud coding application 1730 for implementing point cloud coding may be stored in storage device 1712 and memory 1704 and processed in the same manner as applications are normally processed. Computing device 1700 can include Fig.17 In some embodiments, point cloud encoding hardware 1720 is included. Although Fig.17The computing device 1700 in the embodiment includes an application 1730 and hardware 1720 for point cloud coding, but point cloud coding can be implemented on the computing device using hardware, firmware, software, or any combination thereof. For example, in some embodiments, the point cloud coding application 1730 is encoded in a memory and executed using a processor. In another example, in some embodiments, the point cloud coding hardware 1720 is programmed hardware logic including logic gates specifically designed to implement point cloud coding.

[0066] In some embodiments, the point cloud coding application 1730 includes several applications and / or modules. In some embodiments, the module also includes one or more sub-modules. In some embodiments, fewer or additional modules can be included.

[0067] In some embodiments, the point cloud encoding hardware 1720 includes camera components such as a lens, an image sensor, and / or any other camera components.

[0068] Examples of suitable computing devices include personal computers, laptops, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital video cameras, camera phones, smart phones, portable music players, tablet computers, mobile devices, video players, video disc recorders / players (e.g., DVD recorders / players, high-definition disc recorders / players, ultra-high-definition disc recorders / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smart watches), vehicles (e.g., self-driving vehicles), or any other suitable computing device.

[0069] In order to utilize point cloud coding using the homography transformation method, the device acquires or receives 3D content and processes and / or transmits the content in an optimized manner so that the 3D content can be properly and efficiently displayed. The point cloud coding can be implemented with the help of a user or can be implemented automatically without user involvement.

[0070] In operation, point cloud encoding using the homography transform method sends the homography transform of the 3D patch instead of explicit projection values ​​(such as bounding boxes and patch orientation and rotation). This method has a more compact representation, is more efficient in transmission, and allows faster decoding, especially in cases where 3D points will be reprojected (for example, when generating video in a mobile phone). Tests have shown that using point cloud encoding using the homography transform method improves performance compared to previous implementations.

[0071] Other advantages of using homographies (patches) include: exploiting any orientation (not just 6 axis aligned orientation) (e.g., 10 projections can be modeled using homographies); being able to use different types of cameras (orthogonal X-ray cameras used on depth and textures), generating backward compatible bitstreams compatible with 2D textures (decode attributes only), 2D+depth textures (decode attributes and homographies), and point clouds (decode everything); and homographies are compatible with current patch definitions (e.g., homographies can be obtained using current representations), intra / inter capabilities.

[0072] Some Examples of Point Cloud Coding Using Homography Transformation

[0073] 1. A method of programming in a non-transitory memory of a device, the method comprising:

[0074] Acquiring three-dimensional content; and

[0075] Encode 3D content using homography.

[0076] 2. The method of clause 1, wherein encoding the three-dimensional content using a homography transform comprises converting the three-dimensional content into two-dimensional content.

[0077] 3. The method of clause 1, wherein encoding the three-dimensional content using a homography comprises transmitting homography coefficients of a three-dimensional to two-dimensional transformation.

[0078] 4. The method of clause 1, wherein encoding the three-dimensional content using a homography transform comprises computing a homography matrix.

[0079] 5. A method according to clause 4, wherein the homography matrix is ​​calculated from three matrices.

[0080] 6. A method according to clause 5, wherein the three matrices include an observation matrix, a projection matrix and a viewport matrix.

[0081] 7. A method according to clause 6, wherein the observation matrix represents the camera, the projection matrix represents the pixels in the camera's sensor, and the viewport matrix represents the transformation of pixels in the camera to pixels in the image.

[0082] 8. A device comprising:

[0083] Non-transitory memory for storing applications for:

[0084] Acquiring three-dimensional content; and

[0085] encoding three-dimensional content using a homography; and

[0086] A processor is coupled to the memory, the processor being configured to process the application.

[0087] 9. The apparatus of clause 8, wherein encoding the three-dimensional content using the homography transform comprises converting the three-dimensional content into two-dimensional content.

[0088] 10. The apparatus of clause 8, wherein encoding the three-dimensional content using a homography comprises transmitting homography transform coefficients of a three-dimensional to two-dimensional transform.

[0089] 11. The apparatus of clause 8, wherein encoding the three-dimensional content using a homography transform comprises computing a homography matrix.

[0090] 12. Apparatus according to clause 11, wherein the homography matrix is ​​calculated from three matrices.

[0091] 13. The apparatus of clause 12, wherein the three matrices include a viewing matrix, a projection matrix, and a viewport matrix.

[0092] 14. A device according to clause 13, wherein the observation matrix represents the camera, the projection matrix represents the pixels in the sensor of the camera, and the viewport matrix represents the transformation of pixels in the camera to pixels in the image.

[0093] 15. A system comprising:

[0094] One or more cameras for acquiring three-dimensional content; and

[0095] An encoder for encoding three-dimensional content using a homography.

[0096] 16. The system of clause 15, wherein encoding the three-dimensional content using the homography transform comprises converting the three-dimensional content into two-dimensional content.

[0097] 17. The system of clause 15, wherein encoding the three-dimensional content using a homography comprises transmitting homography transformation coefficients of a three-dimensional to two-dimensional transformation.

[0098] 18. The system of clause 15, wherein encoding the three-dimensional content using a homography transform comprises computing a homography matrix.

[0099] 19. A system according to clause 18, wherein the homography matrix is ​​calculated from three matrices.

[0100] 20. A system according to clause 19, wherein the three matrices include a viewing matrix, a projection matrix and a viewport matrix.

[0101] 21. A system according to clause 20, wherein the observation matrix represents the camera, the projection matrix represents the pixels in the sensor of the camera, and the viewport matrix represents the transformation of pixels in the camera to pixels in the image.

[0102] The present invention has been described by specific embodiments, which contain details that facilitate understanding of the composition and operating principle of the present invention. Such reference to specific embodiments and details thereof herein is not intended to limit the scope of the appended claims. It will be apparent to those skilled in the art that various other modifications may be made in the embodiments selected for illustration without departing from the spirit and scope of the present invention as defined by the claims.

Claims

1. A method performed by a processor coupled to a memory, the method comprising: Acquiring three-dimensional content, including segmenting patches of the three-dimensional content, wherein points of the three-dimensional content having similar characteristics are projected onto the same plane, wherein the three-dimensional position is the distance from the point to the surface, and sending indices indicating which of the planes are selected, the selected planes being used to generate a bounding box; and The three-dimensional content is encoded using a homography transformation, wherein the three-dimensional content is encoded using a homography transformation including: Estimating normals, including: building a kd-tree, checking the local neighborhood, computing the best approximate tangent plane using covariance analysis, smoothing normals, and determining normal orientation, Therein, a segmented depth image bundled together is generated by orthogonally projecting the 3D facets, the segmented depth image is encoded, and auxiliary data representing the homography matrix is ​​transmitted together with the encoded segmented depth image.

2. The method of claim 1, wherein encoding the three-dimensional content using a homography comprises transmitting homography coefficients of a three-dimensional to two-dimensional transformation.

3. A device comprising: Non-transitory memory for storing applications for: Acquiring three-dimensional content, including segmenting patches of the three-dimensional content, wherein points of the three-dimensional content having similar characteristics are projected onto the same plane, wherein the three-dimensional position is the distance from the point to the surface, and sending indices indicating which of the planes are selected, the selected planes being used to generate a bounding box; and The three-dimensional content is encoded using a homography transformation, wherein the three-dimensional content is encoded using a homography transformation including: estimating normals, including: building a kd-tree, checking the local neighborhood, computing the best approximate tangent plane using covariance analysis, smoothing normals, and determining normal orientation; and a processor coupled to the memory, the processor configured to process the application, Therein, a segmented depth image bundled together is generated by orthogonally projecting the 3D facets, the segmented depth image is encoded, and auxiliary data representing the homography matrix is ​​transmitted together with the encoded segmented depth image.

4. The apparatus of claim 3, wherein encoding the three-dimensional content using a homography comprises transmitting homography coefficients of a three-dimensional to two-dimensional transformation.

5. A system comprising: one or more cameras for acquiring three-dimensional content; and The encoder is configured as: Segmenting patches of three-dimensional content, wherein points of the three-dimensional content having similar characteristics are projected onto the same plane, wherein the three-dimensional position is the distance from the point to the surface, and sending indices indicating which of the planes are selected to be used to generate a bounding box; and The three-dimensional content is encoded using a homography transformation, wherein the three-dimensional content is encoded using a homography transformation including: Estimating normals, including: building a kd-tree, checking the local neighborhood, computing the best approximate tangent plane using covariance analysis, smoothing normals, and determining normal orientation, Therein, a segmented depth image bundled together is generated by orthogonally projecting the 3D facets, the segmented depth image is encoded, and auxiliary data representing the homography matrix is ​​transmitted together with the encoded segmented depth image.

6. The system of claim 5, wherein the encoder is further configured to transmit homography transformation coefficients of the three-dimensional to two-dimensional transformation.

Images