Sparse GLCM: gray level co-occurrence matrix calculation for point cloud processing

Abstract

Described herein is a novel approach to classify point cloud data by extending gray level co-occurrence matrix (GLCM) technique from 2D to sparse 3D domain. The method can be applied to a point cloud derived from a mesh set / mesh, such as a real world texturing thing (RWTT) mesh set. Implementations designed for a variety of purposes are described herein: sampling and quantization of RWTT grids, generation of GLCM and corresponding texture descriptors, and selection of potential candidate point clouds based on these extracted descriptors.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 587,574, filed October 3, 2023, entitled “Sparse GLCM: Gray-Level Co-occurrence Matrix Computation for Point Cloud Processing”, entitled 35 USC §119(e), the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] This invention relates to three-dimensional graphics. More specifically, this invention relates to the encoding of three-dimensional graphics. Background Technology

[0004] Point cloud compression is an important technique for processing large sets of 3D data points used in various applications such as virtual reality (VR), augmented reality (AR), telecommunications, autonomous vehicles, and digital preservation of world heritage sites. The goal is to effectively compress the large amounts of data in point clouds without significant loss of detail or accuracy.

[0005] The Moving Picture Experts Group (MPEG) has developed two main point cloud compression standards: geometry-based point cloud compression (G-PCC) and video-based point cloud compression (V-PCC).

[0006] V-PCC leverages existing video compression techniques by projecting 3D point clouds onto a 2D plane and encoding these projections into a video stream. This approach is particularly advantageous for dynamic point clouds, such as those in real-time communication or interactive VR / AR environments.

[0007] G-PCC focuses on directly compressing 3D geometric data from point clouds. G-PCC is particularly effective for static point clouds (such as those used in cultural heritage conservation) or sparse point clouds used for autonomous navigation.

[0008] Due to the success of projection-based methods (also known as video-based methods or V-PCC) in encoding 3D point clouds, the standard is expected to include further 3D data, such as dynamic meshes, in future versions. However, the current version of the standard is only suitable for transmitting unconnected sets of points and still lacks a standardized mechanism for transmitting connections between points, as is required in dynamic mesh compression.

[0009] Advances in AI-based point cloud compression have motivated MPEG to research and potentially integrate AI technologies. Interest lies in learning-based codecs capable of managing a wide range of dynamic point clouds, crucial for applications such as immersive experiences and autonomous navigation. During the 146th MPEG meeting, the MPEG Technical Requirements (WG 2) announced a Call for Proposals (CfP) for AI-based point cloud coding technologies. A key component of these AI-based technologies is a test set capable of challenging the effectiveness of trained models. In this context, methods for classifying dataset samples to ensure they significantly represent the desired use cases are essential. Summary of the Invention

[0010] This paper describes a novel approach for classifying point cloud data by extending the Gray-Level Co-occurrence Matrix (GLCM) technique from 2D to the sparse 3D domain. This method can be applied to point clouds derived from mesh sets / mesh sets, such as Real-World Textured Things (RWTT) mesh sets. Implementations designed for various purposes are described: sampling and quantization of RWTT meshes, generation of GLCM and corresponding texture descriptors, and selection of potential candidate point clouds based on these extracted descriptors.

[0011] In one aspect, a method for programming in a device's non-transitory memory includes: finding a set of voxels; computing a GLCM for each gray-level co-occurrence matrix (GLCM) channel based on the colors of the two furthest voxels in the voxel set; and computing a texture metric based on the GLCM. When multiple color channels exist, one GLCM is computed for each channel. The method also includes performing a color transformation mapping the original multistimulus color space to the dominant monostimulus color space, and computes only one GLCM. The method further includes using oriented, user-specified neighborhoods by relaxing the search space around the voxels in a specific direction specified by an irregular bounding box. The specific direction specified by the irregular bounding box includes vertical, horizontal, or diagonal. Texture metrics include: energy, entropy, correlation, homogeneity, or contrast. The method also includes performing point cloud classification based on the texture metric. The voxel set resides within a sparse signal in a 6-D joint (x, y, z, R, G, B) dimension.

[0012] In another aspect, an apparatus includes: a non-transitory memory for storing an application for: finding a set of voxels; calculating a GLCM for each gray-level co-occurrence matrix (GLCM) channel based on the colors of the two farthest voxels in the voxel set; and calculating a texture metric based on the GLCM; and a processor coupled to the memory, configured to process the application. When multiple color channels are present, one GLCM is calculated for each channel. The application is configured to perform a color transformation mapping the original multistimulus color space to the dominant monostimulus color space and to calculate only one GLCM. The application is configured to use a directional, user-specified neighborhood by relaxing the search space around the voxels in a specific direction specified by an irregular bounding box. The specific direction specified by the irregular bounding box includes vertical, horizontal, or diagonal. Texture metrics include: energy, entropy, correlation, homogeneity, or contrast. The application is configured to perform point cloud classification based on the texture metrics. The voxel set is within a sparse signal in a 6-D joint (x, y, z, R, G, B) dimension.

[0013] On the other hand, a system includes: an encoder configured to: search a set of voxels; compute a GLCM for each gray-level co-occurrence matrix (GLCM) channel based on the colors of the two furthest voxels in the voxel set; compute a texture metric based on the GLCM; and perform point cloud classification based on the texture metric; and a decoder configured to receive the point cloud classification. When multiple color channels exist, one GLCM is computed for each channel. The encoder is configured to perform a color transformation mapping the original multistimulus color space to the dominant monostimulus color space and computes only one GLCM. The encoder is configured to use a directional, user-specified neighborhood by relaxing the search space around the voxels in a specific direction specified by an irregular bounding box. The specific direction specified by the irregular bounding box includes vertical, horizontal, or diagonal. Texture metrics include: energy, entropy, correlation, homogeneity, or contrast. The voxel set is within a sparse signal in a 6-D joint (x, y, z, R, G, B) dimension. Attached Figure Description

[0014] Figure 1 A diagram illustrating the spatial relationships between pixels according to some embodiments is shown.

[0015] Figure 2 A diagram of an exemplary gray-level co-occurrence matrix (GLCM) according to some embodiments is shown.

[0016] Figure 3 A diagram of a sparse GLCM in the RGB color space according to some embodiments is shown.

[0017] Figure 4 A diagram of an oriented bounding box according to some embodiments is shown.

[0018] Figure 5 A diagram showing the generation of texture metrics according to some embodiments is illustrated.

[0019] Figure 6 A diagram illustrating a 3D extension of GLCM according to some embodiments is shown.

[0020] Figure 7 Examples of red, green, and blue GLCMs according to some embodiments are shown.

[0021] Figure 8 A graph illustrating K-means clustering for defining texture categories, according to some embodiments, is shown.

[0022] Figures 9-11 An example of point cloud classification using sparse GLCM is shown according to some embodiments.

[0023] Figure 12 A diagram illustrating category analysis based on some embodiments is shown.

[0024] Figures 13-16 This illustrates how each point cloud, according to some embodiments, resides in its corresponding category within a metric pair.

[0025] Figure 17 A graph of a neural network-based attribute quality index is shown according to some embodiments.

[0026] Figure 18 A flowchart illustrating the implementation of sparse GLCM according to some embodiments is shown.

[0027] Figure 19 A block diagram of an exemplary computing device configured to implement a sparse GLCM method according to some embodiments is shown. Detailed Implementation

[0028] The point clouds in the dataset did not show significant representativeness in terms of attributes. To address the detected deficiencies, the contribution has enhanced the Call for Proposals (CfP) test set by deriving point clouds from meshes selected from a collection of Real-World Textured Things (RWTT) models. Since the collection presents more than 500 models, the challenge becomes choosing which models to use.

[0029] The RWTT dataset comprises publicly accessible textured 3D models generated using contemporary off-the-shelf photo reconstruction tools. The primary goal behind this dataset is to establish a benchmark for geometry processing algorithms designed to handle parametric, textured 3D models derived from real-world sources. In addition to serving as a benchmark for geometry processing, the RWTT dataset provides valuable property information compared to current CfP material test sets. The RWTT dataset consists of a collection of 568 textured models, making manual model selection a considerable challenge. A purely subjective selection process may not align with the objective challenges inherent in evaluating the representativeness of these models.

[0030] CfP materials envision two main use cases: dense point clouds for virtual reality (VR) / augmented reality (AR) / gaming, and sparse point clouds for autonomous driving and robotics.

[0031] Furthermore, it is important to note that dense point cloud categories include both static and dynamic point clouds. This paper discusses static dense point clouds and a novel method for classifying these point clouds. This method is based on an extension of the Gray-Level Co-occurrence Matrix (GLCM) metric, thus adapting it for use with point clouds. Moreover, this method has been applied to the RWTT dataset, resulting in the classification of mesh-derived point clouds into distinct categories. Based on the results of the classification process, a set of eighteen point clouds was distributed into three distinct categories. These point clouds are suggested as potential candidates for improving existing test sets, particularly for attribute encoding evaluation.

[0032] Gray-Level Co-occurrence Matrix (GLCM) is a technique in image processing and computer vision used to capture texture information in images. The pixel values ​​of an image are typically quantized as a discrete set of gray levels. For example, 8-bit quantization can be used, meaning the pixel values ​​range from 0 to 255. For each pixel in the quantized image, its relationship to its neighboring pixels is considered. GLCM counts the number of times a specific pair of pixel values ​​appears within the image in a specified spatial relationship. Spatial relationships can be defined by distance and direction (e.g., horizontal, vertical, diagonal), such as... Figure 1 As shown in the image.

[0033] The GLCM is constructed as a square matrix, where rows and columns represent different gray levels, and each element GLCM(i,j) represents the count of pixel pairs with value (i,j) under a specified spatial relation. To make the GLCM more robust to variations in image size and contrast, it is normalized by dividing each element by the sum of all elements in the matrix. This results in a probability matrix that represents the likelihood of observing a pixel value pair under a specified spatial relation. Figure 2 Examples according to some embodiments are shown.

[0034] Texture metrics (also known as texture features or texture descriptors) are quantitative measures that characterize the texture of an image. These metrics are calculated based on information derived from GLCM or other texture analysis methods. Some common texture metrics that can be calculated are defined in Table 1: contrast, entropy, homogeneity, energy, and correlation. These texture metrics can be used for a wide range of image analysis tasks, including image classification, segmentation, object recognition, and quality assessment.

[0035]

[0036] Table 1

[0037] The interpretation of the measures defined in Table 1 is as follows:

[0038] Contrast measures the intensity difference between adjacent pixels. A high contrast value indicates a significant variation in intensity in an image, corresponding to a coarse or grainy texture. On the other hand, a low contrast value indicates a more uniform or smooth texture.

[0039] Entropy is a measure of randomness or disorder in an image. High entropy values ​​indicate that pixel intensities are distributed in a more chaotic manner, associated with complex or noisy textures. Conversely, low entropy values ​​indicate more ordered or predictable textures.

[0040] Homogeneity measures the similarity or uniformity of pixel intensity in an image. A high homogeneity value means that adjacent pixels have similar intensity values, which is often seen in textures with fine, regular patterns. A lower homogeneity value indicates greater variation in pixel intensity.

[0041] Energy is a measure of the uniformity of the distribution of pixel pairs in a GLCM. A high energy value indicates fewer dominant intensity pairs in the image, resulting in a more uniform texture. A lower energy value means the presence of dominant pixel pairs, which may correspond to repetitive or structured textures.

[0042] Correlation measures the linear dependence between pixel values ​​at different locations in an image. A high correlation value indicates a strong correlation between pixel intensities at different locations, often indicating a textured region with well-defined patterns or directional features. A low correlation value means a weak or no linear relationship between pixel values ​​at different locations, which may indicate a more chaotic or random texture.

[0043] Point clouds are three-dimensional (3D) representations of the surfaces of objects or environments. They consist of a collection of individual data points, each with a set of coordinates in a 3D Cartesian coordinate system (X, Y, Z). In addition to the (X, Y, Z) coordinates, point clouds can also include additional attributes such as color, reflectivity, and normals. Point cloud data needs to be compressed for several reasons, such as storage and transmission efficiency.

[0044] V-PCC and geometry-based point cloud compression (G-PCC) are part of MPEG's efforts to standardize point cloud compression technology. While V-PCC converts point cloud data from 3D to 2D and then encodes it by a 2D video encoder, G-PCC encodes the content directly in 3D space.

[0045] Initially, the first version of G-PCC was primarily aimed at use cases involving static and multi-frame / fused LiDAR point clouds, especially in the context of automotive applications. It lacked tools for inter-frame compression.

[0046] Recently, MPEG has been working on a potential second version of G-PCC, aiming to extend its applicability to dynamic point clouds. This extension includes the incorporation of additional tools for inter-frame coding. Specifically, the development of tools for dynamic "solid" point clouds, previously a focus of Versatile Video Coding for Point Clouds (V-PCC), has prompted G-PCC experts to collaborate on a separate test model known as the Geometry Solid-Test Model (GeS-TM).

[0047] Emerging trends include expanding G-PCC's capabilities to cover: dynamic (inter-frame) point cloud geometry and attribute compression designed for use with LiDAR data (especially in the context of automotive applications) and generally for sparse point clouds; and dynamic (inter-frame) point cloud geometry and attribute compression specifically designed for "solid" point clouds.

[0048] Current G-PCC test models include tools for inter-frame geometric coding. Recent developments have expanded their capabilities to include inter-frame attribute coding, specifically by incorporating Inter-Region-Adaptive Hierarchical Transform (Inter-RAHT). Currently, Inter-RAHT reuses motion vectors estimated for geometry to also perform inter-frame attribute coding.

[0049] Color information can be used to enhance motion estimation, thereby achieving more efficient inter-frame attribute coding while maintaining geometric coding efficiency. A strategy can be implemented that jointly uses geometric and attribute information to perform motion estimation. Distortion is calculated as a weighted sum of color and geometric distortion. The challenge lies in optimally selecting the weighting factor between color and geometry, which can vary on a block-by-block basis.

[0050] One challenge associated with point clouds is their sparsity. Many assumptions that are valid for 2D images cannot be directly extended to point clouds without adaptation. The computation of GLCM serves as one example, and extending the computational framework of GLCM to fit point clouds is another.

[0051] Due to the inherent sparsity of point clouds, neighboring points are not always available for reference or analysis. A more flexible neighborhood definition within the context of point clouds is important. In traditional GLCM definitions, spatial relationships can be defined by the distance and orientation of adjacent pixels in a fully occupied grid. Since the existence of exactly adjacent voxels cannot be guaranteed, for each voxel in the point cloud, the neighborhood is considered as the N nearest voxels in any orientation and distance. This method ensures that the direction of analysis will at least approximately coincide with the surface orientation determined by the distribution of points in space. The colors of the two farthest voxels in the set are then used as a pair to compute the GLCM. Any color space can be used. In the case of multiple color channels, one GLCM is computed for each channel. For example, the GLCM can be computed for each channel and averaged. Another option is to perform any color transformation that maps the original multistimulus color space to a monostimulus-dominant color space and compute only one GLCM using a single channel. Figure 3 A GLCM construction for the RGB color space is shown according to some embodiments.

[0052] like Figure 4 As shown, oriented, user-specified neighborhoods can also be used by relaxing the search space around the voxel in a specific direction. A vertical direction, along with a possible horizontal and diagonal direction, is used as an example, but any oriented bounding box can be used for neighborhood search.

[0053] Once the GLCM is calculated, texture metrics can be calculated as defined in Table 1. Figure 5 A graph is shown illustrating sparse GLCM computation of point clouds according to some embodiments.

[0054] As an application, the sparse GLCM method can be used to classify point clouds into texture categories. This method can be applied to the RWTT dataset, which contains 568 models, 67 large models (more than 1 million faces), 109 models with multiple textures, and 18 different pipelines.

[0055] In an exemplary implementation, the selection process is defined as follows: mesh selection; sampling and quantizing the selected mesh using metric software; removing duplicate points; retaining only the resulting point cloud with more than 500k points for analysis; calculating sparse GLCM for the R, G, and B color components; exporting point clouds for all 208 meshes to calculate contrast, homogeneity, energy, entropy, and correlation texture metrics; applying K-means clustering to the calculated 5-dimensional metric vectors to define texture categories; classifying each point cloud in the set based on its proximity to one of the category centroids; and selecting the 6 nearest point clouds for each centroid as candidates. While a specific example is provided, any modifications are possible.

[0056] Figure 6 A diagram showing a 3D extension of the gray-level co-occurrence matrix (GLCM) calculated for the R, G, and B color components according to some embodiments is shown.

[0057] exist Figure 7 The image shows an example of a GLCM with red, green, and blue channels.

[0058] For each GLCM of the point cloud exported from all 208 meshes, the following metrics are calculated: contrast, homogeneity, energy, entropy, and correlation.

[0059] Apply K-means clustering (or other clustering implementations) to a 5-dimensional matrix vector to define texture categories. Figure 8 Examples of the three defined categories are shown.

[0060] Each point cloud in the set is classified based on its proximity to one of the centroids.

[0061] After applying this method to the RWTT ensemble, eighteen point clouds distributed across three categories were recommended as potential candidates. These point clouds in Figures 9-11 As shown in the image.

[0062] Figure 12 The contrast, homogeneity, energy, entropy, and correlation values ​​of the centroid for each category are shown according to some embodiments.

[0063] Figures 13-16 This illustrates how each point cloud lies within its corresponding category in the metric pair. Classification actually occurs in the 5-dimensional space of all 5 texture metrics.

[0064] As described above, this implementation can jointly use geometric and attribute information to perform motion estimation. Distortion Calculated as color distortion ( ) and geometric distortion ( The weighted sum of ) is as follows:

[0065]

[0066] The challenge lies in optimally selecting the weighting factor between color and geometry. The previously described point cloud classification method can be used to classify each point cloud patch into a specific texture category and adjust the weighting factor based on the category it belongs to. The value. Specific alpha values ​​are selected based on block-based texture information. This can improve coding efficiency in terms of rate distortion.

[0067] Sparse GLCM can be used alone or even in combination as a quality metric for point cloud attribute compression, as shown below:

[0068]

[0069] Or even as Figure 17 As shown in the neural network-based attribute quality index.

[0070] Additional implementations are possible. Other GLCM metrics can be combined. Although the description in this paper considers 3D sparse signals (point clouds), the concept can be extended to any N-dimensional sparse signal. For example, it can be extended to the (x, y, z) spatial domain or (x, y, z, R, G, B). In the case of a 6D signal (x, y, z, R, G, B), a GLCM associated with each dimension x, y, z, R, G, and B can be computed, resulting in 6 sets of descriptors. These 6 sets can be concatenated into a single unified descriptor or combined using weighting factors. Just as the GLCMs for R, G, and B can be used to describe texture properties, the GLCMs for x, y, and z can be used to describe geometric properties. Furthermore, the GLCMs for x, y, z, R, G, and B derived from the joint 6D representation of each point can be used to evaluate geometry and texture in a unified manner.

[0071] To enhance the test set by incorporating more textured point clouds, an objective classification method can be used. Specifically, a subset of point clouds is selected from a pool of six candidate point clouds according to each category.

[0072] Selecting point clouds for evaluation from a pool of more than 200 options can be a highly complex task. In such cases, objective selection methods with clear rationale offer significant advantages over relying solely on subjective judgment. Objective selection methods are preferred because: (a) they provide a systematic and repeatable process for selecting point clouds. This allows other experts to follow the same methodology, ensuring consistency in dataset selection across different activities; (b) subjective selection can introduce bias and arbitrary choices based on personal preferences; and (c) objective methods are transparent and reasonable. They allow researchers to clearly articulate the criteria behind the selection of a particular point cloud.

[0073] Figure 18A flowchart illustrating the implementation of a sparse GLCM according to some embodiments is shown. In step 1800, a set of N voxels closest to the current voxel is searched in any direction and at any distance. The nearest N voxels can be any voxel within a set distance, or the search can proceed from the nearest to the farthest voxels, and the search stops when N voxels have been found. In step 1802, the colors of the two farthest voxels in the set of nearest voxels (e.g., farthest from the current voxel) are used as a pair to compute the GLCM. Any color space can be used. In the case of multiple color channels, one GLCM is computed for each channel. In step 1804, the GLCM is computed for each GLCM channel and averaged. In some embodiments, another option is to perform any color transformation that maps the original multistimulus color space to the dominant monostimulus color space and use this single channel to compute only one GLCM. In some embodiments, a directional, user-specified neighborhood is used by relaxing the search space around the voxel in a particular direction specified by an irregular bounding box. In step 1806, once the GLCM is computed, a texture metric is computed as in the conventional 2D case. Some metrics are energy, entropy, correlation, homogeneity, and contrast, but others can be used. Although this paper describes 3D sparse signals (point clouds), the concept can be extended to any number of sparse dimensions for any signal. For example, the concept can be extended to 3D (x, y, z) spatial dimensions, 6D joint (x, y, z, R, G, B) dimensions, or generally to ND (x1, x2, x3, ..., xn) sparse signals. For example, in the case of (x, y, z, R, G, B), a GLCM associated with each component x, y, z, R, G, and B is computed, resulting in 6 sets of descriptors (metrics). These 6 sets can be concatenated into a single unified descriptor or combined using weighting factors. Just as the GLCMs for R, G, and B can be used to describe texture properties, the GLCMs for x, y, and z can be used to describe geometric properties. Furthermore, the GLCMs for x, y, z, R, G, and B derived from the joint 6D representation of each point can be used to evaluate geometry and texture in a unified manner. In step 1808, point cloud classification is performed based on texture metrics. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is modified.

[0074] Figure 19A block diagram of an exemplary computing device configured to implement a sparse GLCM method according to some embodiments is shown. The computing device 1900 is capable of acquiring, storing, computing, processing, transmitting, and / or displaying information, such as images and videos including 3D content. The computing device 1900 is capable of implementing any encoding / decoding aspects. Typically, a suitable hardware architecture for implementing the computing device 1900 includes a network interface 1902, memory 1904, a processor 1906, I / O devices 1908, a bus 1910, and a storage device 1912. The choice of processor is not critical, as long as a suitable processor with sufficient speed is selected. Memory 1904 can be any conventional computer memory known in the art. Storage device 1912 can include a hard disk drive, CD-ROM, CDRW, DVD, DVDRW, high-definition optical disc / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computing device 1900 can include one or more network interfaces 1902. Examples of network interfaces include network interface cards (NICs) connected to Ethernet or other types of LANs. I / O device 1908 may include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The sparse GLCM application 1930 used to implement the sparse GLCM implementation is likely stored in storage device 1912 and memory 1904, and is typically processed as an application. Figure 19 More or fewer of the components shown can be included in the computing device 1900. In some embodiments, sparse GLCM hardware 1920 is included. Although Figure 19 The computing device 1900 includes an application 1930 and hardware 1920 for a sparse GLCM implementation, but the sparse GLCM method can be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the sparse GLCM application 1930 is programmed in memory and executed using a processor. In another example, in some embodiments, the sparse GLCM hardware 1920 is programmed hardware logic including gates specifically designed to implement the sparse GLCM method.

[0075] In some embodiments, the sparse GLCM application 1930 includes several applications and / or modules. In some embodiments, a module further includes one or more sub-modules. In some embodiments, fewer or additional modules can be included.

[0076] Examples of suitable computing devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc writers / players (e.g., DVD writers / players, high-definition disc writers / players, ultra-high-definition disc writers / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computing device.

[0077] To utilize sparse GLCM methods, devices acquire or receive 3D content (e.g., point cloud content). Sparse GLCM methods can be implemented with user assistance or automatically without user intervention.

[0078] In practice, the sparse GLCM method involves extending the computational framework of GLCM to accommodate point clouds or any multi-dimensional ND sparse signal. Due to the inherent sparsity of point clouds, neighboring points are not always available for reference or analysis. A more flexible neighborhood definition is used within the context of point clouds. In the traditional GLCM definition, spatial relationships can be defined according to the distance and orientation of adjacent pixels in a fully occupied grid. Since the existence of exactly adjacent voxels in a point cloud cannot be guaranteed, the definition of neighborhood is updated to compute GLCM and therefore the associated texture metrics. GLCM of point clouds can be used in a variety of scenarios. First, in the context of point cloud classification, the codec can adjust its encoding parameters accordingly based on the texture characteristics of the input point cloud. For example, if more complex texture patterns are identified, more bits can be allocated to attributes. Second, in the context of geometry-based point cloud compression schemes, motion vector computation in inter-frame coding of geometry and attributes requires balancing the importance of geometry and texture patterns. Currently, the Inter-RAHT attribute coding in version 2 of G-PCC reuses motion vectors for geometry estimation in order to also perform inter-frame coding of attributes. By using GLCM to analyze point cloud texture, the GLCM method can help select weighting factors. Specific alpha values ​​are selected based on block-based texture information. (3) This can improve coding efficiency in the sense of rate distortion. Third, since lossy attribute encoding implies attribute degradation, and the proposed extension of GLCM to point clouds characterizes the texture of point clouds, they can also be used as quality metrics, including in neural network-based quality indices.

[0079] Sparse GLCM: Some Examples of Gray-Level Co-occurrence Matrix Computation for Point Cloud Processing

[0080] 1. A method for programming in a non-transitory memory of a device, the method comprising:

[0081] Find the set of voxels;

[0082] For each Gray-Level Co-occurrence Matrix (GLCM) channel, the GLCM is calculated based on the colors of the two furthest voxels in the voxel set; and

[0083] Texture metrics are calculated based on GLCM.

[0084] 2. The method described in Clause 1, wherein when there are multiple color channels, one GLCM is calculated for each channel.

[0085] 3. The method according to Clause 1 further includes performing a color transformation that maps the original multistimulus color space to the dominant monostimulus color space, and calculating only one GLCM.

[0086] 4. The method described in Clause 1 further includes using a directional, user-specified neighborhood by relaxing the search space around the voxel in a specific direction specified by the non-regular bounding box.

[0087] 5. The method according to Clause 4, wherein the specific direction specified by the irregular bounding box includes vertical, horizontal or diagonal lines.

[0088] 6. The method according to Clause 1, wherein the texture metric includes: energy, entropy, correlation, homogeneity, or contrast.

[0089] 7. The method according to Clause 1 further includes performing point cloud classification based on the texture metric.

[0090] 8. The method according to Clause 1, wherein the set of voxels is within a sparse signal of a 6-D joint (x, y, z, R, G, B) dimension.

[0091] 9. An apparatus comprising:

[0092] Non-transitory memory for storing applications, which are used for:

[0093] Find the set of voxels;

[0094] For each Gray-Level Co-occurrence Matrix (GLCM) channel, the GLCM is calculated based on the colors of the two furthest voxels in the voxel set; and

[0095] Texture metrics are calculated based on GLCM; and

[0096] A processor coupled to the memory, the processor being configured to process the application.

[0097] 10. The apparatus according to Clause 9, wherein when there are multiple color channels, one GLCM is calculated for each channel.

[0098] 11. The apparatus according to Clause 9, wherein the application is configured to perform a color transformation that maps the original multistimulus color space to the dominant monostimulus color space, and computes only one GLCM.

[0099] 12. The apparatus according to Clause 9, wherein the application is configured to use a directional, user-specified neighborhood by relaxing the search space around the voxel in a particular direction specified by an irregular bounding box.

[0100] 13. The apparatus according to Clause 12, wherein the particular direction specified by the irregular bounding box includes vertical, horizontal or diagonal lines.

[0101] 14. The apparatus according to Clause 9, wherein the texture metric includes: energy, entropy, correlation, homogeneity, or contrast.

[0102] 15. The apparatus according to Clause 9, wherein the application is configured to perform point cloud classification based on the texture metric.

[0103] 16. The apparatus according to Clause 9, wherein the voxel set is contained within a sparse signal of a 6-D joint (x, y, z, R, G, B) dimension.

[0104] 17. A system comprising:

[0105] The encoder is configured to:

[0106] Find the set of voxels;

[0107] For each gray-level co-occurrence matrix (GLCM) channel, the GLCM is calculated based on the colors of the two furthest voxels in the voxel set.

[0108] Texture metrics are calculated based on GLCM; and

[0109] Perform point cloud classification based on the texture metric; and

[0110] A decoder configured to receive the point cloud classification.

[0111] 18. The system according to Clause 17, wherein when there are multiple color channels, one GLCM is calculated for each channel.

[0112] 19. The system according to Clause 17, wherein the encoder is configured to perform a color transformation that maps the original multistimulus color space to the dominant monostimulus color space, and computes only one GLCM.

[0113] 20. The system according to Clause 17, wherein the encoder is configured to use a directional, user-specified neighborhood by relaxing the search space around the voxel in a particular direction specified by an irregular bounding box.

[0114] 21. The system according to Clause 20, wherein a particular direction specified by the irregular bounding box includes a vertical, horizontal, or diagonal line.

[0115] 22. The system according to Clause 17, wherein the texture metric includes: energy, entropy, correlation, homogeneity, or contrast.

[0116] 23. The system according to Clause 17, wherein the set of voxels is within a sparse signal of a 6-D joint (x, y, z, R, G, B) dimension.

[0117] The present invention has been described with reference to specific embodiments in conjunction with details to facilitate understanding of the construction and operating principles of the invention. Such references to specific embodiments and their details herein are not intended to limit the scope of the appended claims. It will be readily apparent to those skilled in the art that various modifications may be made to the selected illustrative embodiments without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. A method for programming in a non-transitory memory of a device, the method comprising: Find the set of voxels; For each gray-level co-occurrence matrix (GLCM) channel, the GLCM is calculated based on the colors of the two furthest voxels in the voxel set. as well as Texture metrics are calculated based on GLCM.

2. The method according to claim 1, wherein, When there are multiple color channels, one GLCM is calculated for each channel.

3. The method of claim 1 further comprises performing a color transformation that maps the original multistimulus color space to the dominant monostimulus color space, and calculating only one GLCM.

4. The method of claim 1, further comprising using a directional, user-specified neighborhood by relaxing the search space around the voxel in a specific direction specified by the irregular bounding box.

5. The method according to claim 4, wherein, The specific direction specified by the irregular bounding box includes vertical, horizontal, or diagonal lines.

6. The method according to claim 1, wherein, The texture metrics include: energy, entropy, correlation, homogeneity, or contrast.

7. The method of claim 1, further comprising performing point cloud classification based on the texture metric.

8. The method according to claim 1, wherein, The voxel set is within a sparse signal in a 6-D joint (x, y, z, R, G, B) dimension.

9. An apparatus comprising: Non-transitory memory for storing applications, which are used for: Find the set of voxels; For each gray-level co-occurrence matrix (GLCM) channel, the GLCM is calculated based on the colors of the two furthest voxels in the voxel set. as well as Calculate texture metrics based on GLCM; as well as A processor coupled to the memory, the processor being configured to process the application.

10. The apparatus according to claim 9, wherein, When there are multiple color channels, one GLCM is calculated for each channel.

11. The apparatus according to claim 9, wherein, The application is configured to perform a color transformation that maps the original multistimulus color space to the dominant monostimulus color space, and computes only one GLCM.

12. The apparatus according to claim 9, wherein, The application is configured to use a directional, user-specified neighborhood by relaxing the search space around the voxel in a specific direction specified by the irregular bounding box.

13. The apparatus according to claim 12, wherein, The specific direction specified by the irregular bounding box includes vertical, horizontal, or diagonal lines.

14. The apparatus according to claim 9, wherein, The texture metrics include: energy, entropy, correlation, homogeneity, or contrast.

15. The apparatus according to claim 9, wherein, The application is configured to perform point cloud classification based on the texture metric.

16. The apparatus according to claim 9, wherein, The voxel set is within a sparse signal in a 6-D joint (x, y, z, R, G, B) dimension.

17. A system comprising: The encoder is configured to: Find the set of voxels; For each gray-level co-occurrence matrix (GLCM) channel, the GLCM is calculated based on the colors of the two furthest voxels in the voxel set. Calculate texture metrics based on GLCM; as well as Point cloud classification is performed based on the texture metric. as well as A decoder configured to receive the point cloud classification.

18. The system according to claim 17, wherein, When there are multiple color channels, one GLCM is calculated for each channel.

19. The system according to claim 17, wherein, The encoder is configured to perform a color transformation that maps the original multistimulus color space to the dominant monostimulus color space, and computes only one GLCM.

20. The system according to claim 17, wherein, The encoder is configured to use a directional, user-specified neighborhood by relaxing the search space around the voxel in a specific direction specified by the irregular bounding box.

21. The system according to claim 20, wherein, The specific direction specified by the irregular bounding box includes vertical, horizontal, or diagonal lines.

22. The system according to claim 17, wherein, The texture metrics include: energy, entropy, correlation, homogeneity, or contrast.

23. The system according to claim 17, wherein, The voxel set is within a sparse signal in a 6-D joint (x, y, z, R, G, B) dimension.

Images