Dual-stream point cloud compression method and system based on semantic scene graph

Abstract

This invention discloses a dual-stream point cloud compression method and system based on semantic scene graphs, relating to the field of point cloud compression. It constructs a point cloud compression network comprising an encoder network and a decoder network: In the encoder network, semantic labels are mapped onto the input point cloud, and scene graphs and octree representations are constructed respectively. Semantic modulation parameters are generated based on the scene graph and encoded to obtain a semantic bitstream. The octree representation, under the influence of semantic modulation parameters, undergoes feature enhancement and attention probability prediction to obtain node occupancy probabilities, and a geometric bitstream is generated through arithmetic encoding. These two bitstreams serve as compressed data. In the decoder network, the bitstreams are decoded, and the octree structure and scene graph information are reconstructed. Node feature representations are recovered by combining the semantic modulation parameters. The reconstructed octree structure is mapped back to the point cloud space through spatial querying and indexing to reconstruct the point cloud. This invention solves the problem of local geometric distortion in low-bitrate octree point cloud compression, ensuring reconstruction quality while reducing the transmission bitrate.

Description

Technical Field

[0001] This invention relates to the field of point cloud compression technology, and in particular to a dual-stream point cloud compression method and system based on semantic scene graphs. Background Technology

[0002] Point clouds, as a data format capable of efficiently representing 3D shapes or objects, are gradually becoming a hot topic in fields such as computer vision, computer graphics, and machine learning. With the increasing performance of point cloud acquisition devices, it is possible to obtain point cloud models with multiple levels of detail from 3D scenes, leading to increasingly larger volumes of point cloud data. This large volume of point cloud data increases the cost of transmission and storage, placing a burden on storage capacity and network bandwidth. Point cloud data compression has become one of the key technologies for solving this problem.

[0003] Efficient point cloud compression technology can significantly reduce data storage requirements and transmission bandwidth consumption while maintaining the quality and detail of point cloud data. In recent years, with the introduction of point cloud compression standards and the application of advanced technologies such as deep learning, point cloud compression technology has developed rapidly. However, how to further utilize the geometric spatial information of point clouds to improve the efficiency of point cloud compression remains a significant challenge in point cloud compression research. Summary of the Invention

[0004] To address the above issues, this invention proposes a dual-stream point cloud compression method and system based on semantic scene graphs. By introducing an explicit semantic scene graph modeling mechanism during the point cloud compression process, high-level semantic information and low-level geometric structure information in the point cloud are decoupled, modeled, and co-coded, constructing a dual-stream compression framework that generates semantic bitstreams and geometric bitstreams in parallel. Simultaneously, semantic modulation parameters generated by the semantic scene graph are used to conditionally modulate the geometric features of octree nodes, thereby achieving more accurate node occupancy probability prediction in the entropy coding stage. This effectively reduces the bit rate of point cloud transmission and storage while significantly improving the reconstruction quality and local detail preservation capability of point cloud geometry in low bit rate scenarios, enhancing the adaptability of the point cloud compression system to complex scenes and multiple semantic objects.

[0005] On the one hand, the dual-stream point cloud compression method based on semantic scene graphs has the following specific steps:

[0006] S1. Construct and train a point cloud compression network that includes an encoder network and a decoder network to obtain a trained point cloud compression network.

[0007] In the encoder network, the semantic label mapping module performs semantic label mapping on the point cloud data to be compressed to obtain a point cloud with semantic labels; the scene graph construction module constructs a semantic scene graph from the point cloud with semantic labels and inputs it into the first feature linear modulation generator to generate the first semantic modulation parameters; the scene graph encoding module encodes the semantic scene graph into a semantic bit stream; the octree construction module constructs an octree representation from the point cloud data to be compressed, and the octree representation and the first semantic modulation parameters are fused by the first feature condition enhancement module. The fused features are then used by the first cosine attention probability prediction module to calculate the octet probability of the octree nodes, and then encoded into a geometric bit stream by the arithmetic encoding module.

[0008] In the decoder network, the scene graph parsing module reconstructs the semantic bitstream to obtain a reconstructed semantic scene graph; the reconstructed semantic scene graph is then processed by the spatial query and indexing module to obtain the restored semantic information of the matching octree nodes, and input into the second feature linear modulation generator to generate the second semantic modulation parameters; the arithmetic decoding module decodes the geometric bitstream into an octree occupancy code; the octree occupancy code and the second semantic modulation parameters are input into the second feature condition enhancement module for feature fusion, and the fused features are processed by the second cosine attention probability prediction module to calculate the restored occupancy probability; the octree decoding module generates an octree structure based on the restored occupancy probability; and the octree structure is mapped to point cloud spatial coordinates based on the reconstructed semantic scene graph to obtain reconstructed point cloud data.

[0009] S2, input the point cloud data to be compressed into the trained point cloud compression network, where the encoder network encodes the point cloud data to be compressed into semantic bit streams and geometric bit streams; the decoder network decodes the corresponding semantic bit streams and geometric bit streams into reconstructed point cloud data.

[0010] Preferably, the scene graph construction module is specifically used for:

[0011] A semantic scene graph is constructed by semantically clustering the point cloud after semantic label mapping, and using the clustered point set as a semantic object to construct a semantic scene graph with semantic objects as nodes; the semantic object node attributes include semantic labels and axis-aligned bounding boxes that quantify the spatial range of the semantic object.

[0012] Associating octree nodes with semantic scene graphs, if an octree node to be encoded falls within the axis-aligned bounding box space of a semantic object, assigning the corresponding semantic label to the octree node.

[0013] Preferably, both the first characteristic linear modulation generator and the second characteristic linear modulation generator can be represented as:

[0014] ;

[0015] ;

[0016] ;

[0017] in, Represents a semantic descriptor vector. Indicates the index of a semantic object in the semantic scene graph; Indicates that there are learnable parameters Embedded layer; This represents the discrete semantic label corresponding to the octree node; This represents the scaling factor used for feature modulation; Indicates the scaling factor used to generate the scaling factor. The neural network branch; This represents the offset factor used for feature modulation; Indicates the value used to generate offset coefficients The neural network branch;

[0018] ;

[0019] in, This indicates the modulated output characteristics. Indicates a distinguishing superscript; The octree geometric features representing the original input; This indicates element-wise multiplication; This indicates element-wise addition.

[0020] Preferably, the feature fusion is expressed as:

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] in, This represents the enhanced output features; This represents a pointwise convolution with a kernel of 1. Indicates conditional input features; This represents a depthwise convolution with a kernel size of 3; This represents the output features of the wide receptive field branch in a multi-scale branching model; express Activation function; This represents the output characteristics of the narrow receptive field and channel transform branches in a multi-scale branch; This represents the output of the feature condition enhancement module; This indicates a concatenation operation at the channel dimension.

[0026] Preferably, both the first cosine attention probability prediction module and the second cosine attention probability prediction module include an adaptive sliding window layer, a gated enhanced channel mixer, and a potential guiding node occupancy predictor; the adaptive sliding window layer captures contextual relationships from the input feature sequence to obtain contextual features; the gated enhanced channel mixer enhances the contextual features in the channel dimension to obtain channel-enhanced features; and the potential guiding node occupancy predictor maps the channel-enhanced features to occupancy probabilities.

[0027] Preferably, both the first cosine attention probability prediction module and the second cosine attention probability prediction module are represented as follows:

[0028] ;

[0029] ;

[0030] ;

[0031] in, Represents the query vector; Represents the key vector; Represents a value vector; This represents a pointwise convolution with a kernel of 1. This represents a depthwise convolution with a kernel size of 3; Represents input features; Indicates the first The query vector and the first Attention score between key vectors; This represents the Softmax function; and These are the sub-head feature vectors after multi-head partitioning; Embed tensors for learnable queries; The length of the sliding window. For learnable scaling parameters; Indicates transpose; Indicates modulo; It is a mask matrix; Indicates the probability of occupancy; For the value vector at the corresponding position, This is a splicing operation at the channel dimension.

[0032] Preferably, the gated enhancement channel mixer is represented as:

[0033] ;

[0034] ;

[0035] ;

[0036] in, Indicates a gating signal; Indicates a transformation branch; This indicates an average division along the channel dimension; This represents a pointwise convolution with a kernel of 1. Represents input features; Indicates the output characteristics of the transform branch; express Activation function; This represents a depthwise convolution with a kernel size of 3; This indicates the characteristics after channel enhancement; This indicates element-wise multiplication.

[0037] Preferably, the potential boot node occupancy predictor is represented as:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] in, Indicates the initial state; Indicates fully connected layer operations; Represents input features; Indicates intermediate features; This represents a pointwise convolution with a kernel of 1. This represents a depthwise convolution with a kernel size of 3; Indicates enhanced features; express Activation function; This indicates element-wise multiplication; This represents the final predicted probability distribution; This represents the Softmax function.

[0043] Preferably, the process of mapping the octree structure to point cloud spatial coordinates based on the reconstructed semantic scene graph to obtain reconstructed point cloud data is represented as follows:

[0044] ;

[0045] in, Indicates the quantization step size; Represents the integer grid coordinates of the point cloud when the octree is decoded to a leaf node; This represents the offset of the minimum coordinate value.

[0046] On the other hand, a dual-stream point cloud compression system based on semantic scene graphs includes the following:

[0047] The point cloud compression network construction and training module is used to construct and train a point cloud compression network including an encoder network and a decoder network to obtain a trained point cloud compression network.

[0048] In the encoder network, the semantic label mapping module performs semantic label mapping on the point cloud data to be compressed to obtain a point cloud with semantic labels; the scene graph construction module constructs a semantic scene graph from the point cloud with semantic labels and inputs it into the first feature linear modulation generator to generate the first semantic modulation parameters; the scene graph encoding module encodes the semantic scene graph into a semantic bit stream; the octree construction module constructs an octree representation from the point cloud data to be compressed, and the octree representation and the first semantic modulation parameters are fused by the first feature condition enhancement module. The fused features are then used by the first cosine attention probability prediction module to calculate the octet probability of the octree nodes, and then encoded into a geometric bit stream by the arithmetic encoding module.

[0049] In the decoder network, the scene graph parsing module reconstructs the semantic bitstream to obtain a reconstructed semantic scene graph; the reconstructed semantic scene graph is then processed by the spatial query and indexing module to obtain the restored semantic information of the matching octree nodes, and input into the second feature linear modulation generator to generate the second semantic modulation parameters; the arithmetic decoding module decodes the geometric bitstream into an octree occupancy code; the octree occupancy code and the second semantic modulation parameters are input into the second feature condition enhancement module for feature fusion, and the fused features are processed by the second cosine attention probability prediction module to calculate the restored occupancy probability; the octree decoding module generates an octree structure based on the restored occupancy probability; and the octree structure is mapped to point cloud spatial coordinates based on the reconstructed semantic scene graph to obtain reconstructed point cloud data.

[0050] The point cloud compression and reconstruction module is used to input the point cloud data to be compressed into the trained point cloud compression network. The encoder network encodes the point cloud data to be compressed into semantic bit streams and geometric bit streams; the decoder network decodes the corresponding semantic bit streams and geometric bit streams into reconstructed point cloud data.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) This invention proposes a strategy of introducing a semantic scene graph to achieve the collaborative compression of geometry and semantics. This strategy constructs a semantic scene graph to express the object-level semantic information, spatial position relationship and scale attribute in the point cloud in a structured way, and encodes and decodes it as an independent information flow, so that the compression process can make full use of the semantic prior information contained in the point cloud, thereby improving the overall compression efficiency.

[0053] (2) This invention employs a dual-stream compression structure that generates semantic bitstream and geometric bitstream in parallel, decoupling semantic information from geometric occupancy information in the model, thus avoiding the problem of mutual interference between multiple sources of information in a single bitstream. Compared with traditional single-stream point cloud compression methods, this dual-stream structure can more stably maintain the overall structure and local details of the point cloud under low bitrate conditions, effectively alleviating the problem of geometric distortion;

[0054] (3) This invention utilizes semantic modulation parameters generated from semantic scene graphs to conditionally modulate the geometric features of octree nodes through the FiLM mechanism, enabling nodes of different semantic categories or scene regions to possess differentiated characteristics in the feature representation and probability modeling stages. Compared with traditional probability prediction methods based solely on spatial context, this approach can significantly improve the accuracy of octree node occupancy probability distribution modeling, thereby reducing the number of bits required for entropy coding. Attached Figure Description

[0055] The present invention will now be described in further detail with reference to the accompanying drawings;

[0056] Figure 1 This is a flowchart of a two-stream point cloud compression method based on a semantic scene graph, according to an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram of the point cloud compression model of the two-stream point cloud compression method based on semantic scene graphs according to an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the local enhanced context coding module of the two-stream point cloud compression method based on semantic scene graph according to an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of the adaptive length sliding window attention module of the two-stream point cloud compression method based on semantic scene graph in an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of a spatially gated enhanced channel mixer for a two-stream point cloud compression method based on a semantic scene graph, according to an embodiment of the present invention.

[0061] Figure 6 This is a schematic diagram of a potential guide node occupancy predictor for a two-stream point cloud compression method based on a semantic scene graph, according to an embodiment of the present invention.

[0062] Figure 7 This is a schematic diagram of the point-to-point peak signal-to-noise ratio and chamfer distance at a bit rate per point for the two-stream point cloud compression method based on semantic scene graphs according to an embodiment of the present invention; wherein, (a) represents the rate-distortion performance curve; and (b) represents the chamfer distance performance curve.

[0063] Figure 8 This is a structural block diagram of a dual-stream point cloud compression system based on a semantic scene graph, according to an embodiment of the present invention. Detailed Implementation

[0064] The present invention will be further described below through specific embodiments.

[0065] like Figure 1 As shown, the specific steps of the two-stream point cloud compression method based on semantic scene graphs are as follows:

[0066] S1. Construct and train a point cloud compression network that includes an encoder network and a decoder network to obtain a trained point cloud compression network.

[0067] The point cloud compression model includes an encoder network and a decoder network. The encoder network includes a semantic label mapping module, a semantic scene graph construction module, a FiLM (Feature-wise Linear Modulation) generator, a scene graph encoding module, an octree construction module, a feature conditional enhancement module, a cosine attention probability prediction module, and an arithmetic encoding module. The decoder network includes an arithmetic decoding module, a feature conditional enhancement module, a cosine attention probability prediction module, a scene graph decoding module, spatial indexing and querying, a FiLM generator, and an octree decoding module.

[0068] Specifically, such as Figure 2As shown, the point cloud compression model proposed in this embodiment has two parts: an encoder network and a decoder network, which are used to encode point cloud data into a bitstream and decode the encoded bitstream into a reconstructed point cloud, respectively. First, the input point cloud data is semantically labeled to obtain point clouds with semantic labels, and a semantic scene graph is constructed based on this through a scene graph construction module. On one hand, the semantic scene graph is input into a FiLM generator to generate semantic modulation parameters for modulating octree node features. On the other hand, the semantic scene graph is encoded by a scene graph encoding module to generate a semantic bitstream. In parallel, an octree representation is constructed from the point cloud data to obtain an octree representation, which is then input into a semantic feature enhancement module along with the semantic modulation parameters for feature fusion. The fused features are then used by a cosine attention probability prediction module to calculate the octagonal node octagonal probability, and a geometric bitstream is generated based on the octagonal probability through an arithmetic encoding module. In the decoding network, the geometric bitstream and the semantic bitstream are decoded respectively to obtain the octagonal node occupancy code and the semantic information matching the octagonal node recovered through spatial query and indexing. The FiLM generator generates semantic modulation parameters for the decoding stage based on the recovered semantic information, and the parameters and the octagonal node occupancy code are input together into the semantic feature enhancement module and the cosine attention probability prediction module for feature recovery processing. Finally, the point cloud is reconstructed by combining the octagonal decoding module and the scene graph decoding module, which can effectively achieve point cloud compression.

[0069] The input point cloud is processed by the semantic label mapping module and then fed in parallel into the scene graph construction module and the octree construction module. The octree construction module recursively partitions and quantizes the point cloud to generate an octree representation of the point cloud. The scene graph construction module generates a scene graph based on the semantic information and spatial relationships of the point cloud. Figure 1On one hand, the input is fed into the FiLM generator to generate feature condition parameters for modulating octree node features. On the other hand, it is fed into the scene graph encoding module for semantic information encoding. The octree representation and feature condition parameters are fused in the feature condition enhancement module to obtain a conditional node representation. The octree representation is used by the context building module to generate a context window, and the context window and the conditional node representation are input into the cosine attention probability prediction module to calculate the occupancy probability distribution of child nodes. Then, the arithmetic encoding module performs entropy encoding on the octree occupancy information based on the occupancy probability distribution to generate a geometric bitstream. The geometric bitstream and the semantic bitstream generated by the scene graph encoding module together constitute the point cloud compression output. At the decoding end, arithmetic decoding is performed on the bitstream and the octree decoding module reconstructs the octree structure of the point cloud. At the same time, the scene graph is restored by the scene graph parsing module and the FiLM generator generates feature condition parameters for the decoding stage. With the cooperation of the feature condition enhancement module and the cosine attention probability prediction module, the node feature representation is restored. Finally, the reconstructed octree structure is mapped back to the point cloud space through spatial query and indexing to obtain the reconstructed point cloud.

[0070] In a specific embodiment, the scene graph construction module sequentially includes semantic clustering and object extraction, geometric query and label matching, specifically including:

[0071] In semantic clustering and object extraction, the point set obtained by clustering (in To calculate the properties of its axis-aligned bounding box, first obtain the minimum and maximum coordinates of the object along the three axes from the point cloud file clustering, and then obtain the geometric center of the object from the minimum and maximum coordinates. That is, the center of the bounding box, the spatial extent vector of the object. By the object in The difference between the maximum and minimum coordinate values ​​in the three dimensions determines the object's length, width, and height. Specific formula:

[0072] ;

[0073] in, and Represent the object point set respectively All points in The minimum and maximum values ​​of the coordinates along the axis; similarly, for and By performing the same extreme value statistics on the coordinate components of the direction, the minimum boundary coordinate vector in three-dimensional space can be obtained. With the maximum boundary coordinate vector .

[0074] Furthermore, the geometric center of the object is calculated using the boundary coordinates. The calculation formula is as follows:

[0075] ;

[0076] in The geometric center coordinates of the axis-aligned bounding box of the semantic object are represented.

[0077] Furthermore, the spatial extent vector of the object The coordinates of the object are determined by the difference between its maximum and minimum coordinates in three dimensions, calculated using the following formula:

[0078] ;

[0079] in This represents the spatial extent vector of the object.

[0080] In the internal query and label matching process, the encoder determines whether the current octree node to be encoded falls inside a previously extracted semantic bounding box. If the octree node... Located in semantic objects Inside the bounding box, the node is assigned to the object. semantic tags The specific calculation formula is as follows:

[0081] ;

[0082] in, This represents the spatial matching function between octree nodes and semantic bounding boxes. This represents the index for traversing the coordinate axes in three-dimensional space. This indicates that the current octree node to be encoded is in Spatial coordinate values ​​along the axis, and They represent the first semantic object bounding box exist Minimum and maximum boundary coordinates along the axis.

[0083] In a specific embodiment, the FiLM generator aims to conditionally modulate the geometric features of octree nodes using semantic information provided by the semantic scene graph. This process mainly includes three steps: first, semantic vector embedding; second, obtaining modulation parameters from the obtained semantic vector through the FiLM generator; and finally, embedding semantic features into the input features through feature affine transformation. The specific implementation is as follows:

[0084] Semantic vector embedding first involves obtaining the discrete semantic labels corresponding to the octree nodes output by the scene graph construction module. This is achieved by mapping the integer label to a continuous semantic descriptor vector through a learnable embedding layer. The calculation formula is as follows:

[0085] ;

[0086] in, Indicates that there are learnable parameters Embedded layer, This is the extracted high-dimensional semantic feature vector.

[0087] Adjustment parameter generation, utilizing the semantic descriptor Scaling factors for feature modulation are generated using two independent multilayer perceptrons. and offset factor This process achieves a nonlinear mapping from the semantic space to the feature modulation space. The specific formula is as follows:

[0088] ;

[0089] ;

[0090] in, and It means Output results obtained through MLP and These represent the neural network branches used to generate the scaling and offset coefficients, respectively.

[0091] Feature affine transformation, and finally, the generated modulation parameters are used to transform the octree geometric features of the original input. Performing element-wise affine transformations injects semantic information into the geometric features, resulting in modulated output features. The specific calculation formula is as follows:

[0092] ;

[0093] in, This indicates element-wise multiplication. This indicates element-wise addition.

[0094] In a specific implementation, the octree construction module mainly includes three steps: spatial discretization, node attribute extraction, and finally, breadth-first search-based serialization to generate node features. The specific implementation is as follows:

[0095] Spatial discretization maps floating-point point cloud coordinates to an integer grid of a preset size. This is based on a preset octree depth. Each point in the original point cloud file is subjected to coordinate quantization to obtain integer coordinates. The quantification formula is as follows:

[0096] ;

[0097] in, and The axis coordinates are mapped in the same way. Indicates the original point cloud in Coordinates on the axis This represents the depth of the octree.

[0098] In calculating node attributes and occupancy information, starting from the root node, the cube containing the points is recursively divided into 8 sub-cubes. For the current level... To determine which child quadrant the current point falls within, the index value is calculated based on the offset of the current point's spatial coordinates relative to the center of the parent node. The calculation formula is as follows:

[0099] ;

[0100] in, This represents the spatial bisection index along each coordinate axis. When the value is 0, it means that the point is located in the negative half space of the plane dividing the current node. When the value is 1, it means that the point is located in the positive half space.

[0101] Furthermore, an 8-bit binary occupancy code is generated based on the non-empty state of the child nodes. , is represented as:

[0102] ;

[0103] Among them, the occupancy code Used to characterize the geometric features of the current node. This represents the spatial index of the child node. Indicates an indicator function, This represents the weighting coefficient of the binary bits.

[0104] Based on breadth-first search (BFS) serialization, a BFS strategy is used to traverse the non-empty nodes of each level of the octree, constructing a continuous sequence of node features. Each node in the sequence contains three key pieces of information: its occupancy state, its depth, and its index within its parent node. The serialized input feature matrix... Represented as:

[0105] ;

[0106] in, Represents all non-empty nodes eigenvectors The set, This indicates a breadth-first search operation. This represents the serialized input feature matrix.

[0107] In specific embodiments, such as Figure 3 As shown, the feature condition enhancement module (or local enhancement context coding module) includes local sensing units and sensing enhancement residual blocks.

[0108] The local perceptual unit (LPU) uses depthwise convolution to extract neighborhood relevance, as shown in the following formula:

[0109] ;

[0110] in, This represents a depthwise convolution with a kernel size of 3. This represents pointwise convolution with a kernel of 1.

[0111] The perceptual enhancement residual block utilizes multi-scale branches to extract richer features. It consists of two branches: branch one provides a wide receptive field, and branch two provides narrow receptive field and channel transformation capabilities. Then, Concat performs channel concatenation and feature fusion to obtain the final output.

[0112] The formula for branch one is expressed as follows:

[0113] ;

[0114] in, This represents a depthwise convolution with a kernel size of 3. This represents pointwise convolution with a kernel of 1. express Activation function.

[0115] The formula for branch two is expressed as follows:

[0116] ;

[0117] in, This represents pointwise convolution with a kernel of 1. express Activation function.

[0118] Feature fusion and output are represented by the following formula:

[0119] ;

[0120] in, This indicates a concatenation operation at the channel dimension.

[0121] In a specific embodiment, the cosine attention probability prediction module (or adaptive length sliding window attention module) sequentially includes an adaptive sliding window layer, a gated enhancement channel mixer, and a potential guiding node occupancy predictor.

[0122] The data with enhanced features is input into the cosine attention probability prediction module to obtain the probability prediction of the nodes, as follows.

[0123] The data with enhanced features is input into an adaptive sliding window layer to obtain the attention scores of the features, such as... Figure 4 As shown, the specific process is as follows:

[0124] Feature generation based on lightweight convolution, input features First, a query vector is generated using a depthwise separable convolutional structure. key vector Sum value vector ,pass Extract the intra-channel spatial context of the point sequence. The following formula is used to achieve information aggregation between channels:

[0125] , ;

[0126] in, express Depth convolution; express Pointwise convolution.

[0127] Enhanced multi-head cosine attention computation, introducing learnable query embeddings. The attention weights are optimized using a scaling factor. The query vector and the first Attention score between key vectors The calculation formula is as follows:

[0128] ;

[0129] in, and These are the sub-head feature vectors after multi-head partitioning; Learnable query embedding tensors are used to inject semantic prior biases; This indicates the calculation of cosine similarity, used to enhance numerical stability; The length of the sliding window. Learnable scaling parameters Together they constitute the adaptive length scaling factor; This is a mask matrix used to implement sliding window constraints and causal masking.

[0130] Finally, by weighted aggregation and concatenation mapping of the outputs of each sub-head, the module output features are obtained. The calculation formula is as follows:

[0131] ;

[0132] in, For the value vector at the corresponding position, This is a splicing operation at the channel dimension.

[0133] The attention score is fed into a gated enhancement channel mixer (or spatially gated enhancement channel mixer) to obtain further features, such as... Figure 5 As shown, the details are as follows:

[0134] The gated enhancement channel mixer sequentially proceeds through channel expansion projection, channel segmentation, spatial feature extraction and activation, and gated residual fusion. The specific process is as follows:

[0135] Channel expansion and projection, using Pointwise convolution projects features into a high-dimensional space to increase the channel dimension and obtain richer feature representations, providing sufficient parameter space for the gating mechanism.

[0136] Channel segmentation and branch generation divide the projected high-dimensional features into two parts along the channel dimension on an average basis: the first feature branch. Controlling the throughput of information flow and the second feature branch The spatial features to be processed are specifically represented by the following formula:

[0137] ;

[0138] in, As a gating signal As a transformation branch.

[0139] Spatial feature extraction and nonlinear activation, targeting Branches, applications Depthwise convolution extracts contextual information from the local space, and then the SiLU activation function is introduced for nonlinear transformation. The calculation formula is as follows:

[0140] ;

[0141] in, express Activation function This indicates element-wise multiplication.

[0142] Gated fusion and residual join, utilizing the element-wise multiplication operation in the above formula, actually utilize... Dynamic scaling The channel information is used to achieve adaptive information flow control. Finally, the features are restored to their original dimensions through output projection and compared with the input features. Perform residual connections to obtain the final output. :

[0143] ;

[0144] The obtained features are fed into the potential guiding node occupancy predictor to obtain the predicted probability of each node, such as... Figure 5 As shown, the details are as follows:

[0145] The latent guiding node occupancy predictor mainly includes linear projection initialization, implicit Gaussian feature interaction, and final probability generation; the specific processing flow of the latent guiding node occupancy predictor is as follows:

[0146] Linear projection initialization, using the features output by the gated hybrid module. Input fully connected layer Projecting features onto the initial state This adjusts the feature distribution space, laying the foundation for subsequent convolution operations. Then, it sequentially passes through... depthwise convolution and Pointwise convolution Extracting intermediate features The specific formula is expressed as follows:

[0147] ;

[0148] ;

[0149] in, This indicates a fully connected layer operation. and These represent depthwise convolution and pointwise convolution, respectively.

[0150] Implicit Gaussian feature interaction, this step simulates nonlinear mappings in high-dimensional space through element-wise multiplication interactions. First, the features are calculated... gating activation value The initial state is then further processed through pointwise convolution and depthwise convolution for feature mixing, and finally, residual connections are used to combine the features. Adding the interaction features together yields the enhanced features. The specific formula is expressed as follows:

[0151] ;

[0152] in, express Activation function This indicates element-wise multiplication. This structure effectively preserves the original feature information and introduces higher-order nonlinear features.

[0153] The final probability generation utilizes enhanced features Predict the occupancy status of the current node's child nodes. First, project the features onto a 255-dimensional output space using an activation function and a linear layer. This dimension corresponds to the octree node after removing all empty states. The non-empty occupancy scenarios are considered. Finally, the probability is normalized using the Softmax function to obtain the final predicted probability distribution. The specific calculation formula is as follows:

[0154] ;

[0155] The probability distribution Used to guide geometric reconstruction in subsequent entropy encoding or decoding processes.

[0156] At the decoding end, arithmetic decoding is performed on the bitstream and the octree structure of the point cloud is reconstructed by the octree decoding module. At the same time, the scene graph parsing module recovers the scene graph and the FiLM generator generates the feature condition parameters for the decoding stage. With the cooperation of the feature condition enhancement module and the cosine attention probability prediction module, the node feature representation is recovered. Finally, the reconstructed octree structure is mapped back to the point cloud space through spatial query and indexing to obtain the reconstructed point cloud.

[0157] The process involves scene graph analysis and semantic information recovery, geometric feature recovery loop, and finally point cloud reconstruction, as detailed below:

[0158] Scene graph parsing and semantic information recovery involve decoding the semantic bitstream to reconstruct the semantic scene graph. The axis-aligned bounding box (AABB) attribute and corresponding semantic labels of each semantic object in the scene are recovered by decoding the semantic bitstream. The specific calculation formula is as follows:

[0159] ;

[0160] ;

[0161] in, and The minimum and maximum boundary coordinate vectors are represented. It represents the center of a geometric object. This represents a spatial range vector used to determine the length, width, and height of an object.

[0162] The spatial query and indexing module determines whether the spatial location of the octree node to be recovered falls within the reconstructed semantic bounding box. This is determined by the following formula:

[0163] ;

[0164] in, This represents the spatial matching function between the decoded node and the semantic bounding box. This represents the index for traversing the coordinate axes in three-dimensional space. Indicates the current decoding node is in Spatial coordinate values ​​along the axis, and They represent the first semantic object bounding box exist Minimum and maximum boundary coordinates along the axis.

[0165] Semantic modulation parameter generation involves a FiLM generator producing conditional parameters for feature recovery. These integer labels are then mapped to continuous semantic descriptor vectors through a learnable embedding layer. The calculation formula is as follows:

[0166] ;

[0167] in, Indicates that there are learnable parameters Embedded layer, This is the extracted high-dimensional semantic feature vector.

[0168] Adjusting parameter generation using semantic descriptors Scaling factors for feature modulation are generated using two independent multilayer perceptrons. and offset factor This process achieves a nonlinear mapping from the semantic space to the feature modulation space. The specific formula is as follows:

[0169] ;

[0170] ;

[0171] in, and It means Output results obtained through MLP and These represent the neural network branches used to generate the scaling and offset coefficients, respectively.

[0172] The octree decoding module then decodes the occupancy code of the current node based on the probability distribution provided by the cosine attention probability prediction module. To accurately predict the probability of the next level, the features of the current node need to be recovered. This involves retrieving the geometric features from the previous time step. The semantic parameters are input into this module, and the specific formula is as follows:

[0173] ;

[0174] Among them, This indicates element-wise multiplication. This indicates element-wise addition.

[0175] The modulated features are transmitted through local sensing units. and Afterwards, the channels are concatenated using the following formula:

[0176] ;

[0177] ;

[0178] ;

[0179] After being processed by a gated enhancement channel mixer, the features are used to generate the final probability distribution through a potential guiding node occupancy predictor. The specific calculation formula is as follows:

[0180] ;

[0181] When the octree is decoded to a leaf node, the integer grid coordinates of the point cloud are obtained. The point cloud reconstruction module performs the inverse process of spatial discretization, with the specific formula as follows:

[0182] ;

[0183] in, To quantize the step size, This represents the minimum coordinate offset. The final output is a reconstructed point cloud containing accurate geometry and latent semantic consistency.

[0184] S2, input the point cloud data to be compressed into the trained point cloud compression network, and use the geometric bit stream and semantic bit stream as the compressed point cloud data; the decoder network reconstructs the compressed point cloud data into reconstructed point cloud data.

[0185] The experimental environment in this embodiment includes a workstation equipped with an Intel(R) Xeon(R) Gold 6226R processor (2.90GHz), an NVIDIA RTX 4090 graphics card (24GB VRAM) and 128GB DDR4 memory, and the operating system is Ubuntu 20.04 LTS; the experiment uses the PyTorch deep learning framework and CUDA 11.8 acceleration is enabled.

[0186] The dataset used in this embodiment is SemanticKITTI, a large-scale LiDAR point cloud dataset for autonomous driving. It was obtained by scanning with a Velodyne HDL-64E sensor and contains a total of 45.49 million points. Sequences 00 to 10 are used as the training set, and sequences 11 to 21 are used as the test set.

[0187] The training process in this embodiment uses the Adam optimizer, with an initial learning rate set to 1e-4. A StepLR strategy is employed to decay the learning rate to 0.1 every 10 epochs. The batch size is set to 8, and the number of training epochs is 100. All training is repeated under the same random seed to ensure the stability and reproducibility of the results. The training loss function uses semantically weighted cross-entropy loss, as detailed below:

[0188] ;

[0189] in, Represents the loss function. This represents the total number of octree nodes in the current batch. Indicates the first The semantic labels of each node, This represents the predicted probability distribution of the potential boot node occupancy predictor output. Indicates the actual occupancy code. This represents the semantic weight coefficient.

[0190] like Figure 7As shown in Figure (a), this figure illustrates the rate-distortion performance curves on the SemanticKITTI dataset. The horizontal axis represents the number of bits per point, i.e., the compressed bitrate; a smaller value indicates a higher compression ratio. The vertical axis represents the point-to-point peak signal-to-noise ratio (PSNR), in decibels; a larger value indicates higher geometric quality of the reconstructed point cloud. The solid red line in the figure represents the performance curve of our method, and the dashed blue line represents the performance curve of the comparison method, OctAttention. It can be seen from the figure that, under different bitrate settings, the curve of our method is consistently higher than that of the comparison method. This means that under the same transmission bandwidth (bitrate), our method can recover a higher quality point cloud geometry; or, to achieve the same reconstruction quality (PSNR), our method requires fewer bits and has higher compression efficiency. Figure 7 As shown in Figure (b), the chamfer distance performance curves on the SemanticKITTI dataset are presented. The vertical axis represents the chamfer distance, i.e. Figure 7 In Figure (b), the Chamfer Distance, measured in millimeters (mm), measures the geometric error between the original and reconstructed point clouds. A smaller value indicates a smaller error and higher reconstruction accuracy. As shown in the figure, the reconstruction error of both methods gradually decreases with increasing bitrate (horizontal axis). However, across the entire bitrate range, the chamfer distance of this method (solid red line) is consistently lower than that of the contrasting OctAttention method (dashed blue line). Especially in the low bitrate range (e.g., below 0.5 bpp), this method maintains a lower geometric error, demonstrating that introducing semantic scene maps for feature modulation effectively reduces distortion of local geometric structures and significantly improves the geometric accuracy of point cloud reconstruction.

[0191] In summary, through quantitative experimental analysis on the SemanticKITTI dataset, our proposed method outperforms existing OctAttention methods in both objective evaluation metrics (PSNR and CD), verifying that our invention can effectively reduce the transmission bit rate while ensuring the quality of point cloud reconstruction, and has excellent rate-distortion performance.

[0192] like Figure 8 As shown, the present invention also discloses a dual-stream point cloud compression system based on semantic scene graphs, comprising:

[0193] The point cloud compression network construction and training module 801 is used to construct and train a point cloud compression network including an encoder network and a decoder network to obtain a trained point cloud compression network.

[0194] In the encoder network, the semantic label mapping module performs semantic label mapping on the point cloud data to be compressed to obtain a point cloud with semantic labels; the scene graph construction module constructs a semantic scene graph from the point cloud with semantic labels and inputs it into the first feature linear modulation generator to generate the first semantic modulation parameters; the scene graph encoding module encodes the semantic scene graph into a semantic bit stream; the octree construction module constructs an octree representation from the point cloud data to be compressed, and the octree representation and the first semantic modulation parameters are fused by the first feature condition enhancement module. The fused features are then used by the first cosine attention probability prediction module to calculate the octet probability of the octree nodes, and then encoded into a geometric bit stream by the arithmetic encoding module.

[0195] In the decoder network, the scene graph parsing module reconstructs the semantic bitstream to obtain a reconstructed semantic scene graph; the reconstructed semantic scene graph is then processed by the spatial query and indexing module to obtain the restored semantic information of the matching octree nodes, and input into the second feature linear modulation generator to generate the second semantic modulation parameters; the arithmetic decoding module decodes the geometric bitstream into an octree occupancy code; the octree occupancy code and the second semantic modulation parameters are input into the second feature condition enhancement module for feature fusion, and the fused features are processed by the second cosine attention probability prediction module to calculate the restored occupancy probability; the octree decoding module generates an octree structure based on the restored occupancy probability; and the octree structure is mapped to point cloud spatial coordinates based on the reconstructed semantic scene graph to obtain reconstructed point cloud data.

[0196] The point cloud compression and restoration module 802 is used to input the point cloud data to be compressed into the trained point cloud compression network. The encoder network encodes the point cloud data to be compressed into semantic bit streams and geometric bit streams; the decoder network decodes the corresponding semantic bit streams and geometric bit streams into reconstructed point cloud data.

[0197] The specific implementation of the semantic scene graph-based dual-stream point cloud compression system is the same as the semantic scene graph-based dual-stream point cloud compression method, and will not be described again in this embodiment.

[0198] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A two-stream point cloud compression method based on semantic scene graphs, characterized in that, Includes the following steps: S1. Construct and train a point cloud compression network that includes an encoder network and a decoder network to obtain a trained point cloud compression network. In the encoder network, the semantic label mapping module performs semantic label mapping on the point cloud data to be compressed to obtain a point cloud with semantic labels; the scene graph construction module constructs a semantic scene graph from the point cloud with semantic labels and inputs it into the first feature linear modulation generator to generate the first semantic modulation parameters; the scene graph encoding module encodes the semantic scene graph into a semantic bit stream. The octree construction module constructs an octree from the point cloud data to be compressed to obtain an octree representation. The octree representation is then fused with the first semantic modulation parameter in the first feature condition enhancement module. The fused features are then used by the first cosine attention probability prediction module to calculate the octet probability of the octree node, and then encoded into a geometric bit stream by the arithmetic coding module. In the decoder network, the scene graph parsing module reconstructs the semantic bitstream to obtain a reconstructed semantic scene graph; the reconstructed semantic scene graph is then processed by the spatial query and indexing module to obtain the restored semantic information of the matching octree nodes, and input into the second feature linear modulation generator to generate the second semantic modulation parameters; the arithmetic decoding module decodes the geometric bitstream into an octree occupancy code; the octree occupancy code and the second semantic modulation parameters are input into the second feature condition enhancement module for feature fusion, and the fused features are processed by the second cosine attention probability prediction module to calculate the restored occupancy probability; the octree decoding module generates an octree structure based on the restored occupancy probability; and the octree structure is mapped to point cloud spatial coordinates based on the reconstructed semantic scene graph to obtain reconstructed point cloud data. S2, input the point cloud data to be compressed into the trained point cloud compression network, where the encoder network encodes the point cloud data to be compressed into semantic bit stream and geometric bit stream; The decoder network decodes the corresponding semantic bitstream and geometric bitstream into reconstructed point cloud data.

2. The dual-stream point cloud compression method based on semantic scene graphs according to claim 1, characterized in that, The scene graph construction module is specifically used for: A semantic scene graph is constructed by semantically clustering the point cloud after semantic label mapping, and using the clustered point set as a semantic object to construct a semantic scene graph with semantic objects as nodes; the semantic object node attributes include semantic labels and axis-aligned bounding boxes that quantify the spatial range of the semantic object. Associating octree nodes with semantic scene graphs, if an octree node to be encoded falls within the axis-aligned bounding box space of a semantic object, assigning the corresponding semantic label to the octree node.

3. The dual-stream point cloud compression method based on semantic scene graphs according to claim 1, characterized in that, Both the first characteristic linear modulation generator and the second characteristic linear modulation generator can be represented as: ； ； ； in, Represents a semantic descriptor vector. Indicates the index of a semantic object in the semantic scene graph; Indicates that there are learnable parameters Embedded layer; This represents the discrete semantic label corresponding to the octree node; This represents the scaling factor used for feature modulation; Indicates the scaling factor used to generate the scaling factor. The neural network branch; This represents the offset factor used for feature modulation; Indicates the value used to generate offset coefficients The neural network branch; ； in, This indicates the modulated output characteristics. Indicates a distinguishing superscript; The octree geometric features representing the original input; This indicates element-wise multiplication; This indicates element-wise addition.

4. The dual-stream point cloud compression method based on semantic scene graphs according to claim 1, characterized in that, The feature fusion is represented as: ； ； ； ； in, This represents the enhanced output features; This represents a pointwise convolution with a kernel of 1. Indicates conditional input features; This represents a depthwise convolution with a kernel size of 3; This represents the output features of the wide receptive field branch in a multi-scale branching model; express Activation function; This represents the output characteristics of the narrow receptive field and channel transform branches in a multi-scale branch; This represents the output of the feature condition enhancement module; This indicates a concatenation operation at the channel dimension.

5. The dual-stream point cloud compression method based on semantic scene graphs according to claim 1, characterized in that, Both the first cosine attention probability prediction module and the second cosine attention probability prediction module include an adaptive sliding window layer, a gated enhancement channel mixer, and a potential guiding node occupancy predictor; the adaptive sliding window layer captures contextual relationships from the input feature sequence to obtain contextual features; The gated enhancement channel mixer enhances the context features in the channel dimension to obtain channel-enhanced features; the potential guiding node occupancy predictor maps the channel-enhanced features to occupancy probabilities.

6. The dual-stream point cloud compression method based on semantic scene graphs according to claim 5, characterized in that, The first cosine attention probability prediction module and the second cosine attention probability prediction module are both represented as follows: ； ； ； in, Represents the query vector; Represents the key vector; Represents a value vector; This represents a pointwise convolution with a kernel of 1. This represents a depthwise convolution with a kernel size of 3; Represents input features; Indicates the first The query vector and the first Attention score between key vectors; This represents the Softmax function; and These are the sub-head feature vectors after multi-head partitioning; Embed tensors for learnable queries; The length of the sliding window. For learnable scaling parameters; Indicates transpose; Indicates modulo; It is a mask matrix; Indicates the probability of occupancy; For the value vector at the corresponding position, This is a splicing operation at the channel dimension.

7. The dual-stream point cloud compression method based on semantic scene graphs according to claim 5, characterized in that, The gated enhancement channel mixer is represented as follows: ； ； ； in, Indicates a gating signal; Indicates a transformation branch; This indicates an average division along the channel dimension; This represents a pointwise convolution with a kernel of 1. Represents input features; Indicates the output characteristics of the transform branch; express Activation function; This represents a depthwise convolution with a kernel size of 3; This indicates the characteristics after channel enhancement; This indicates element-wise multiplication.

8. The dual-stream point cloud compression method based on semantic scene graphs according to claim 5, characterized in that, The potential boot node occupancy predictor is represented as: ； ； ； ； in, Indicates the initial state; Indicates fully connected layer operations; Represents input features; Indicates intermediate features; This represents a pointwise convolution with a kernel of 1. This represents a depthwise convolution with a kernel size of 3; Indicates enhanced features; express Activation function; This indicates element-wise multiplication; This represents the final predicted probability distribution; This represents the Softmax function.

9. The dual-stream point cloud compression method based on semantic scene graphs according to claim 1, characterized in that, The method of mapping the octree structure to point cloud spatial coordinates based on the reconstructed semantic scene graph to obtain reconstructed point cloud data is represented as follows: ； in, Indicates the quantization step size; Represents the integer grid coordinates of the point cloud when the octree is decoded to a leaf node; This represents the offset of the minimum coordinate value.

10. A dual-stream point cloud compression system based on semantic scene graphs, characterized in that, Including the following: The point cloud compression network construction and training module is used to construct and train a point cloud compression network including an encoder network and a decoder network to obtain a trained point cloud compression network. In the encoder network, the semantic label mapping module performs semantic label mapping on the point cloud data to be compressed to obtain a point cloud with semantic labels; the scene graph construction module constructs a semantic scene graph from the point cloud with semantic labels and inputs it into the first feature linear modulation generator to generate the first semantic modulation parameters; the scene graph encoding module encodes the semantic scene graph into a semantic bit stream. The octree construction module constructs an octree from the point cloud data to be compressed to obtain an octree representation. The octree representation is then fused with the first semantic modulation parameter in the first feature condition enhancement module. The fused features are then used by the first cosine attention probability prediction module to calculate the octet probability of the octree node, and then encoded into a geometric bit stream by the arithmetic coding module. In the decoder network, the scene graph parsing module reconstructs the semantic bitstream to obtain a reconstructed semantic scene graph; the reconstructed semantic scene graph is then processed by the spatial query and indexing module to obtain the restored semantic information of the matching octree nodes, and input into the second feature linear modulation generator to generate the second semantic modulation parameters; the arithmetic decoding module decodes the geometric bitstream into an octree occupancy code; the octree occupancy code and the second semantic modulation parameters are input into the second feature condition enhancement module for feature fusion, and the fused features are processed by the second cosine attention probability prediction module to calculate the restored occupancy probability; the octree decoding module generates an octree structure based on the restored occupancy probability; and the octree structure is mapped to point cloud spatial coordinates based on the reconstructed semantic scene graph to obtain reconstructed point cloud data. The point cloud compression and restoration module is used to input the point cloud data to be compressed into the trained point cloud compression network, where the encoder network encodes the point cloud data to be compressed into semantic bit streams and geometric bit streams. The decoder network decodes the corresponding semantic bitstream and geometric bitstream into reconstructed point cloud data.

Images