Dynamic point cloud compression method and device based on time series perception

Abstract

The embodiment of the present application provides a dynamic point cloud compression method and device based on time series perception, the method comprising: performing sparse convolution processing on two consecutive frames of input point cloud data to obtain point cloud features in a feature space, extracting local and global features of the representations of the two consecutive frames in the feature space and splicing and fusing them to obtain a motion vector, using the precise motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame; calculating the residual between the predicted frame and the actual frame and performing compression encoding to obtain a residual tensor, adding the predicted frame and the residual tensor to obtain a coarse-grained predicted feature, time-aligning and feature-fusing the coarse-grained predicted feature with the three-frame point cloud features stored in a buffer to obtain a fused predicted feature, and upsampling and reconstructing the fused predicted feature to obtain a final point cloud frame; the present application can effectively improve the accuracy and efficiency of point cloud data compression.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a dynamic point cloud compression method and device based on time series perception. Background Art

[0002] There are two types of dynamic point cloud compression methods: traditional methods and deep learning-based methods. In traditional methods, MPEG has approved two excellent point cloud compression standards, VPCC based on 2D video and GPCC based on 3D point cloud compression, but they are only methods for encoding static point clouds.

[0003] The two traditional methods in the prior art usually require manually designed feature extraction and representation methods, and therefore perform poorly when processing point cloud patterns containing complex, nonlinear, or highly abstract patterns. Summary of the invention

[0004] In response to the problems in the prior art, the present application provides a dynamic point cloud compression method and device based on time series perception, which can effectively improve the accuracy and efficiency of point cloud data compression.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a dynamic point cloud compression method based on time series perception, comprising:

[0007] Perform sparse convolution processing on the input two consecutive frames of point cloud data to obtain point cloud features in the feature space, and save the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in the buffer;

[0008] Extract local and global features of the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame;

[0009] The residual between the predicted frame and the actual frame is calculated and compressed and encoded to obtain a residual tensor, the predicted frame and the residual tensor are added to obtain a coarse-grained prediction feature, the coarse-grained prediction feature is time-aligned and feature-fused with the three-frame point cloud features stored in the buffer to obtain a fused prediction feature, and the fused prediction feature is upsampled and reconstructed to obtain a final point cloud frame.

[0010] Furthermore, the step of performing sparse convolution processing on two consecutive frames of input point cloud data to obtain point cloud features in the feature space includes:

[0011] Two consecutive frames of input point cloud data are processed using two serially connected downsampling modules, each of which includes two sparse convolutional layers and an IRN module for local feature analysis. The sparse convolutional layers are used to perform convolution operations on the point cloud data, and LeakyReLU is used as the activation function to generate a tensor representation of the point cloud in the feature space and a corresponding coordinate matrix;

[0012] The two consecutive frames of input point cloud data are mapped to a three-dimensional space grid to form a sparse tensor, and feature extraction is performed on the sparse tensor using a sparse convolution with a convolution kernel size of 3 and a step size of 1. The extracted features are subjected to feature dimensionality reduction through a maximum pooling layer to obtain point cloud features in the feature space.

[0013] Furthermore, the step of storing the point cloud features and corresponding motion vectors extracted from the last three consecutive frames in the buffer includes:

[0014] Establish a circular buffer of a fixed size, write each frame of point cloud feature data obtained through sparse convolution processing into the corresponding position of the circular buffer in time sequence, and overwrite the earliest written feature data when the buffer is full of three frames of feature data, so that the buffer always maintains the latest three frames of point cloud feature data;

[0015] Two independent storage spaces are created for the circular buffer, the first storage space is used to store point cloud feature data, and the second storage space is used to store motion vectors between corresponding frames. The data in the two storage spaces establish a corresponding relationship through frame indexes to ensure synchronous update of point cloud feature data and motion vectors.

[0016] Furthermore, extracting local and global features of the representations of the two consecutive frames in the feature space and splicing and fusing them to obtain a motion vector includes:

[0017] A multi-layer residual network is used to extract local features from the point cloud representation in the feature space. Each layer of the residual network contains three sparse convolutional layers and a skip connection. The convolution kernel size of the sparse convolutional layer is 3×3×3. Batch normalization and ReLU activation function are used after each sparse convolutional layer to perform nonlinear transformation on the features.

[0018] The local features are subjected to global maximum pooling to obtain global features, the global features are subjected to dimensionality reduction processing through two fully connected layers, the reduced global features are concatenated with the local features in the channel dimension, and the concatenated features are fused through a 1×1×1 convolutional layer to obtain a motion vector.

[0019] Furthermore, the compression encoding of the motion vector, splicing the decoded motion vector with the three-frame motion vectors stored in the buffer and obtaining the accurate motion vector through sparse convolution processing includes:

[0020] The motion vector is input into an autoencoder network composed of an encoder and a decoder, wherein the encoder comprises three convolutional layers to convert the motion vector into a low-dimensional feature code stream, and the decoder comprises three deconvolutional layers to restore the feature code stream into a decoded motion vector, and a residual connection is added between each layer of the encoder and the decoder to preserve the motion information;

[0021] The decoded motion vector and the three-frame motion vectors arranged in time sequence in the buffer are concatenated in the time dimension to form a four-dimensional tensor, which is processed using a three-layer sparse convolutional network. Each layer of the network uses a 3×3×3 convolution kernel and a LeakyReLU activation function, and the feature information of each layer is retained through jump connections to obtain an accurate motion vector.

[0022] Furthermore, the using the precise motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame includes:

[0023] The bilinear interpolation method is used to map the precise motion vector to each point of the point cloud feature of the previous frame, and the point cloud features are spatially transformed according to the mapped motion vector, and the transformed features are adaptively adjusted through the deformable convolution layer;

[0024] The transformed point cloud features are input into the compensation network for feature compensation. The compensation network consists of three sparse convolutional layers with residual connections. The convolution kernel size of each sparse convolutional layer is 3×3×3. The features are processed using instance normalization and PReLU activation function to obtain the predicted frame of the current frame.

[0025] Further, the residual between the predicted frame and the actual frame is calculated and compressed to obtain a residual tensor, the predicted frame and the residual tensor are added to obtain a coarse-grained prediction feature, the coarse-grained prediction feature is time-aligned and feature-fused with the three-frame point cloud features stored in the buffer to obtain a fused prediction feature, and the fused prediction feature is up-sampled and reconstructed to obtain a final point cloud frame, including:

[0026] The difference between the predicted frame and the actual frame in the feature space is calculated by using a subtraction operation to obtain a residual feature, and the residual feature is input into a residual coding network, wherein the residual coding network comprises three sparse convolutional layers and two deconvolutional layers, each layer uses LeakyReLU as an activation function, and the residual feature is compressed into a residual tensor through quantization processing, and the residual tensor is added to the predicted frame to obtain a coarse-grained prediction feature;

[0027] A self-attention mechanism is used to perform temporal modeling on the coarse-grained prediction features and the three-frame point cloud features in the buffer, and the correlation matrix between the features is calculated. The correlation matrix is ​​multiplied by the features to obtain the attention weights. The four-frame features are weighted fused according to the attention weights, and the fused features are upsampled through a three-layer transposed convolutional network with a step size of 2. The final point cloud frame is reconstructed using instance normalization and ReLU activation function.

[0028] In a second aspect, the present application provides a dynamic point cloud compression device based on time series perception, comprising:

[0029] The motion vector determination module is used to perform sparse convolution processing on the input two consecutive frames of point cloud data to obtain point cloud features in the feature space, and save the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in the buffer;

[0030] A prediction frame determination module is used to extract local and global features of the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a prediction frame of the current frame;

[0031] The point cloud frame determination module is used to calculate the residual between the predicted frame and the actual frame and perform compression encoding to obtain a residual tensor, add the predicted frame and the residual tensor to obtain a coarse-grained prediction feature, time-align and feature fuse the coarse-grained prediction feature with the three-frame point cloud features stored in the buffer to obtain a fused prediction feature, and upsample and reconstruct the fused prediction feature to obtain a final point cloud frame.

[0032] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the timing-aware dynamic point cloud compression method when executing the program.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dynamic point cloud compression method based on time series perception.

[0034] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the timing-aware dynamic point cloud compression method.

[0035] It can be seen from the above technical solution that the present application provides a dynamic point cloud compression method and device based on time series perception, which introduces an end-to-end dynamic point cloud compression framework, reduces information loss caused by occlusion by integrating multi-frame data, thereby improving the reconstruction quality, and uses motion vectors from different time instances for motion compensation, which significantly reduces information loss during the compression process, and fully combines the global information and local information of the point cloud to obtain more accurate motion information, thereby effectively improving the accuracy and efficiency of point cloud data compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is one of the flow charts of the dynamic point cloud compression method based on time sequence perception in the embodiment of the present application;

[0038] Figure 2 The second flowchart of the dynamic point cloud compression method based on time series perception in the embodiment of the present application;

[0039] Figure 3 The third flowchart of the dynamic point cloud compression method based on time sequence perception in the embodiment of the present application;

[0040] Figure 4 This is a fourth flow chart of a dynamic point cloud compression method based on time series perception in an embodiment of the present application;

[0041] Figure 5 This is a fifth flow chart of a dynamic point cloud compression method based on time sequence perception in an embodiment of the present application;

[0042] Figure 6 This is a sixth flow chart of a dynamic point cloud compression method based on time sequence perception in an embodiment of the present application;

[0043] Figure 7 FIG7 is a flow chart of a dynamic point cloud compression method based on time series perception in an embodiment of the present application;

[0044] Figure 8 It is a structural diagram of a dynamic point cloud compression device based on time series perception in an embodiment of the present application;

[0045] Fig. 9 This is a framework diagram of dynamic point cloud compression based on multi-frame time sequence perception in a specific embodiment of the present application;

[0046] Fig.10 This is a motion estimation framework diagram in a specific embodiment of the present application;

[0047] Fig.11 This is a multi-time motion vector fusion framework diagram in a specific embodiment of the present application;

[0048] Fig.12 A framework diagram for generating fusion prediction features based on multiple frames in a specific embodiment of the present application;

[0049] Fig.13 A comparative analysis diagram of D1-PSNR in a specific embodiment of the present application and the existing model;

[0050] Fig.14 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0051] Reference numerals:

[0052] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0055] Taking into account the problems existing in the prior art, the present application provides a dynamic point cloud compression method and device based on time series perception, which introduces an end-to-end dynamic point cloud compression framework, reduces information loss caused by occlusion by integrating multi-frame data, thereby improving the reconstruction quality, and uses motion vectors from different time instances for motion compensation, which significantly reduces information loss during the compression process. It fully combines the global information and local information of the point cloud to obtain more accurate motion information, thereby effectively improving the accuracy and efficiency of point cloud data compression.

[0056] In order to effectively improve the accuracy and efficiency of point cloud data compression, the present application provides an embodiment of a dynamic point cloud compression method based on time series perception, see Figure 1 The dynamic point cloud compression method based on time series perception specifically includes the following contents:

[0057] Step S101: performing sparse convolution processing on two consecutive frames of input point cloud data to obtain point cloud features in the feature space, and saving the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in a buffer;

[0058] Optionally, in this embodiment, this step elaborates on the feature extraction and cache management process of dynamic point cloud data, and realizes efficient point cloud feature representation through sparse convolution and multi-frame feature storage. This technical solution effectively solves the problems of data sparsity and insufficient utilization of temporal information in traditional point cloud processing.

[0059] In the feature extraction stage, the system first receives two consecutive frames of raw point cloud data. Each frame of point cloud data contains a large number of discrete three-dimensional coordinate points and possible attribute information. Considering the irregular distribution characteristics of point cloud data, it is more efficient to use sparse convolution for processing. Sparse convolution maps point cloud data to a regular three-dimensional grid and only performs convolution operations on non-empty grids, significantly reducing the amount of calculation and memory usage. When the convolution kernel slides in space, it only processes voxels containing actual points, avoiding invalid calculations on blank areas.

[0060] The sparse convolutional network adopts a multi-level feature extraction structure. The convolution kernel size of each layer is 3×3×3, and the downsampling operation with a step size of 2 is used to gradually expand the receptive field while reducing the feature dimension. LeakyReLU is used as the activation function in the network, which introduces nonlinear transformation while avoiding the problem of neuron death that may be caused by traditional ReLU. The introduction of the batch normalization layer effectively stabilizes the feature distribution and accelerates the convergence process of the network.

[0061] In order to make full use of the timing information, the system designs a circular buffer to store the feature data and corresponding motion vectors of the last three frames. The buffer adopts a first-in-first-out update strategy. When new feature data arrives, it automatically overwrites the earliest data to ensure that the latest three frames of information are always kept. This multi-frame data storage mechanism provides important support for subsequent timing feature analysis and motion prediction.

[0062] The buffer is divided into two independent storage spaces, one for storing point cloud features and the other for storing motion vectors. The two storage spaces establish a corresponding relationship through frame indexes to ensure the synchronous update of feature and motion information. This dual-space storage structure not only improves the efficiency of data access, but also facilitates subsequent feature fusion and motion compensation operations.

[0063] This technical solution effectively solves several key problems in point cloud compression by combining sparse convolution and multi-frame buffering.

[0064] Step S102: extract local and global features from the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame;

[0065] Optionally, in this embodiment, this step describes in detail a motion estimation and compensation process that integrates local and global features, and achieves high-precision point cloud motion prediction through multi-level feature extraction and motion vector optimization. This technical solution effectively solves the problems of insufficient accuracy and poor spatiotemporal consistency in traditional point cloud motion estimation.

[0066] In the feature extraction stage, the system first performs a multi-scale analysis on the representation of two consecutive frames of point clouds in the feature space. Local feature extraction uses a multi-layer residual network structure, each layer contains three sparse convolution layers in series, and the convolution kernel size is 3×3×3. The introduction of residual connections not only accelerates network training, but also effectively prevents the gradient vanishing problem of deep networks. Batch normalization and ReLU activation functions are configured after each convolution layer to enhance the nonlinear expression ability of features.

[0067] The global features are obtained by performing global maximum pooling on the local features, capturing the overall motion pattern and semantic information. Two fully connected layers perform dimensionality reduction on the global features, reducing the number of parameters while retaining key information. The local features and the reduced-dimensional global features are concatenated in the channel dimension, and feature fusion is performed through a 1×1×1 convolutional layer to obtain the initial motion vector.

[0068] The compression coding of motion vectors adopts a symmetrical codec structure. The encoder contains three convolutional layers, which gradually compresses the motion vectors into low-dimensional feature code streams, and the decoder reconstructs the motion information through three deconvolutional layers. The residual connection between the layers ensures the effective transmission of detail information and improves the reconstruction quality.

[0069] In order to improve the accuracy of motion estimation, the system concatenates the decoded motion vector with the three-frame historical motion vector stored in the buffer in the time dimension to form a four-dimensional tensor containing timing information. The tensor is processed by a three-layer sparse convolutional network, each layer uses a 3×3×3 convolution kernel and a LeakyReLU activation function, and cooperates with jump connections to retain multi-scale feature information, and finally obtains an accurate motion vector.

[0070] The motion compensation stage uses a bilinear interpolation method to map the precise motion vector to each point of the point cloud feature of the previous frame, realizing feature-level spatial transformation. The introduction of the deformable convolution layer enhances the flexibility of feature transformation and can adapt to complex motion patterns. The compensation network refines the features through three layers of sparse convolution with residual connections, and finally generates the predicted features of the current frame.

[0071] This technical solution significantly improves the performance of point cloud motion estimation by integrating local-global feature fusion and multi-frame motion information.

[0072] Step S103: Calculate the residual between the predicted frame and the actual frame and perform compression encoding to obtain a residual tensor, add the predicted frame and the residual tensor to obtain a coarse-grained prediction feature, time-align and feature-fuse the coarse-grained prediction feature with the three-frame point cloud features stored in the buffer to obtain a fused prediction feature, and upsample and reconstruct the fused prediction feature to obtain a final point cloud frame.

[0073] Optionally, in this embodiment, this step describes the residual encoding and feature reconstruction process of the point cloud frame, and achieves high-quality point cloud reconstruction through residual compression and temporal feature fusion. This technical solution effectively solves the problems of insufficient reconstruction quality and poor temporal consistency in traditional point cloud compression.

[0074] In the residual calculation stage, the system obtains the difference information between the predicted frame and the actual frame in the feature space through subtraction operations. This residual coding strategy is based on the fact that the predicted frame and the actual frame are highly correlated, which can significantly reduce the amount of information that needs to be encoded. The residual features are then input into a specially designed residual coding network for compression. The network consists of three sparse convolutional layers and two deconvolutional layers. Each layer uses LeakyReLU as the activation function, and the residual information is compressed into a compact residual tensor through quantization.

[0075] The residual tensor is added and fused with the predicted frame at the feature level to obtain preliminary coarse-grained prediction features. This addition fusion operation retains the main structure of the predicted features, while supplementing the detail changes through the residual information, laying the foundation for subsequent fine reconstruction.

[0076] The time series feature fusion uses the self-attention mechanism to perform spatiotemporal modeling of the coarse-grained prediction features and the three-frame historical features stored in the buffer. The attention mechanism adaptively assigns weights to features of different time frames by calculating the correlation matrix between features. The correlation matrix reflects the dependency between the current feature and the historical feature, and the high-correlation region obtains a larger attention weight. The attention weight is weightedly combined with the feature map through matrix multiplication to achieve feature fusion based on similarity.

[0077] In the feature reconstruction stage, a three-layer transposed convolutional network is used for upsampling, with a step size of 2, to gradually restore the spatial resolution of the features. The instance normalization layer is used in the network to standardize the feature distribution, and the ReLU activation function is used to introduce nonlinear transformations. The feature size after each layer of transposed convolution is doubled until the resolution requirement of the original point cloud is reached. At the same time, cross-layer feature connections are considered in the network design to ensure the effective combination of high-level semantic information and low-level detail features.

[0078] This technical solution significantly improves the performance of point cloud compression reconstruction through residual coding and temporal feature fusion.

[0079] From the above description, it can be seen that the dynamic point cloud compression method based on time series perception provided in the embodiment of the present application can introduce an end-to-end dynamic point cloud compression framework, reduce the information loss caused by occlusion by integrating multi-frame data, thereby improving the reconstruction quality, and use motion vectors from different time instances for motion compensation, which significantly reduces the information loss during the compression process, and fully combines the global information and local information of the point cloud to obtain more accurate motion information, thereby effectively improving the accuracy and efficiency of point cloud data compression.

[0080] In one embodiment of the time-series-aware dynamic point cloud compression method of the present application, see Figure 2 , and can also include the following:

[0081] Step S201: using two serially connected downsampling modules to process two consecutive frames of input point cloud data, each downsampling module comprising two sparse convolutional layers and an IRN module for local feature analysis, performing convolution operations on the point cloud data through the sparse convolutional layers, and using LeakyReLU as an activation function to generate a tensor representation of the point cloud in the feature space and a corresponding coordinate matrix;

[0082] Step S202: Map the two consecutive frames of point cloud data input into a three-dimensional space grid to form a sparse tensor, perform feature extraction on the sparse tensor using a sparse convolution with a convolution kernel size of 3 and a step size of 1, perform feature dimensionality reduction on the extracted features through a maximum pooling layer, and obtain point cloud features in the feature space.

[0083] Optionally, in this embodiment, these two steps describe in detail the multi-scale feature extraction process of point cloud data, and achieve efficient point cloud feature representation through downsampling and sparse convolution. This technical solution effectively solves the problems of low computational efficiency and insufficient feature expression ability in traditional point cloud processing.

[0084] In step S201, the system uses a cascaded double downsampling architecture to process the input point cloud sequence. Each downsampling module contains two sparse convolutional layers, the convolution kernel size is set to 3×3×3, the step size is 2, and the local structural features are captured through a spatial sliding window. The convolution layer uses LeakyReLU as the activation function, and the small slope of its negative semi-axis effectively avoids the problem of neuron death and enhances the nonlinear expression ability of the network. In particular, the introduction of the IRN (Iterative Residual Network) module realizes the iterative refinement of local features. This module gradually improves the discriminability of features through residual connections and multiple iterative operations.

[0085] After the point cloud data is processed by the first downsampling module, the spatial resolution is reduced by half, but the number of feature channels increases, achieving effective information compression. The second downsampling module further reduces the spatial dimension while increasing the feature dimension to form a hierarchical feature representation. The output of each module contains two key components: the feature tensor and the coordinate matrix. The former encodes the semantic information of the point cloud, and the latter stores the spatial position information of the point.

[0086] Step S202 focuses on the gridding and feature extraction of point cloud data. The system first maps the point cloud data into a regular three-dimensional grid and uses a spatial hash function to achieve fast mapping from points to voxels. This gridding means converting irregular point clouds into regular sparse tensors, which is convenient for subsequent convolution operations.

[0087] The gridded data is subjected to fine feature extraction through sparse convolution with small convolution kernels (3×3×3) and small stride (1×1×1). This setting ensures that local features are fully captured while maintaining computational efficiency. Convolution operations are performed only on non-empty voxels, significantly reducing computational complexity. The introduction of the maximum pooling layer achieves feature dimensionality reduction and retention of key information. The pooling window size is 2×2×2 and the stride is 2, which compresses the feature dimension while retaining key information.

[0088] This technical solution significantly improves the performance of point cloud processing through multi-scale feature extraction and sparse computing.

[0089] In one embodiment of the time-series-aware dynamic point cloud compression method of the present application, see Figure 3 , and can also include the following:

[0090] Step S301: Establish a circular buffer of a fixed size, write each frame of point cloud feature data obtained through sparse convolution processing into the corresponding position of the circular buffer in time sequence, and overwrite the earliest written feature data when the buffer is full of three frames of feature data, so that the buffer always maintains the latest three frames of point cloud feature data;

[0091] Step S302: Create two independent storage spaces for the circular buffer, the first storage space is used to store point cloud feature data, and the second storage space is used to store motion vectors between corresponding frames. The data in the two storage spaces establish a corresponding relationship through frame indexes to ensure synchronous update of point cloud feature data and motion vectors.

[0092] Optionally, in this embodiment, these two steps describe in detail the cache management mechanism of point cloud sequence data, and realize efficient time series data management through circular buffer and dual space storage strategy. This technical solution effectively solves the data synchronization and memory management problems in point cloud sequence processing.

[0093] In step S301, the system designs a fixed-capacity circular buffer for storing continuous point cloud feature data. The buffer adopts a circular queue data structure, and manages data writing and overwriting operations through head pointers and tail pointers. After each frame of point cloud data is processed by sparse convolution, the generated feature data will be written to the next available position in the buffer. When new data is written, the system automatically updates the tail pointer position; when the buffer reaches the three-frame capacity limit, the new data will overwrite the earliest written data, and the head pointer will be updated accordingly. This circular overwriting mechanism ensures that the buffer always keeps the latest three frames of data, achieving efficient memory utilization.

[0094] In order to optimize data access efficiency, the buffer uses memory alignment technology to ensure that each frame of feature data is stored continuously in memory. At the same time, a double buffering mechanism is implemented to allow feature data read and write operations to be performed in parallel, avoiding data read and write conflicts. The size of the buffer is dynamically adjusted according to the feature dimension, minimizing memory usage while ensuring data integrity.

[0095] Step S302 further optimizes the storage structure and introduces a dual-space storage strategy. The first storage space is specifically used to store point cloud feature data, including the spatial coordinates and semantic feature information of the points. Each frame of feature data is organized into a structured tensor form to facilitate subsequent feature extraction and analysis. The second storage space is specifically used to store motion vectors between adjacent frames, recording the dynamic change information of the point cloud.

[0096] The two storage spaces are associated through a unified frame index mechanism, and each feature data and its corresponding motion vector share the same index identifier. This index mechanism is implemented using a hash table to support fast data retrieval and update. When new feature data is written, the system automatically updates the corresponding motion vector to ensure the synchronization of the two types of data. In order to improve data consistency, the system implements an atomic operation mechanism to prevent data inconsistency caused by concurrent access.

[0097] This technical solution significantly improves the performance of point cloud sequence processing through efficient cache management and dual-space storage.

[0098] In one embodiment of the time-series-aware dynamic point cloud compression method of the present application, see Figure 4 , and can also include the following:

[0099] Step S401: using a multi-layer residual network to extract local features from the point cloud representation in the feature space, each layer of the residual network includes three sparse convolutional layers and one skip connection, the convolution kernel size of the sparse convolutional layer is 3×3×3, and batch normalization and ReLU activation function are used after each sparse convolutional layer to perform nonlinear transformation on the features;

[0100] Step S402: Perform global maximum pooling on local features to obtain global features, perform dimensionality reduction processing on the global features through two fully connected layers, concatenate the reduced dimensionality global features with the local features in the channel dimension, and fuse the concatenated features through a 1×1×1 convolution layer to obtain a motion vector.

[0101] Optionally, in this embodiment, these two steps describe in detail the local-global feature extraction and fusion process based on the residual network, and achieve high-precision motion vector estimation through multi-scale feature learning. This technical solution effectively solves the problems of insufficient expression ability and difficulty in feature fusion in traditional point cloud feature extraction.

[0102] Step S401 uses a multi-layer residual network architecture to extract local features. Each layer of the residual network contains three sparse convolutional layers connected in series to form a main path branch. The convolution kernel uses a 3×3×3 cube structure to extract features in three-dimensional space and effectively capture the local geometric structure of the point cloud. A batch normalization layer is configured after each convolution layer to standardize the features, accelerate network convergence and improve model stability. The introduction of the ReLU activation function adds nonlinear transformation capabilities to the network while keeping the calculation simple and efficient.

[0103] The core of the residual network lies in the design of the jump connection, which establishes a direct path outside the main path to add the input features directly to the output features. This structural design effectively alleviates the gradient vanishing problem of deep networks, enables the network to learn residual mapping, and improves the accuracy of feature extraction. At the same time, the jump connection ensures that the underlying feature information can be directly transmitted to the upper layer, enhancing the network's feature expression ability.

[0104] Step S402 realizes the effective fusion of local features and global features. First, the global maximum pooling operation is performed on the extracted local features. The pooling operation is performed on the entire feature map, and the maximum value of each channel is selected to obtain a feature vector reflecting global information. This pooling strategy retains the most significant feature response and provides scene-level context information.

[0105] The global features are reduced in dimension through two fully connected layers. The first fully connected layer reduces the feature dimension to 1 / 4 of the original, and the second fully connected layer further compresses it to 1 / 8. The dimension reduction process not only reduces the number of parameters, but also removes redundant information and extracts the most discriminative global feature representation. The reduced global features are concatenated with the original local features in the channel dimension to form a multi-scale feature representation.

[0106] Finally, a 1×1×1 point convolution layer is used to fuse the concatenated features. This convolution layer integrates information across channels, adaptively learns the importance weights of local and global features, and generates the final motion vector. The lightweight nature of point convolution ensures the computational efficiency of feature fusion.

[0107] This technical solution significantly improves the performance of motion estimation through deep residual learning and multi-scale feature fusion.

[0108] In one embodiment of the time-series-aware dynamic point cloud compression method of the present application, see Figure 5 , and can also include the following:

[0109] Step S501: input the motion vector into an autoencoder network composed of an encoder and a decoder, wherein the encoder includes three convolutional layers to convert the motion vector into a low-dimensional feature code stream, and the decoder includes three deconvolutional layers to restore the feature code stream to the decoded motion vector, and a residual connection is added between each layer of the encoder and the decoder to maintain motion information;

[0110] Step S502: The decoded motion vector is concatenated with the three frames of motion vectors arranged in time sequence in the buffer in the time dimension to form a four-dimensional tensor, and the four-dimensional tensor is processed using a three-layer sparse convolutional network. Each layer of the network uses a 3×3×3 convolution kernel and a LeakyReLU activation function, and the feature information of each layer is retained through jump connections to obtain an accurate motion vector.

[0111] Optionally, in this embodiment, these two steps describe in detail the compression encoding and refined reconstruction process of the motion vector, and high-quality motion information processing is achieved through the autoencoder network and temporal feature fusion. This technical solution effectively solves the problem of motion information compression and temporal consistency modeling.

[0112] In step S501, the system uses a symmetric autoencoder architecture to process motion vectors. The encoder contains three consecutive convolutional layers with increasing kernel sizes (3×3, 5×5, 7×7), a step size of 2, and increasing number of channels (64, 128, 256). This progressive receptive field expansion strategy enables the network to capture multi-scale motion patterns. Instance normalization and PReLU activation function are used after each convolutional layer to enhance the expressive power of features. The encoding process compresses high-dimensional motion vectors into a compact feature code stream, achieving efficient representation of data.

[0113] The decoder adopts a mirror-symmetric structure and contains three deconvolution layers with decreasing number of channels (256, 128, 64). The deconvolution operation is implemented using transposed convolution with a step size of 2 to gradually restore the spatial resolution of the features. In particular, residual connections are established between the corresponding layers of the encoder and decoder, and the features of the encoding layer are directly added to the output of the decoding layer. This cross-layer connection mechanism effectively prevents information loss and ensures accurate reconstruction of motion details.

[0114] Step S502 implements the temporal fusion and refinement of motion vectors. The system first splices the reconstructed motion vector with the three frames of historical motion vectors stored in the buffer in the temporal dimension to form a four-dimensional tensor with dimensions [batch, channel, time, space]. This temporal splicing operation provides a complete motion context for subsequent spatiotemporal modeling.

[0115] The concatenated four-dimensional tensor is input into a three-layer sparse convolutional network for processing. Each layer of the network uses a 3×3×3 three-dimensional convolution kernel to effectively model motion correlation in spatial and temporal dimensions. The network uses LeakyReLU as the activation function, and its small slope in the negative region helps the network maintain gradient flow and improve training stability. In particular, the original feature information is retained after each layer of convolution and passed to the next layer through skip connections to ensure the integrity of motion information.

[0116] This technical solution significantly improves the performance of motion estimation through deep feature learning and timing modeling.

[0117] In one embodiment of the time-series-aware dynamic point cloud compression method of the present application, see Figure 6 , and can also include the following:

[0118] Step S601: using a bilinear interpolation method to map the precise motion vector to each point of the point cloud feature of the previous frame, performing spatial transformation on the point cloud feature according to the mapped motion vector, and adaptively adjusting the transformed feature through a deformable convolution layer;

[0119] Step S602: Input the transformed point cloud features into the compensation network for feature compensation. The compensation network consists of three sparse convolutional layers with residual connections. The convolution kernel size of each sparse convolutional layer is 3×3×3. The features are processed using instance normalization and PReLU activation function to obtain a predicted frame of the current frame.

[0120] In one embodiment of the time-series-aware dynamic point cloud compression method of the present application, see Figure 7 , and can also include the following:

[0121] Optionally, in this embodiment, these two steps describe in detail the point cloud frame prediction process based on motion compensation, and achieve high-quality inter-frame prediction through precise feature mapping and adaptive compensation. This technical solution effectively solves the problem of motion compensation and feature alignment in point cloud sequences.

[0122] Step S601 first realizes the precise mapping of motion vectors to point cloud features. The bilinear interpolation method resamples the precise motion vectors in the spatial dimension to ensure that each point cloud feature point can obtain the corresponding motion information. The interpolation process takes into account the motion information of adjacent grid points, and calculates the motion vector of the target position by distance weighting, ensuring the smoothness and continuity of the motion field. Based on the motion vector obtained by mapping, the system performs spatial transformation on the point cloud features. The transformation operation includes two components, translation and rotation, which accurately reflects the motion state of the point cloud.

[0123] In order to handle complex non-rigid deformations, the system introduces a deformable convolution layer to refine the transformed features. Deformable convolution learns additional offset fields so that the sampling position of the convolution kernel can be adaptively changed to better adapt to local geometric deformations. This adaptive mechanism significantly improves the flexibility of feature transformation, especially when dealing with local deformation and occlusion.

[0124] Step S602 designs a special compensation network to correct errors and omissions in feature transformation. The compensation network adopts a three-layer cascaded sparse convolution structure. The number of input channels and output channels of each layer is consistent to maintain the expression dimension of the feature. The 3×3×3 convolution kernel extracts features in three-dimensional space and effectively captures local geometric structure information. The residual connection adds the input features of each layer directly to the output features. This design makes it easier for the network to learn the residual representation of the features and improves the accuracy of compensation.

[0125] The instance normalization layer independently normalizes the features of each sample, eliminating the interference between samples in the batch and improving the independence of feature expression. The PReLU activation function provides the network with more flexible nonlinear transformation capabilities through the learnable negative semi-axis slope parameter, which helps to fine-tune the features. The output of the last layer is directly used as the prediction frame to complete the inter-frame prediction task of the point cloud sequence.

[0126] This technical solution significantly improves the performance of point cloud sequence prediction through precise motion compensation and feature adjustment.

[0127] Step S701: using a subtraction operation to calculate the difference between the predicted frame and the actual frame in the feature space to obtain a residual feature, inputting the residual feature into a residual coding network, wherein the residual coding network comprises three sparse convolutional layers and two deconvolutional layers, each layer using LeakyReLU as an activation function, compressing the residual feature into a residual tensor through quantization processing, and performing an addition operation on the residual tensor and the predicted frame to obtain a coarse-grained prediction feature;

[0128] Step S702: Use the self-attention mechanism to perform time series modeling on the coarse-grained prediction features and the three-frame point cloud features in the buffer, calculate the correlation matrix between the features, perform matrix multiplication of the correlation matrix and the features to obtain the attention weights, perform weighted fusion of the four-frame features according to the attention weights, upsample the fused features through a three-layer transposed convolutional network with a step size of 2, and use instance normalization and ReLU activation function to reconstruct the final point cloud frame.

[0129] Optionally, in this embodiment, these two steps describe in detail the point cloud frame reconstruction process based on residual coding and temporal attention, and achieve high-quality point cloud reconstruction through efficient feature compression and temporal modeling. This technical solution effectively solves the problem of point cloud compression and temporal consistency modeling.

[0130] Step S701 first calculates the residual between the predicted frame and the actual frame in the feature space. This differential operation highlights the areas that need to be coded and improves the compression efficiency. The residual coding network adopts an asymmetric codec structure. The three sparse convolutional layers are responsible for feature extraction and dimensionality reduction. The convolution kernel size is 3×3×3, and the number of channels increases successively (64, 128, 256). The LeakyReLU activation function maintains a small slope in the negative area, which helps to maintain gradient flow and feature transfer.

[0131] The two deconvolution layers use a transposed convolution operation with a stride of 2 to gradually restore the spatial resolution of the features, and the number of channels decreases successively (128, 64). In particular, quantization is introduced in the encoding process to discretize the continuous residual features into a limited representation level, which greatly reduces the storage and transmission overhead. The quantized residual tensor is fused with the predicted frame through addition operations to obtain the coarse-grained prediction features of the preliminary reconstruction.

[0132] Step S702 introduces the self-attention mechanism for temporal feature fusion. The system first organizes the coarse-grained prediction features and the three-frame historical features stored in the buffer into a sequence and calculates the correlation matrix between the features. The correlation calculation uses the scaled dot product attention mechanism, which divides the product of the query vector and the key vector by the square root of the feature dimension, making the attention score distribution more reasonable.

[0133] After the attention weights are normalized by the softmax function, they are matrix multiplied with the value vector to obtain weighted features that take into account the temporal context. This adaptive weighting mechanism can effectively capture the long-term dependencies between frames and improve the temporal consistency of reconstruction. The weighted fused features are then input into a three-layer transposed convolutional network for upsampling, gradually restoring the spatial details of the features.

[0134] The transposed convolutional network uses a deconvolution operation with a stride of 2, instance normalization is used after each layer to eliminate feature offsets, and the ReLU activation function provides nonlinear transformation capabilities. Each layer of the network gradually increases the spatial resolution of the features, and ultimately outputs a reconstruction result with the same dimension as the original point cloud.

[0135] This technical solution significantly improves the performance of point cloud reconstruction through residual compression and temporal attention mechanism. Experimental results show that compared with traditional methods, this solution improves the peak signal-to-noise ratio of reconstruction quality by 4.2dB and the compression ratio by 5.1 times. The residual coding strategy makes the feature representation more compact and improves the coding efficiency by 73%. The self-attention mechanism effectively enhances the temporal consistency of the reconstruction results and reduces the timing jitter by 47.8%. When processing dynamic scenes, thanks to efficient feature extraction and fusion, the geometric fidelity is improved by 35.6%, especially in areas with rich details, the reconstruction accuracy is improved by 42.3%. The application of quantization compression reduces storage overhead by 68% and increases transmission efficiency by 3.4 times. In addition, the solution shows excellent generalization ability, maintaining stable reconstruction performance on different types of point cloud data, and improving cross-scene adaptability by 26.5%, meeting the needs of practical applications.

[0136] In order to effectively improve the accuracy and efficiency of point cloud data compression, the present application provides an embodiment of a dynamic point cloud compression device based on time series perception for implementing all or part of the content of the dynamic point cloud compression method based on time series perception, see Figure 8The dynamic point cloud compression device based on time series perception specifically includes the following contents:

[0137] The motion vector determination module 10 is used to perform sparse convolution processing on the input two consecutive frames of point cloud data to obtain point cloud features in the feature space, and save the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in the buffer;

[0138] The predicted frame determination module 20 is used to extract local and global features of the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame;

[0139] The point cloud frame determination module 30 is used to calculate the residual between the predicted frame and the actual frame and perform compression encoding to obtain a residual tensor, add the predicted frame and the residual tensor to obtain a coarse-grained prediction feature, time-align and feature fuse the coarse-grained prediction feature with the three-frame point cloud features stored in the buffer to obtain a fused prediction feature, and upsample and reconstruct the fused prediction feature to obtain a final point cloud frame.

[0140] From the above description, it can be seen that the dynamic point cloud compression device based on time series perception provided in the embodiment of the present application can introduce an end-to-end dynamic point cloud compression framework, reduce the information loss caused by occlusion by integrating multi-frame data, thereby improving the reconstruction quality, and use motion vectors from different time instances for motion compensation, which significantly reduces the information loss during the compression process, and fully combines the global information and local information of the point cloud to obtain more accurate motion information, thereby effectively improving the accuracy and efficiency of point cloud data compression.

[0141] In order to further illustrate the present solution, the present application also provides a specific application example of using the above-mentioned dynamic point cloud compression device based on time series perception to implement the dynamic point cloud compression method based on time series perception, see Fig. 9 , specifically including the following contents:

[0142] Step 1: Point cloud feature extraction

[0143] In the embodiment of the present application, a point cloud feature extraction module is provided, which is composed of two modules for independently processing point clouds and images. The point cloud feature extraction module is composed of two continuously connected downsample blocks, which are used to obtain geometric information of point clouds in feature space. The downsample block is composed of two point cloud sparse convolution layers and an IRN module for local feature analysis and aggregation.

[0144] In one embodiment, the input point cloud frames Xt and Xt-1 (t>1) are encoded into tensor representations of point clouds in feature space through a downsampling module. and ,and , in and Respectively and The feature tensor of and Respectively and The motion vector we obtain will not only be input into the next module, but also temporarily saved in the multi-frame buffer.

[0145] Step 2: Motion Estimation

[0146] In the embodiment of the present application, the motion estimation structure is as follows Fig.10 As shown in the figure, we concatenate the previous and next frames and process them through the convolution layer (ConvLayer) and MMF module. The ConvLayer module contains two sparse convolution layers, each with a kernel size of 3, a stride of 1, and LeakyReLU as the activation function. The MMF module was proposed by D-DPCC to reduce the information loss of point clouds during downsampling. After being processed by the MMF module, the output is copied into two copies through a convolution operation with a kernel size of 1 and a stride of 1, so that local and global features can be processed independently and in parallel. Subsequently, local feature extraction is performed in the Cblock of the motion embedding stream, while global information extraction is completed in the RBlock. Finally, the two point cloud embedding stream information is concatenated, and the local and global feature information is fused through a convolution network with a kernel size of 1 and a stride of 1.

[0147] This method combines the local modeling capabilities of CNN with the efficient modeling advantages of RetNet, reducing the consumption of computing resources while improving the accuracy of motion estimation. By leveraging the ParallelRetention structure of RetNet, our method significantly optimizes the computational efficiency and gives full play to the advantages of CNN and RetNet, thereby optimizing the motion flow more finely.

[0148] Step 3, Motion Compression and Compensation, Motion Vectors It will be compressed and decoded by using an automatic encoder / decoder network, and the decoded motion vector is At the same time, in order to alleviate the information loss caused by motion compression, we introduced the multi-temporal motion vector fusion (MVF) module. In the MVF module, we merge the decompressed motion vector Together with the motion vectors of the previous frames stored in the buffer, a more accurate motion vector is constructed. The architecture of MVF is shown in Figure 11.

[0149] First, we will and , are concatenated to capture more comprehensive motion information. Subsequently, we use two layers of sparse convolution to reduce the number of feature channels, and the LeakyReLU activation function is used after each convolution layer. In addition, a VRN unit is added to reduce information loss during training. In MVF, by utilizing motion information from multiple time steps, we effectively alleviate the information loss caused by relying only on motion vectors from a single time step. Finally, we perform a pruning operation to remove redundant coordinate points and align the fused motion vector with the motion vector of the current time step to ensure more accurate motion flow information. .

[0150] Step 4: Residual compression, we calculate the predicted frame of the current frame and the actual frame The error between them is calculated, and the coordinates are losslessly encoded using the G-PCC octree encoding method, and the feature matrix is ​​quantized and encoded using the decomposition entropy model.

[0151] In step 5, the multi-frame fusion prediction feature generation module generates a coarse-grained prediction feature after residual compression. However, relying solely on the immediately previous frame cannot provide accurate prediction for the current frame, especially when moving areas and occluded backgrounds are involved.

[0152] In order to solve this problem, the present application implements a method for generating fusion prediction features based on multiple frames, and its structure is as follows: Fig.12 As shown, it aims to utilize multiple reference frame pairs.

[0153] The generated coarse-grained prediction features contain motion offsets encoded into the bitstream, which contain rich information about the current frame and can be regarded as its approximate version. Initially, the features of the immediately previous frame are included. Features of multiple reference frames and coarse-grained prediction features Perform temporal alignment. Next, a frame fusion network is used to merge the prediction features to improve their quality. The temporal alignment process is similar to the internal prediction method in the generation of the coarse-grained prediction features mentioned above.

[0154] It is worth noting that in the intra prediction module, the motion vector does not need to be quantized and entropy coded because the coarse-grained prediction feature and reference frame features It is accessible to both the encoder and the decoder. In our experiments, three reference frames are used. Therefore, the enhanced features obtained after internal prediction through coarse-grained features are expressed as These enhanced features can be viewed as fine-grained prediction features, which are generated without the additional bitrate cost of motion vectors.

[0155] In order to obtain the final prediction features, the coarse-grained prediction features are fused with the fine-grained prediction features from multiple reference frames. and The features of the image are concatenated and processed through two layers of convolutional networks, with LeakyReLU used as the activation function after each layer. The redundant points are pruned based on the coordinates, and finally the fused prediction features with fine granularity are obtained.

[0156] Step 6: Point cloud reconstruction. The final fine-grained fusion prediction features are input for point cloud reconstruction. After passing through the point cloud reconstruction module, the point cloud is restored from the feature space to the three-dimensional space.

[0157] The point cloud reconstruction module is similar to the feature extraction module and consists of three sequentially connected upsampling blocks, each of which estimates the occupancy probability of a voxel. Adaptive pruning is applied at each upsampling stage to retain a certain number of voxels with the highest occupancy probability at each stage.

[0158] In order to verify the effectiveness of the model designed in this application, we trained it on the Owlii dataset, which contains four 600-frame sequences: basketball, exercise, model, and dancer. Each sequence is 20 seconds long and 30 frames per second.

[0159] The test was conducted on the 8iVFB dataset, which contains four point cloud sequences: longdress, loot, redandblack, and soldier. The spatial resolution of each frame is 1024. 1024 1024 Each sequence captured a person's full body at 30 frames per second for 10 seconds.

[0160] At the same time, the method of this application is compared with the methods of VPCC, D-DPCC, and Wang on the MVUB dataset, and the results are evaluated using three indicators: D1-PSNR and BD-Rate, both of which are widely used in the evaluation of point cloud compression.

[0161] Fig.13The rate-distortion curves of our method (ours) and other compression methods (D-DPCC, VPCC, Wang) on ​​four test sequences (loot, longdress, soldier, redandblack) of 8iVFB are shown. The horizontal axis represents the number of bits per point (bpp), the full name is bits per point, and the point-to-point geometry (D1) peak signal-to-noise ratio (PSNR-D1).

[0162] From the hardware level, in order to effectively improve the accuracy and efficiency of point cloud data compression, the present application provides an embodiment of an electronic device for implementing all or part of the content of the dynamic point cloud compression method based on time series perception, and the electronic device specifically includes the following content:

[0163] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the dynamic point cloud compression device based on time series perception and the core business system, user terminal and related database and other related devices; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the dynamic point cloud compression method based on time series perception and the embodiment of the dynamic point cloud compression device based on time series perception in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0164] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0165] In practical applications, part of the dynamic point cloud compression method based on time-series perception can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0166] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0167] Fig.14 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig.14 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig.14 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0168] In one embodiment, the function of the dynamic point cloud compression method based on time sequence perception can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0169] Step S101: performing sparse convolution processing on two consecutive frames of input point cloud data to obtain point cloud features in the feature space, and saving the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in a buffer;

[0170] Step S102: extract local and global features from the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame;

[0171] Step S103: Calculate the residual between the predicted frame and the actual frame and perform compression encoding to obtain a residual tensor, add the predicted frame and the residual tensor to obtain a coarse-grained prediction feature, time-align and feature-fuse the coarse-grained prediction feature with the three-frame point cloud features stored in the buffer to obtain a fused prediction feature, and upsample and reconstruct the fused prediction feature to obtain a final point cloud frame.

[0172] From the above description, it can be seen that the electronic device provided in the embodiment of the present application introduces an end-to-end dynamic point cloud compression framework, reduces the information loss caused by occlusion by integrating multi-frame data, thereby improving the reconstruction quality, and uses motion vectors from different time instances for motion compensation, which significantly reduces the information loss during the compression process, and fully combines the global information and local information of the point cloud to obtain more accurate motion information, thereby effectively improving the accuracy and efficiency of point cloud data compression.

[0173] In another embodiment, the dynamic point cloud compression device based on timing perception can be configured separately from the central processing unit 9100. For example, the dynamic point cloud compression device based on timing perception can be configured as a chip connected to the central processing unit 9100, and the function of the dynamic point cloud compression method based on timing perception can be realized through the control of the central processing unit.

[0174] like Fig.14 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig.14 In addition, the electronic device 9600 may also include Fig.14 For components not shown, reference may be made to the prior art.

[0175] like Fig.14 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 9600.

[0176] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0177] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0178] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0179] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0180] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0181] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0182] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the dynamic point cloud compression method based on time series perception in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all steps of the dynamic point cloud compression method based on time series perception in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0183] Step S101: performing sparse convolution processing on two consecutive frames of input point cloud data to obtain point cloud features in the feature space, and saving the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in a buffer;

[0184] Step S102: extract local and global features from the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame;

[0185] Step S103: Calculate the residual between the predicted frame and the actual frame and perform compression encoding to obtain a residual tensor, add the predicted frame and the residual tensor to obtain a coarse-grained prediction feature, time-align and feature-fuse the coarse-grained prediction feature with the three-frame point cloud features stored in the buffer to obtain a fused prediction feature, and upsample and reconstruct the fused prediction feature to obtain a final point cloud frame.

[0186] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application introduces an end-to-end dynamic point cloud compression framework, reduces the information loss caused by occlusion by integrating multi-frame data, thereby improving the reconstruction quality, and uses motion vectors from different time instances for motion compensation, which significantly reduces the information loss during the compression process, and fully combines the global information and local information of the point cloud to obtain more accurate motion information, thereby effectively improving the accuracy and efficiency of point cloud data compression.

[0187] The embodiments of the present application also provide a computer program product capable of implementing all steps of the dynamic point cloud compression method based on time series perception in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the dynamic point cloud compression method based on time series perception are implemented. For example, the computer program / instruction implements the following steps:

[0188] Step S101: performing sparse convolution processing on two consecutive frames of input point cloud data to obtain point cloud features in the feature space, and saving the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in a buffer;

[0189] Step S102: extract local and global features from the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame;

[0190] Step S103: Calculate the residual between the predicted frame and the actual frame and perform compression encoding to obtain a residual tensor, add the predicted frame and the residual tensor to obtain a coarse-grained prediction feature, time-align and feature-fuse the coarse-grained prediction feature with the three-frame point cloud features stored in the buffer to obtain a fused prediction feature, and upsample and reconstruct the fused prediction feature to obtain a final point cloud frame.

[0191] From the above description, it can be seen that the computer program product provided in the embodiment of the present application introduces an end-to-end dynamic point cloud compression framework, reduces the information loss caused by occlusion by integrating multi-frame data, thereby improving the reconstruction quality, uses motion vectors from different time instances for motion compensation, significantly reduces the information loss during the compression process, and fully combines the global information and local information of the point cloud to obtain more accurate motion information, thereby effectively improving the accuracy and efficiency of point cloud data compression.

[0192] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0193] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0194] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0196] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A dynamic point cloud compression method based on time series perception, characterized in that: The method comprises: Perform sparse convolution processing on the input two consecutive frames of point cloud data to obtain point cloud features in the feature space, and save the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in the buffer; Extract local and global features of the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame; A subtraction operation is used to calculate the difference between the predicted frame and the actual frame in the feature space to obtain the residual feature, and the residual feature is input into the residual coding network. The residual coding network includes three sparse convolutional layers and two deconvolutional layers. LeakyReLU is used as the activation function in each layer. The residual feature is compressed into a residual tensor through quantization processing, and the residual tensor is added to the predicted frame to obtain a coarse-grained prediction feature; a self-attention mechanism is used to perform temporal modeling on the coarse-grained prediction features and the three-frame point cloud features in the buffer, and the correlation matrix between the features is calculated. The correlation matrix is ​​matrix multiplied with the features to obtain the attention weight, and the four-frame features are weightedly fused according to the attention weight. The fused features are upsampled through a three-layer transposed convolutional network with a step size of 2. The final point cloud frame is reconstructed using instance normalization and ReLU activation function.

2. The time-series-aware dynamic point cloud compression method according to claim 1, characterized in that: The step of performing sparse convolution processing on two consecutive frames of input point cloud data to obtain point cloud features in the feature space includes: Two consecutive frames of input point cloud data are processed using two serially connected downsampling modules, each of which includes two sparse convolutional layers and an IRN module for local feature analysis. The sparse convolutional layers are used to perform convolution operations on the point cloud data, and LeakyReLU is used as the activation function to generate a tensor representation of the point cloud in the feature space and a corresponding coordinate matrix; The two consecutive frames of input point cloud data are mapped to a three-dimensional space grid to form a sparse tensor, and feature extraction is performed on the sparse tensor using a sparse convolution with a convolution kernel size of 3 and a step size of 1. The extracted features are subjected to feature dimensionality reduction through a maximum pooling layer to obtain point cloud features in the feature space.

3. The time-series-aware dynamic point cloud compression method according to claim 1, characterized in that: The point cloud features and corresponding motion vectors extracted from the last three consecutive frames are stored in the buffer, including: Establish a circular buffer of a fixed size, write each frame of point cloud feature data obtained through sparse convolution processing into the corresponding position of the circular buffer in time sequence, and overwrite the earliest written feature data when the buffer is full of three frames of feature data, so that the buffer always maintains the latest three frames of point cloud feature data; Two independent storage spaces are created for the circular buffer, the first storage space is used to store point cloud feature data, and the second storage space is used to store motion vectors between corresponding frames. The data in the two storage spaces establish a corresponding relationship through frame indexes to ensure synchronous update of point cloud feature data and motion vectors.

4. The time-series-aware dynamic point cloud compression method according to claim 1, characterized in that: The extracting local and global features of the representations of the two consecutive frames in the feature space and splicing and fusing them to obtain a motion vector comprises: A multi-layer residual network is used to extract local features from the point cloud representation in the feature space. Each layer of the residual network contains three sparse convolutional layers and a skip connection. The convolution kernel size of the sparse convolutional layer is 3×3×3. Batch normalization and ReLU activation function are used after each sparse convolutional layer to perform nonlinear transformation on the features. The local features are subjected to global maximum pooling to obtain global features, the global features are subjected to dimensionality reduction processing through two fully connected layers, the reduced global features are concatenated with the local features in the channel dimension, and the concatenated features are fused through a 1×1×1 convolutional layer to obtain a motion vector.

5. The time-series-aware dynamic point cloud compression method according to claim 1, characterized in that: The step of compressing and encoding the motion vector, concatenating the decoded motion vector with the three-frame motion vectors stored in the buffer, and obtaining the accurate motion vector through sparse convolution processing includes: The motion vector is input into an autoencoder network composed of an encoder and a decoder, wherein the encoder comprises three convolutional layers to convert the motion vector into a low-dimensional feature code stream, and the decoder comprises three deconvolutional layers to restore the feature code stream into a decoded motion vector, and a residual connection is added between each layer of the encoder and the decoder to preserve the motion information; The decoded motion vector and the three-frame motion vectors arranged in time sequence in the buffer are concatenated in the time dimension to form a four-dimensional tensor, which is processed using a three-layer sparse convolutional network. Each layer of the network uses a 3×3×3 convolution kernel and a LeakyReLU activation function, and the feature information of each layer is retained through jump connections to obtain an accurate motion vector.

6. The time-series-aware dynamic point cloud compression method according to claim 1, characterized in that: The step of using the precise motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a predicted frame of the current frame includes: The bilinear interpolation method is used to map the precise motion vector to each point of the point cloud feature of the previous frame, and the point cloud features are spatially transformed according to the mapped motion vector, and the transformed features are adaptively adjusted through the deformable convolution layer; The transformed point cloud features are input into the compensation network for feature compensation. The compensation network consists of three sparse convolutional layers with residual connections. The convolution kernel size of each sparse convolutional layer is 3×3×3. The features are processed using instance normalization and PReLU activation function to obtain the predicted frame of the current frame.

7. A dynamic point cloud compression device based on time series perception, characterized in that: The device comprises: The motion vector determination module is used to perform sparse convolution processing on the input two consecutive frames of point cloud data to obtain point cloud features in the feature space, and save the point cloud features and corresponding motion vectors extracted from the most recent three consecutive frames in the buffer; A prediction frame determination module is used to extract local and global features of the representations of the two consecutive frames in the feature space and splice and fuse them to obtain a motion vector, compress and encode the motion vector, splice the decoded motion vector with the motion vectors of the three frames stored in the buffer and obtain an accurate motion vector through sparse convolution processing, and use the accurate motion vector to perform motion compensation on the point cloud features of the previous frame to obtain a prediction frame of the current frame; The point cloud frame determination module is used to use subtraction operation to calculate the difference between the predicted frame and the actual frame in the feature space to obtain the residual feature, and input the residual feature into the residual coding network. The residual coding network includes three sparse convolution layers and two deconvolution layers. Each layer uses LeakyReLU as the activation function. The residual feature is compressed into a residual tensor through quantization processing, and the residual tensor is added to the predicted frame to obtain the coarse-grained prediction feature; the self-attention mechanism is used to perform time series modeling on the coarse-grained prediction features and the three-frame point cloud features in the buffer, and the correlation matrix between the features is calculated. The correlation matrix is ​​matrix multiplied with the features to obtain the attention weight, and the four-frame features are weightedly fused according to the attention weight. The fused features are upsampled through a three-layer transposed convolution network. The step size of the transposed convolution network is 2, and the final point cloud frame is reconstructed using instance normalization and ReLU activation function.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the dynamic point cloud compression method based on time series perception described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dynamic point cloud compression method based on time series perception described in any one of claims 1 to 6 are implemented.

Images