Point cloud compression method, system, device and storage medium based on large language model

Abstract

The present application discloses a point cloud compression method, system, device, and storage medium based on a large language model, which relates to the field of point cloud compression technology. The point cloud compression method includes: dividing the original point cloud data into clusters and establishing a K-ary tree corresponding to each cluster; straightening and truncating each K-ary tree to obtain each point cloud block; based on a preset code book, mapping each node in the point cloud block to a text token, and inputting each text token into a large language model to obtain the probability distribution corresponding to each text token; inputting each text token and the probability distribution corresponding to each text token into an arithmetic encoder to obtain point cloud compressed data corresponding to the original point cloud data. The technical solution of this application aims to solve the technical problem of how to achieve point cloud compression through a large language model without alignment training.

Description

Technical Field

[0001] The present application relates to the field of point cloud compression technology, and in particular to a point cloud compression method, system, device and storage medium based on a large language model. Background Art

[0002] 3D point cloud data is a collection of surface point data collected by various 3D data acquisition instruments. This data records various geometric attributes and 3D information at discrete points on the object's surface, including but not limited to 3D coordinates, normal vectors, color information, concave and convex features, and laser reflection intensity. However, the sheer volume of point cloud data presents significant challenges for efficient storage, processing, and real-time transmission. Therefore, after point cloud data is generated, it is typically compressed.

[0003] When compressing point cloud data using a large language model, the model must perform alignment training on the text and point cloud data to achieve cross-modal understanding of the point cloud data. This alignment training is costly. Therefore, how to achieve point cloud compression using a large language model without performing alignment training remains a technical problem that has yet to be solved by those skilled in the art.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a point cloud compression method, system, device and storage medium based on a large language model, aiming to solve the technical problem of how to achieve point cloud compression through a large language model without alignment training.

[0006] To achieve the above objectives, this application proposes a point cloud compression method based on a large language model, the method comprising:

[0007] Divide the original point cloud data into clusters and establish a K-ary tree corresponding to each cluster;

[0008] Straightening and truncating each of the K-ary trees to obtain each point cloud block, wherein the point cloud block is composed of a preset number of nodes in the K-ary tree, and the nodes in the K-ary tree are composed of multiple points in the original point cloud data;

[0009] Based on a preset codebook, each node in the point cloud block is mapped to a text token, and each text token is input into a large language model to obtain a probability distribution corresponding to each text token;

[0010] Input each of the text tokens and the probability distribution corresponding to each of the text tokens into an arithmetic encoder to obtain point cloud compression data corresponding to the original point cloud data.

[0011] In one embodiment, before the step of establishing a K-ary tree corresponding to each of the clusters, the method further includes:

[0012] Normalizing the point cloud coordinates in each cluster to obtain normalized clusters;

[0013] A K-ary tree corresponding to each of the normalized clusters is established.

[0014] In one embodiment, the step of inputting each of the text tokens into a large language model comprises:

[0015] Shifting the order of the text tokens to obtain an initial sentence, wherein the initial sentence is a semantically coherent sentence composed of the text tokens;

[0016] Adding a start token at the beginning of the initial sentence and adding an end token at the end of the initial sentence to obtain a model input sentence;

[0017] The model input sentence is input into the large language model.

[0018] In one embodiment, before the step of inputting each of the text tokens into the large language model, the method further includes:

[0019] Fine-tuning the parameters of large language models via low-rank adapters.

[0020] In one embodiment, the step of establishing a K-ary tree corresponding to the cluster includes:

[0021] Determining a bounding box of the cluster, wherein the bounding box is a minimum cuboid containing all points in the cluster;

[0022] Evenly divide the bounding box into K sub-regions, and construct a K-ary tree with the region in the bounding box as the parent node and the K sub-regions as the child nodes;

[0023] The steps of dividing the sub-region and adding the new nodes obtained after the division to the K-ary tree are executed in a loop until the K-ary tree reaches a preset depth or all sub-regions cannot be divided again if it is detected that the sub-region contains points in the original point cloud data.

[0024] In one embodiment, the step of straightening and truncating the K-ary tree to obtain a point cloud block includes:

[0025] Traversing each node in the K-ary tree to straighten the K-ary tree;

[0026] The nodes are sorted in a traversal order to obtain a traversal sequence, and a sequence consisting of the preset number of consecutive nodes in the traversal sequence is used as a point cloud block.

[0027] In one embodiment, the step of dividing the original point cloud data into clusters includes:

[0028] The points in the original point cloud data are divided into clusters using the K-means clustering algorithm;

[0029] or,

[0030] The points in the original point cloud data are divided into clusters using the DBSCAN algorithm.

[0031] In addition, to achieve the above objectives, the present application also proposes a point cloud compression system based on a large language model, the point cloud compression system based on a large language model comprising:

[0032] A point cloud data partitioning module is used to divide the original point cloud data into clusters and establish a K-ary tree corresponding to each cluster;

[0033] a point cloud block determination module, configured to straighten and truncate each of the K-ary trees to obtain each point cloud block, wherein a point cloud block is composed of a preset number of nodes in the K-ary tree, and a node in the K-ary tree is composed of multiple points in the original point cloud data;

[0034] A probability distribution prediction module is configured to map each node in the point cloud block to a text token based on a preset codebook, and input each text token into a large language model to obtain a probability distribution corresponding to each text token;

[0035] The point cloud compression module is used to input each of the text tokens and the probability distribution corresponding to each of the text tokens into an arithmetic encoder to obtain point cloud compressed data corresponding to the original point cloud data.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a point cloud compression device based on a large language model, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the point cloud compression method based on a large language model as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the point cloud compression method based on the large language model as described above are implemented.

[0038] One or more technical solutions proposed in this application have at least the following technical effects:

[0039] This application divides the original point cloud data into clusters and establishes a K-ary tree corresponding to each cluster. It can divide the original point cloud data into clusters based on the spatial position of each point in the original point cloud data, and then organize the point cloud in each cluster through the K-ary tree to enrich the context in the point cloud data; each K-ary tree is straightened and truncated to obtain each point cloud block, where the point cloud block is composed of a preset number of nodes in the K-ary tree, and the nodes in the K-ary tree are composed of multiple points in the original point cloud data. The length of the text that needs to be processed by the large language model can be shortened by truncation, thereby reducing the encoding length; based on the preset code book, the point cloud blocks are straightened and truncated to obtain the point cloud blocks. Each node is mapped to a text token, and each text token is input into the large language model to obtain the probability distribution corresponding to each text token. Without alignment training, the nodes can be mapped to text tokens based on the mapping relationship in the codebook, and then the large language model can understand the meaning of the sentences composed of each text token and predict the probability distribution of each text token; then each text token and the probability distribution corresponding to each text token are input into the arithmetic encoder to obtain the point cloud compression data corresponding to the original point cloud data. Based on the arithmetic encoder, the original point cloud data can be encoded to achieve compression of the original point cloud data.

[0040] In this way, the present application can achieve compression of point cloud data without performing alignment training on a large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 A flowchart of the first embodiment of the point cloud compression method based on a large language model of this application is provided;

[0044] Figure 2 This is a schematic diagram of the clustering process of an embodiment of the point cloud compression method based on a large language model of the present application;

[0045] Figure 3 This is a token movement diagram of an embodiment of a point cloud compression method based on a large language model of the present application;

[0046] Figure 4 This is a compression diagram of a specific embodiment of the point cloud compression method based on a large language model of the present application;

[0047] Figure 5 This is a schematic diagram of the module structure of the point cloud compression system based on the large language model according to an embodiment of the present application;

[0048] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the point cloud compression method based on the large language model in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] Since existing technologies require alignment training when compressing point cloud data, resulting in high cost of compressing point cloud data, this application provides a solution that can achieve point cloud data compression without alignment training.

[0053] The main solution of this application is: dividing the original point cloud data into clusters, and establishing a K-ary tree corresponding to each cluster; straightening and truncating each K-ary tree to obtain each point cloud block, wherein the point cloud block is composed of a preset number of nodes in the K-ary tree, and the nodes in the K-ary tree are composed of multiple points in the original point cloud data; based on a preset code book, each node in the point cloud block is mapped to a text token, and each text token is input into a large language model to obtain the probability distribution corresponding to each text token; each text token and the probability distribution corresponding to each text token are input into an arithmetic encoder to obtain point cloud compressed data corresponding to the original point cloud data.

[0054] It should be noted that the execution entity of this embodiment can be a point cloud compression device based on a large language model with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, server, etc. The following uses a tablet computer as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the embodiment of the present application provides a point cloud compression method based on a large language model, referring to Figure 1 , Figure 1This is a flowchart of the first embodiment of the point cloud compression method based on a large language model of this application.

[0056] In this embodiment, the point cloud compression method based on the large language model of this application includes steps S10 to S40:

[0057] Step S10, dividing the original point cloud data into clusters, and establishing a K-ary tree corresponding to each cluster;

[0058] It should be noted that the original point cloud data refers to the point cloud data that needs to be compressed, and the K-ary tree refers to a tree structure in which all nodes in the tree have K child nodes.

[0059] In this embodiment, after obtaining the point cloud data that needs to be compressed, in order to avoid excessive data processing and the data being too scattered and difficult to find contextual features in the point cloud data, the point cloud data that needs to be compressed can be divided into clusters, and then each cluster can be converted into a K-ary tree in parallel.

[0060] In a feasible embodiment, the above step S10 further includes:

[0061] Step S101, dividing the points in the original point cloud data into clusters using the K-means clustering algorithm;

[0062] In this embodiment, after obtaining the point cloud data that needs to be compressed, multiple initial center points can be randomly determined in the point cloud data, and the distances between the remaining non-initial center points and the initial center points can be calculated, and then the non-initial center points can be divided into clusters with the shortest distance to the initial center points. In this way, point cloud data of various shapes can be divided into multiple clusters.

[0063] Step S102: Divide the points in the original point cloud data into clusters using the DBSCAN algorithm.

[0064] It should be noted that the DBSCAN algorithm (Density-Based Spatial Clustering of Applications with Noise) is a density-based spatial clustering algorithm that defines a cluster as the largest set of density-connected points and can find clusters of any shape and size in a noisy spatial database.

[0065] In this embodiment, the DBSCAN algorithm may be used to treat a region with a higher density as a cluster, so that a point cloud data may be divided into clusters according to the density of the point cloud.

[0066] In this embodiment, the present application can organize similar point clouds together through clustering, so that the large language model can learn and understand the 3D shape corresponding to the point cloud data to be compressed without the help of text to understand the point cloud.

[0067] Please refer to Figure 2 , Figure 2 The original point cloud data in is the 3D point cloud data of the aircraft. After the clustering algorithm, the point cloud data of the aircraft can be divided into N clusters. In order to facilitate decoding after compression, after dividing the point cloud data into multiple clusters, the offset coordinates of each cluster need to be retained, for example Figure 2 The center point coordinates of the cluster in the nose part can be used as the offset coordinates.

[0068] In a feasible embodiment, the above step S10 further includes:

[0069] Step S103, determining a bounding box of the cluster, wherein the bounding box is a minimum cuboid containing all points in the cluster;

[0070] Step S104, evenly dividing the bounding box into K sub-regions, and constructing a K-ary tree with the region in the bounding box as a parent node and the K sub-regions as child nodes;

[0071] Step S105 is to loop through the steps of dividing the sub-region and adding the new nodes obtained after the division to the K-ary tree if it is detected that the sub-region contains points in the original point cloud data, until the K-ary tree reaches a preset depth or all sub-regions cannot be divided again.

[0072] For example, when constructing Figure 2 When the octree (K=8) corresponding to the cluster of the nose part is used, the bounding box of the cluster can be determined first. The bounding box is the smallest cuboid containing all the points in the cluster, and the bounding box will become the space represented by the root node of the octree. Then perform recursive segmentation: starting from the root node, divide the bounding box evenly into 8 sub-regions (that is, sub-cubes), and each sub-cube corresponds to the 8 sub-nodes of the root node. Then check whether each sub-region contains points in the original point cloud data (the cluster corresponding to the nose). If a sub-region has no points, it is marked as an empty node and is not further divided in subsequent recursions. For sub-regions containing points, repeat the above process, continue to divide it evenly into 8 sub-cubes, and check whether these smaller sub-regions contain points, until the depth of the octree reaches the preset depth or all the divided sub-regions cannot be divided again. In this way, the clustering of the nose part can be converted into an octree. Using basically the same method, Figure 2 Each cluster is converted into an octree in parallel.

[0073] It should be noted that, in a feasible embodiment, before establishing the K-ary tree corresponding to the cluster, the point cloud coordinates in each cluster can also be normalized to obtain each normalized cluster, and then the K-ary tree corresponding to the normalized cluster can be established. Normalization can be achieved by subtracting the offset or by other methods, and the present invention does not limit the normalization method. It is understood that through normalization, the gap between point clouds can be reduced, thereby improving the ability of the K-ary tree to organize context.

[0074] In this embodiment, the present application organizes the point cloud by a K-ary tree, which can divide the point cloud into multiple small blocks, and even multiple small blocks can form a point cloud block, thereby establishing a correspondence between the point cloud block and the small blocks. In this way, when the subsequent large language model predicts the probability distribution of the text token corresponding to the small block, it can be predicted based on the context between the small block and the point cloud block in the K-ary tree, thereby improving the accuracy of the predicted probability distribution.

[0075] Step S20: straightening and truncating each of the K-ary trees to obtain each point cloud block, wherein the point cloud block is composed of a preset number of nodes in the K-ary tree, and the nodes in the K-ary tree are composed of multiple points in the original point cloud data;

[0076] It should be noted that the preset number refers to the length of the pre-set text processing volume. Since the large language model cannot understand the tree structure, it is also necessary to convert the context relationship in the K-ary tree into text that the large language model can understand.

[0077] In this embodiment, the K-ary tree can be converted from multidimensional to one-dimensional, making it easier for the large language model to understand. The one-dimensional data can also be truncated to a specific length, so that when the large language model makes predictions, it can not only reduce the amount of data to be processed but also prioritize small blocks with richer contextual information.

[0078] In a feasible implementation manner, the above step S20 further includes steps S201 to S202:

[0079] Step S201, traversing each node in the K-ary tree to straighten the K-ary tree;

[0080] In step S202 , the nodes are sorted in a traversal order to obtain a traversal sequence, and a sequence consisting of the preset number of consecutive nodes in the traversal sequence is used as a point cloud block.

[0081] It should be noted that the traversal method can be depth-first traversal or breadth-first traversal, which is not limited in this application. The traversal sequence is a sequence obtained by sorting the traversed nodes in the traversal order.

[0082] In this embodiment, after obtaining the K-ary tree corresponding to each cluster, the K-ary tree can be traversed through a depth-first traversal or a breadth-first traversal to convert all nodes in the K-ary tree into a one-dimensional array. Then, to reduce the amount of data processing, the one-dimensional array can be truncated based on a pre-set text processing length to obtain a point cloud block.

[0083] For example, a point cloud block may be composed of the first M (preset number is M) nodes in a one-dimensional array, a point cloud block may be composed of the last M nodes in a one-dimensional array, or a point cloud block may be composed of M consecutive nodes in a one-dimensional array. The present invention is not limited to this.

[0084] Step S30, mapping each node in the point cloud block to a text token based on a preset codebook, and inputting each text token into a large language model to obtain a probability distribution corresponding to each text token;

[0085] It should be noted that the codebook refers to a file that stores the mapping relationship between the density of the point cloud or the features corresponding to the point cloud and the text. The large language model refers to a deep learning model trained on massive text data. Each node in the point cloud block refers to the node that has not been truncated or removed in the K-ary tree. The text token is the basic unit of model processing. The text token can be a single character, part of a word (such as a root, affix), or a complete word. The probability distribution corresponding to the text token refers to the frequency of occurrence of the text token in the predicted original point cloud data.

[0086] In this embodiment, a codebook can be pre-set based on different application scenarios. Therefore, after obtaining a point cloud block, each node in the point cloud block can be mapped to a text token such as a character or word based on the codebook. Since these text tokens are understandable by the large language model, they can be input into the large language model to obtain the frequency of each text token in the original point cloud data.

[0087] It's also important to note that the large language model predicts the frequencies of the text tokens corresponding to the nodes in the point cloud block, sequentially following the order of the nodes. After obtaining the frequency of a text token, the point cloud block is shifted left to align with the large language model before predicting the frequency of the next text token. The large language model then predicts the frequency of the next text token.

[0088] In a feasible implementation manner, the above step S30 includes steps S301 to S302:

[0089] Step S301, shifting the order of the text tokens to obtain an initial sentence, wherein the initial sentence is a semantically coherent sentence composed of the text tokens;

[0090] Step S302: adding a start token to the beginning of the initial sentence and an end token to the end of the initial sentence to obtain a model input sentence;

[0091] Step S303: input the model input sentence into the large language model.

[0092] It should be noted that a start token refers to a text symbol that represents the beginning of a sentence, and an end token refers to a text symbol that represents the end of a sentence.

[0093] For example, please refer to Figure 3 , Figure 3 This is a token movement diagram of an embodiment of the point cloud compression method based on a large language model of this application. Figure 3 In the figure, the mapping relationship in the codebook is represented by a solid line, the mapping relationship after semantic movement is represented by a dotted line, and the dotted line represents the direction of movement. After obtaining each node in the point cloud block, the text token corresponding to each node can be obtained based on the mapping relationship of the solid line, and then the position of the text token can be moved so that each text token can be combined into a sentence that can be understood by the large language model, that is, the initial sentence is obtained. Then, in order to facilitate the large language model to identify the starting and ending positions of the sentence, a start token can be added to the beginning of the initial sentence, an end token can be added to the end of the initial sentence, and the initial sentence after adding the start token and the end token can be used as the model input sentence. The model input sentence is then input into the large language model to obtain the probability distribution corresponding to each text token through the large language model.

[0094] It's worth noting that a common approach to cross-modality in large language models is to train paired datasets containing text and other modalities through alignment training. The essence of alignment training is to identify a token in the text space so that its features are similar in the token space of the other modality. Notably, the tokens selected for alignment initially exist in the text space. This process involves varying different tokens to find a suitable match, hence the term "variable token mapping." The method proposed in this application, however, employs an invariant token mapping, fixing the corresponding tokens from the outset. Learning then adjusts the spatial positions of the text tokens to produce the initial sentence. In this application, the alignment task is complete once the appropriate spatial positions are determined, thus eliminating the need for traditional alignment training methods. Furthermore, this application eliminates the need for accompanying text data during cross-modal alignment. In particular, since alignment training is unnecessary, this application is particularly useful when processing shapeless point clouds. This is because describing shapeless point clouds using text is inherently challenging, while this application, through invariant token mapping, can produce text that a large language model can understand, eliminating the need for textual description of shapeless point clouds. In addition, since this application does not require training of additional alignment modules, nor does it require supplementary text-point cloud pairing data training, the expansion of this application is quite convenient.

[0095] In a feasible implementation manner, before the above step S30, the method further includes:

[0096] Step S100: Adjust the parameters of the large language model through a low-rank adapter.

[0097] It can be understood that the low-rank adapter is a parameter adjustment method for large language models, which can be used to reduce the number of trainable parameters of downstream tasks while keeping most of the weights of the original model unchanged.

[0098] In this embodiment, the present application adjusts the parameters of the large language model through a low-rank adapter, which can avoid fine-tuning all parameters of the large language model, thereby saving a large amount of GPU memory resources.

[0099] In step S40 , each of the text tokens and the probability distribution corresponding to each of the text tokens are input into an arithmetic encoder to obtain point cloud compression data corresponding to the original point cloud data.

[0100] For example, please refer to Figure 4 , Figure 4 This is a flow chart of a specific embodiment of the point cloud compression method based on a large language model of this application. Figure 4 In the cluster i is Figure 2The clustering of the tail of the aircraft in the middle is then performed in sequence to construct a K-ary tree of clusters. The steps of straightening and truncating the K-ary tree can obtain the side information code stream and point cloud block i (large block i), where the small blocks in the point cloud block refer to the nodes in the point cloud block. Then, the small blocks are mapped to text tokens through the codebook, and after being combined into the initial sentence through the invariant token mapping, the start token and the end token are added, and the model input sentence is input into the large language model to obtain the probability distribution of each text token. Among them, the large language model is frozen and does not require training. The probability distribution can be obtained by coordinating it with the large language model through a low-rank adapter. Then, the probability distribution and the model input sentence are input into the arithmetic encoder to obtain the encoded code stream. That is, the compressed point cloud data is obtained.

[0101] In this embodiment, the present application divides the original point cloud data into clusters and establishes a K-ary tree corresponding to each cluster. The original point cloud data can be divided into clusters based on the spatial position of each point in the original point cloud data, and then the point cloud in each cluster is organized by the K-ary tree, which can enrich the context in the point cloud data; each K-ary tree is straightened and truncated to obtain each point cloud block, wherein the point cloud block is composed of a preset number of nodes in the K-ary tree, and the nodes in the K-ary tree are composed of multiple points in the original point cloud data. The length of the text that needs to be processed by the large language model can be shortened by truncation, thereby reducing the encoding length; based on the preset code book, Each node in the point cloud block is mapped to a text token, and each text token is input into the large language model to obtain the probability distribution corresponding to each text token. Without alignment training, the nodes can be mapped to text tokens based on the mapping relationship in the code book, and then the large language model can understand the meaning of the sentence composed of each text token and predict the probability distribution of each text token. Then, each text token and the probability distribution corresponding to each text token are input into the arithmetic encoder to obtain the point cloud compressed data corresponding to the original point cloud data. The original point cloud data can be encoded based on the arithmetic encoder to achieve compression of the original point cloud data. In this way, the present application achieves the purpose of compressing point cloud data without alignment training of the large language model.

[0102] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the point cloud compression method based on the large language model of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0103] This application also provides a point cloud compression system based on a large language model, please refer to Figure 5 , the point cloud compression system based on the large language model includes:

[0104] The point cloud data partitioning module 10 is used to divide the original point cloud data into clusters and establish a K-ary tree corresponding to each cluster;

[0105] a point cloud block determination module 20 for straightening and truncating each of the K-ary trees to obtain each point cloud block, wherein a point cloud block is composed of a preset number of nodes in the K-ary tree, and a node in the K-ary tree is composed of multiple points in the original point cloud data;

[0106] a probability distribution prediction module 30 for mapping each node in the point cloud block into a text token based on a preset codebook, and inputting each text token into a large language model to obtain a probability distribution corresponding to each text token;

[0107] The point cloud compression module 40 is used to input each of the text tokens and the probability distribution corresponding to each of the text tokens into an arithmetic encoder to obtain point cloud compressed data corresponding to the original point cloud data.

[0108] In one embodiment, the point cloud compression system based on the large language model further includes:

[0109] A normalization module, configured to normalize the point cloud coordinates in each cluster to obtain normalized clusters;

[0110] The point cloud data partitioning module is further configured to establish a K-ary tree corresponding to each of the normalized clusters.

[0111] In one embodiment, the probability distribution prediction module 30 is further configured to:

[0112] Shifting the order of the text tokens to obtain an initial sentence, wherein the initial sentence is a semantically coherent sentence composed of the text tokens;

[0113] Adding a start token at the beginning of the initial sentence and adding an end token at the end of the initial sentence to obtain a model input sentence;

[0114] The model input sentence is input into the large language model.

[0115] In one embodiment, the point cloud compression system based on a large language model further includes:

[0116] A parameter adjustment module is used to adjust the parameters of the large language model through a low-rank adapter.

[0117] In one embodiment, the point cloud data segmentation module 10 is further configured to:

[0118] Determining a bounding box of the cluster, wherein the bounding box is a minimum cuboid containing all points in the cluster;

[0119] Evenly divide the bounding box into K sub-regions, and construct a K-ary tree with the region in the bounding box as the parent node and the K sub-regions as the child nodes;

[0120] The steps of dividing the sub-region and adding the new nodes obtained after the division to the K-ary tree are executed in a loop until the K-ary tree reaches a preset depth or all sub-regions cannot be divided again if it is detected that the sub-region contains points in the original point cloud data.

[0121] In one embodiment, the point cloud block determination module 20 is further configured to:

[0122] Traversing each node in the K-ary tree to straighten the K-ary tree;

[0123] The nodes are sorted in a traversal order to obtain a traversal sequence, and a sequence consisting of the preset number of consecutive nodes in the traversal sequence is used as a point cloud block.

[0124] In one embodiment, the point cloud data segmentation module 10 is further configured to:

[0125] The points in the original point cloud data are divided into clusters using the K-means clustering algorithm;

[0126] The points in the original point cloud data are divided into clusters using the DBSCAN algorithm.

[0127] The large language model-based point cloud compression system provided in this application utilizes the large language model-based point cloud compression method described in the aforementioned embodiments, addressing the technical problem of achieving point cloud compression using a large language model without alignment training. Compared to the prior art, the large language model-based point cloud compression system provided in this application achieves the same beneficial effects as the large language model-based point cloud compression method described in the aforementioned embodiments. Other technical features of the large language model-based point cloud compression system are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0128] The present application provides a point cloud compression device based on a large language model, and the point cloud compression device based on a large language model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the point cloud compression method based on the large language model in the above-mentioned embodiment one.

[0129] Reference below Figure 6, which shows a schematic structural diagram of a point cloud compression device based on a large language model suitable for implementing the embodiments of the present application. The point cloud compression device based on a large language model in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The point cloud compression device based on the large language model shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0130] like Figure 6 As shown, the point cloud compression device based on the large language model may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the point cloud compression device based on the large language model are also stored in the RAM 1004. The processing device 1001, ROM 1002 and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the point cloud compression device based on a large language model to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a point cloud compression device based on a large language model with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0131] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0132] The large language model-based point cloud compression device provided in this application utilizes the large language model-based point cloud compression method described in the aforementioned embodiment, solving the technical problem of achieving point cloud compression using a large language model without alignment training. Compared to the prior art, the large language model-based point cloud compression device provided in this application achieves the same beneficial effects as the large language model-based point cloud compression method described in the aforementioned embodiment. Other technical features of this large language model-based point cloud compression device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0133] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0134] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0135] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the point cloud compression method based on the large language model in the above-mentioned embodiment.

[0136] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0137] The computer-readable storage medium may be included in the point cloud compression device based on a large language model; or it may exist independently without being assembled into the point cloud compression device based on a large language model.

[0138] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a point cloud compression device based on a large language model, the point cloud compression device based on the large language model: divides the original point cloud data into clusters, and establishes a K-ary tree corresponding to each cluster; straightens and truncates each K-ary tree to obtain each point cloud block, wherein the point cloud block is composed of a preset number of nodes in the K-ary tree, and the nodes in the K-ary tree are composed of multiple points in the original point cloud data; based on a preset code book, each node in the point cloud block is mapped to a text token, and each text token is input into a large language model to obtain a probability distribution corresponding to each text token; each text token and the probability distribution corresponding to each text token are input into an arithmetic encoder to obtain point cloud compressed data corresponding to the original point cloud data.

[0139] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0140] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0141] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0142] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned large language model-based point cloud compression method. This computer-readable storage medium addresses the technical problem of achieving point cloud compression using a large language model without alignment training. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the large language model-based point cloud compression method provided in the aforementioned embodiments, and are not further elaborated here.

[0143] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned point cloud compression method based on a large language model.

[0144] The computer program product provided in this application solves the technical problem of achieving point cloud compression using a large language model without alignment training. Compared to the prior art, the beneficial effects of the computer program product provided in this application are similar to those of the point cloud compression method based on a large language model provided in the aforementioned embodiments, and are not further elaborated here.

[0145] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A point cloud compression method based on a large language model, characterized in that: The method includes: Divide the original point cloud data into clusters and establish a K-ary tree corresponding to each cluster; Straightening and truncating each of the K-ary trees to obtain each point cloud block, wherein the point cloud block is composed of a preset number of nodes in the K-ary tree, and the nodes in the K-ary tree are composed of multiple points in the original point cloud data; Based on a preset codebook, each node in the point cloud block is mapped to a text token; Shifting the order of the text tokens to obtain an initial sentence, wherein the initial sentence is a semantically coherent sentence composed of the text tokens; Adding a start token at the beginning of the initial sentence and adding an end token at the end of the initial sentence to obtain a model input sentence; Inputting the model input sentence into a large language model to obtain a probability distribution corresponding to each of the text tokens; Input each of the text tokens and the probability distribution corresponding to each of the text tokens into an arithmetic encoder to obtain point cloud compression data corresponding to the original point cloud data.

2. The method according to claim 1, wherein Before the step of establishing the K-ary trees corresponding to the clusters, the method further includes: Normalizing the point cloud coordinates in each cluster to obtain normalized clusters; A K-ary tree corresponding to each of the normalized clusters is established.

3. The method according to claim 1, wherein Before the step of inputting each of the text tokens into the large language model, the method further comprises: Fine-tuning the parameters of large language models via low-rank adapters.

4. The method according to claim 1, wherein The step of establishing a K-ary tree corresponding to the cluster comprises: Determining a bounding box of the cluster, wherein the bounding box is a minimum cuboid containing all points in the cluster; Evenly divide the bounding box into K sub-regions, and construct a K-ary tree with the region in the bounding box as the parent node and the K sub-regions as the child nodes; The steps of dividing the sub-region and adding the new nodes obtained after the division to the K-ary tree are executed in a loop until the K-ary tree reaches a preset depth or all sub-regions cannot be divided again if it is detected that the sub-region contains points in the original point cloud data.

5. The method according to claim 1, wherein The step of straightening and truncating the K-ary tree to obtain a point cloud block includes: Traversing each node in the K-ary tree to straighten the K-ary tree; The nodes are sorted in a traversal order to obtain a traversal sequence, and a sequence consisting of the preset number of consecutive nodes in the traversal sequence is used as a point cloud block.

6. The method according to claim 1, wherein The step of dividing the original point cloud data into clusters includes: The points in the original point cloud data are divided into clusters using the K-means clustering algorithm; or, The points in the original point cloud data are divided into clusters using the DBSCAN algorithm.

7. A point cloud compression system based on a large language model, characterized in that: The system comprises: A point cloud data partitioning module is used to divide the original point cloud data into clusters and establish a K-ary tree corresponding to each cluster; a point cloud block determination module, configured to straighten and truncate each of the K-ary trees to obtain each point cloud block, wherein a point cloud block is composed of a preset number of nodes in the K-ary tree, and a node in the K-ary tree is composed of multiple points in the original point cloud data; a probability distribution prediction module, configured to map each node in the point cloud block to a text token based on a preset codebook, shift the order of each text token, and obtain an initial sentence, wherein the initial sentence is a semantically coherent sentence composed of each text token; Adding a start token at the beginning of the initial sentence and adding an end token at the end of the initial sentence to obtain a model input sentence; Inputting the model input sentence into a large language model to obtain a probability distribution corresponding to each of the text tokens; The point cloud compression module is used to input each of the text tokens and the probability distribution corresponding to each of the text tokens into an arithmetic encoder to obtain point cloud compressed data corresponding to the original point cloud data.

8. A point cloud compression device based on a large language model, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the point cloud compression method based on a large language model according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the point cloud compression method based on a large language model according to any one of claims 1 to 6 are implemented.

Images