A data processing method and apparatus

Abstract

The application provides a data processing method and device, and relates to the field of data processing. The method comprises the following steps: a sending end generates tree structure to-be-compressed data according to original data, and determines data possession information of the original data in the tree structure by using a recurrent network layer contained in a data compression model. The data possession information is used to indicate data distribution of the original data in the tree structure. Then, the sending end compresses the to-be-compressed data according to the data possession information to obtain compressed data. A receiving end determines data possession information of the compressed data in the tree structure by using the recurrent network layer contained in the data compression model. The compressed data is decompressed according to the data possession information to obtain decompressed data. The application uses one recurrent network layer to replace a multi-layer MLP network in the prior art for context prediction, reduces the complexity of the data compression model, and reduces the required computing resources when the context prediction obtains the data possession information.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data processing method and apparatus. Background Technology

[0002] After obtaining the spatial coordinates of each sampling point on the surface of an object, the resulting set of points is called a point cloud. For example, in the field of virtual reality / augmented reality (VR / AR), point clouds can be used to represent digital people and virtual objects; similarly, in the field of autonomous driving, point clouds can be used to simulate reference objects to achieve precise vehicle positioning and navigation. Typically, point clouds are very large in size, and to reduce the storage space they occupy, they are compressed before storage.

[0003] Typically, octree-based compression algorithms (Octree Squeeze) octreeify point clouds, perform context prediction, and then compress the point cloud based on the prediction results. However, the Octtree Squeeze algorithm's context prediction process involves using a multi-layer perceptron (MLP) to fuse features at each level to obtain the prediction result. Each feature fusion level uses at least one MLP layer, which consumes significant computational resources for point cloud compression devices and results in lengthy compression times. Therefore, reducing the model complexity and required computational resources for point cloud compression is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a data processing method and apparatus that solves the problems of high model complexity and large computational resource consumption in point cloud compression in the prior art.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, this application provides a data processing method applicable to a sending end or a device that supports computing devices in implementing the method, such as a chip system. The method includes: the sending end generating tree-structured data to be compressed based on the original data, and determining data occupancy information within the tree structure using a recurrent network layer included in a data compression model. This data occupancy information indicates the data distribution of the original data within the tree structure. Furthermore, the sending end compresses the data to be compressed based on this data occupancy information to obtain compressed data. Thus, this application uses a single recurrent network layer instead of the multi-layered MLP network used in the prior art for context prediction, reducing the complexity of the data compression model and the computational resources required for obtaining data occupancy information during context prediction. Additionally, compared to the prior art which stores the features of each node and intermediate features obtained by fusing features, the data processing method provided in this application does not require storing features and intermediate features, reducing the storage space occupied by data compression.

[0007] In one example, before using a data compression model to determine data occupancy information within the tree structure, the data processing method may further include: acquiring raw data via a sensor, where the raw data is three-dimensional data. For example, the sensor may include at least one of lidar, millimeter-wave radar, and sonar.

[0008] In another example, the data processing method may further include, before using a data compression model to determine the data occupancy information in the tree structure, acquiring raw data via a camera, wherein the raw data is two-dimensional data.

[0009] As an optional implementation, this data processing method also includes displaying at least one of a tree structure and data occupancy information. When the data compression process is lengthy, displaying at least one of the tree structure and data occupancy information helps users monitor the data compression process and pinpoint the location of errors (such as data compression stalls or stops).

[0010] As an optional implementation, the data occupancy information in the tree structure is determined using a data compression model. This includes: the sending end inputs the data occupancy information of the (i-1)th layer node in the tree structure into the recurrent network layer to obtain the data occupancy information of the i-th layer node, where i is a positive integer and the (i-1)th layer node is the parent node of the i-th layer node. Compared to existing point cloud compression devices that store each feature and the fused features, the data processing method provided in this application can use the data occupancy information of the parent node of the node to be encoded to obtain the data occupancy information of the node to be encoded when performing context prediction. This eliminates the need for feature fusion between the features of the node to be encoded and the features of its parent node, reducing the use of MLP networks, lowering the complexity of the data compression model, and reducing the computational resources required to obtain data occupancy information for context prediction.

[0011] As an optional implementation, a data compression model is used to determine the data occupancy information in the tree structure. This includes: the sending end inputting at least one of the following into a recurrent network layer: the data occupancy information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the (i-1)th layer node, to obtain the data occupancy information of the (i-1)th layer node. The summary information of the (i-1)th layer node is used to describe all prediction information from the ancestor node to the (i-1)th layer node. Since the recurrent network layer does not need to use the intermediate features obtained by fusing the features of the node to be encoded and the features of its parent node, the sending end does not need to store the features of the node to be encoded, the features of the parent node, and the intermediate features, which reduces the storage space required for context prediction. Furthermore, compared to the existing point cloud compression devices that need to start from the root node for context prediction of each node, in the data processing method provided in this application, the recurrent network layer can use the features of the node to be encoded and the summary information extracted from the parent node of the node to be encoded to obtain the data occupancy information and summary information of the node to be encoded. This eliminates the need for the data processing method provided in this application to start from the root node, which reduces the computational resources required for context prediction.

[0012] As an optional implementation, the data processing method further includes: the sending end inputting at least one of the following into a recurrent network layer: the data occupancy information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node, to obtain the summary information of the i-th layer node. This summary information describes all predicted information from the ancestor nodes of the i-th layer node to the i-th layer node itself. The summary information of the i-th layer node can be obtained iteratively. For example, if the recurrent network layer includes at least one Long Short-Term Memory (LSTM) layer, the hyperbolic tangent function (tanh) and gate structure (σ function) of the LSTM layer can be used to selectively utilize the preceding information of the i-th layer node to obtain the data occupancy information and summary information of the node to be encoded. In the data processing method provided in this application, the recurrent network layer in the data compression model can utilize all the prediction information from the root node to the parent node of the node to be encoded, and this prediction information can be selectively memorized and retained by the recurrent network layer. This allows the recurrent network layer to utilize the information of all ancestor nodes of the node to be encoded when the data compression model predicts the data occupancy information of the node to be encoded, thereby improving the accuracy of context prediction. By using the data occupancy information obtained in this application to compress the data to be compressed, the data compression ratio is improved.

[0013] As an optional implementation, the data compression model also includes a feature extraction layer. The data processing method further includes: the sending end inputting at least one of the following bytes—the position, depth, and child node numbers of the i-th layer node, and the occupancy status bytes of the (i-1)-th layer node—into the feature extraction layer to obtain the features of the i-th layer node. For example, the aforementioned feature extraction layer includes at least one MLP layer.

[0014] As a possible example, the sending end inputs the position, depth, and child node numbers of the i-th layer node, the occupancy bytes of the (i-1)-th layer node, and the occupancy bytes of at least one sibling node of the i-th layer node into the feature extraction layer to obtain the features of the i-th layer node. The sibling nodes of the i-th layer node refer to other nodes that share the same parent node as the i-th layer node. Since the features of the i-th layer node include the occupancy bytes of its parent and sibling nodes, the sending end reduces the prediction probability of some obviously impossible occupancy bytes of the i-th layer node when performing context prediction for that i-th layer node, thereby improving the prediction accuracy of the data occupancy information of the i-th layer node.

[0015] As an optional implementation, the data compression model also includes a dimension adjustment layer. The data processing method further includes: the sender inputs the data occupancy information of the i-th layer node into the dimension adjustment layer to obtain an occupancy prediction table, which indicates the predicted probability of each occupancy byte of the i-th layer node. For example, the dimension adjustment layer includes at least one MLP layer, which can be used to adjust the output dimension of the data occupancy information to obtain a visualized prediction probability result.

[0016] Secondly, this application provides a data processing method applicable to a receiving end or a computing device capable of implementing the method, such as a chip system. The method includes: the receiving end acquiring compressed data and using a recurrent network layer included in a data compression model to determine data occupancy information in a tree structure, the data occupancy information indicating the data distribution of the compressed data in the tree structure. The receiving end also decompresses the compressed data based on the data occupancy information to obtain decompressed data. This application uses a single recurrent network layer instead of the multi-layer MLP network in the prior art for context prediction, reducing the computational resources required for obtaining data occupancy information during context prediction. Furthermore, compared to the prior art which stores the features of each node and intermediate features obtained by fusing features, the data processing method provided in this application does not require storing features and intermediate features, reducing the storage space occupied by data compression.

[0017] In one possible example, the recurrent network layer may include at least one LSTM layer.

[0018] As an optional implementation, the data compression model is used to determine the data occupancy information in the tree structure, including: the receiving end inputs the data occupancy information of the (i-1)th layer node in the tree structure into the recurrent network layer to obtain the data occupancy information of the i-th layer node, where i is a positive integer and the (i-1)th layer node is the parent node of the i-th layer node.

[0019] As an optional implementation, the receiver uses a data compression model to determine the data occupancy information in the tree structure, including: the receiver inputs at least one of the data occupancy information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the (i-1)th layer node into the recurrent network layer to obtain the data occupancy information of the (i-1)th layer node. The summary information of the (i-1)th layer node is used to describe all the prediction information from the ancestor node of the (i-1)th layer node to the (i-1)th layer node.

[0020] As an optional implementation, the data processing method further includes: the receiving end inputting at least one of the data possession information of the (i-1)th layer node in the tree structure, the summary information of the (i-1)th layer node, and the features of the i-th layer node into the recurrent network layer to obtain the summary information of the i-th layer node. The summary information of the i-th layer node is used to describe all the prediction information from the ancestor node of the i-th layer node to the i-th layer node.

[0021] As an optional implementation, the data compression model also includes a feature extraction layer. The data processing method further includes: the receiving end inputting at least one of the following bytes—the position, depth, and child node numbers of the i-th layer node, and the occupancy status bytes of the (i-1)-th layer node—into the feature extraction layer to obtain the features of the i-th layer node. For example, the feature extraction layer may include at least one MLP layer.

[0022] In one possible example, the receiving end inputs at least one of the following into the feature extraction layer: the position, depth, and child node number of the i-th layer node, and the occupancy byte of the (i-1)-th layer node, to obtain the features of the i-th layer node. This includes: the receiving end inputting the position, depth, and child node number of the i-th layer node, the occupancy byte of the (i-1)-th layer node, and the occupancy byte of at least one sibling node of the i-th layer node into the feature extraction layer to obtain the features of the i-th layer node.

[0023] As an optional implementation, the data compression model also includes a dimension adjustment layer. The data processing method further includes: the receiver inputting the data occupancy information of the i-th layer node into the dimension adjustment layer to obtain an occupancy prediction table, which indicates the predicted probability of each occupancy byte of the i-th layer node. For example, the dimension adjustment layer includes at least one MLP layer.

[0024] Thirdly, this application provides a data processing apparatus, the beneficial effects of which can be found in the description of any aspect of the first aspect, and will not be repeated here. The data processing apparatus has the function of implementing the behavior in the method examples of any aspect of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the data processing apparatus is applied to a sending end, and includes: a preprocessing unit for generating tree-structured data to be compressed based on the original data; a context prediction unit for determining data occupancy information in the tree structure using a data compression model, the data occupancy information indicating the data distribution of the original data in the tree structure, the data compression model including a recurrent network layer for determining the data occupancy information; and an encoding unit for compressing the data to be compressed based on the data occupancy information to obtain compressed data.

[0025] In conjunction with the data processing apparatus provided in the third aspect, as an optional implementation, the data processing apparatus further includes: an acquisition unit for acquiring raw data through sensors, wherein the raw data is three-dimensional data, and the sensors include at least one of lidar, millimeter-wave radar, and sonar.

[0026] In conjunction with the data processing device provided in the third aspect, as an optional implementation, the data processing device further includes: a data acquisition unit for acquiring raw data via a camera, wherein the raw data is two-dimensional data.

[0027] In conjunction with the data processing apparatus provided in the third aspect, as an optional implementation, the data processing apparatus further includes: a display unit for displaying a tree structure and / or data occupancy information.

[0028] In conjunction with the data processing device provided in the third aspect, as an optional implementation, the context prediction unit is specifically used to input the data occupancy information of the (i-1)th layer node in the tree structure into the recurrent network layer to obtain the data occupancy information of the i-th layer node, where i is a positive integer and the (i-1)th layer node is the parent node of the i-th layer node. For example, the recurrent network layer includes at least one LSTM layer.

[0029] In conjunction with the data processing device provided in the third aspect, as an optional implementation, the context prediction unit is specifically used to input at least one of the data possession information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the (i-1)th layer node into the recurrent network layer to obtain the data possession information of the (i-1)th layer node. The summary information of the (i-1)th layer node is used to describe all prediction information from the ancestor node of the (i-1)th layer node to the (i-1)th layer node.

[0030] In conjunction with the data processing device provided in the third aspect, as an optional implementation, the context prediction unit is also used to input at least one of the data possession information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node into the recurrent network layer to obtain the summary information of the i-th layer node. The summary information of the i-th layer node is used to describe all prediction information from the ancestor node of the i-th layer node to the i-th layer node.

[0031] In conjunction with the data processing apparatus provided in the third aspect, as an optional implementation, the data compression model further includes a feature extraction layer. This context prediction unit is also used to input at least one of the following bytes into the feature extraction layer: the position, depth, and child node number of the node in layer i, and the occupancy status byte of the node in layer (i-1), to obtain the features of the node in layer i. For example, the feature extraction layer includes at least one MLP layer.

[0032] In one possible example, the context prediction unit is also used to input the position, depth, and child node number of the i-th layer node, the occupancy byte of the (i-1)-th layer node, and the occupancy byte of at least one sibling node of the i-th layer node into the feature extraction layer to obtain the features of the i-th layer node.

[0033] In conjunction with the data processing apparatus provided in the third aspect, as an optional implementation, the data compression model further includes a dimension adjustment layer. This context prediction unit is also used to input the data occupancy information of the i-th layer node into the dimension adjustment layer to obtain an occupancy prediction table, which indicates the predicted probability of each occupancy byte of the i-th layer node. For example, the dimension adjustment layer includes at least one MLP layer.

[0034] Fourthly, this application provides a data processing apparatus, the beneficial effects of which can be found in the descriptions of any two aspects of the second aspect, and will not be repeated here. The data processing apparatus has the function of implementing the behaviors in the method examples of any two aspects of the second aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the data processing apparatus is applied to a receiving end, and includes: an acquisition unit for acquiring compressed data; a context prediction unit for determining data occupancy information in a tree structure using a data compression model, the data occupancy information indicating the data distribution of compressed data in the tree structure, the data compression model including a recurrent network layer for determining the data occupancy information; and a decompression unit for decompressing the compressed data according to the data occupancy information to obtain decompressed data.

[0035] In conjunction with the data processing device provided in the fourth aspect, as an optional implementation, the context prediction unit is specifically used to input the data occupancy information of the (i-1)th layer node in the tree structure into the recurrent network layer to obtain the data occupancy information of the i-th layer node, where i is a positive integer and the (i-1)th layer node is the parent node of the i-th layer node. For example, the recurrent network layer includes at least one LSTM layer.

[0036] In conjunction with the data processing device provided in the fourth aspect, as an optional implementation, the context prediction unit is specifically used to input at least one of the data possession information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the (i-1)th layer node into the recurrent network layer to obtain the data possession information of the (i-1)th layer node. The summary information of the (i-1)th layer node is used to describe all prediction information from the ancestor node of the (i-1)th layer node to the (i-1)th layer node.

[0037] In conjunction with the data processing device provided in the fourth aspect, as an optional implementation, the context prediction unit is also used to input at least one of the data possession information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node into the recurrent network layer to obtain the summary information of the i-th layer node. The summary information of the i-th layer node is used to describe all prediction information from the ancestor node of the i-th layer node to the i-th layer node.

[0038] In conjunction with the data processing apparatus provided in the fourth aspect, as an optional implementation, the data compression model further includes a feature extraction layer. This context prediction unit is also used to input at least one of the following bytes into the feature extraction layer: the position, depth, and child node number of the node in layer i, and the occupancy status byte of the node in layer (i-1), to obtain the features of the node in layer i. For example, the feature extraction layer includes at least one MLP layer.

[0039] In one possible example, the context prediction unit is also used to input the position, depth, and child node number of the i-th layer node, the occupancy byte of the (i-1)-th layer node, and the occupancy byte of at least one sibling node of the i-th layer node into the feature extraction layer to obtain the features of the i-th layer node.

[0040] In conjunction with the data processing apparatus provided in the fourth aspect, as an optional implementation, the data compression model further includes a dimension adjustment layer. This context prediction unit is also used to input the data occupancy information of the i-th layer node into the dimension adjustment layer to obtain an occupancy prediction table, which indicates the predicted probability of each occupancy byte of the i-th layer node. For example, the dimension adjustment layer includes at least one MLP layer.

[0041] Fifthly, this application provides a computing device including at least one processor and a memory for storing a set of computer instructions; when the processor executes the set of computer instructions, it performs the operation steps of the data processing method in the first aspect or any possible implementation of the first aspect, or in any possible implementation of the second aspect and the second aspect.

[0042] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed by a computing device, implement the operational steps of the method of the first aspect and any possible implementation of the first aspect, or the second aspect and any possible implementation of the second aspect.

[0043] In a seventh aspect, this application provides a computer program product that, when run on a computer, causes a computing device to perform the operational steps of the method of the first aspect and any possible implementation thereof, or the second aspect and any possible implementation thereof.

[0044] Eighthly, this application provides a chip including a memory and a processor. The memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to perform the operation steps of the method in the first aspect and any possible implementation of the first aspect, or the method in the second aspect and any possible implementation of the second aspect.

[0045] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0046] Figure 1 A schematic diagram of a communication system provided in this application;

[0047] Figure 2 A schematic diagram of a point cloud compression and decompression system provided in this application;

[0048] Figure 3 A schematic diagram illustrating a point cloud compression and decompression process provided in this application;

[0049] Figure 4 A schematic diagram of a context prediction method for the Oct Squeeze algorithm provided by the prior art;

[0050] Figure 5 A flowchart illustrating a data processing method provided in this application;

[0051] Figure 6 A schematic diagram of a tree structure provided in this application;

[0052] Figure 7 A schematic diagram of a data compression model provided in this application;

[0053] Figure 8 A flowchart illustrating another data processing method provided in this application;

[0054] Figure 9 A schematic diagram of an LSTM network structure is provided in this application;

[0055] Figure 10 A schematic diagram of data processing provided in this application;

[0056] Figure 11 A schematic diagram of another data compression model provided in this application;

[0057] Figure 12 A schematic diagram of a data processing apparatus provided in this application;

[0058] Figure 13A schematic diagram of the structure of a computing device provided in this application. Detailed Implementation

[0059] To ensure clarity and brevity in the description of the following embodiments, a brief introduction to the relevant technologies is given first.

[0060] A point cloud is a dataset of points. The points in a point cloud can be represented by three-dimensional coordinates (X, Y, Z). Points located on three-dimensional coordinates (X, Y, Z) can include attribute information such as color, classification value, and intensity value.

[0061] Point clouds typically contain large amounts of data, and storing them requires significant storage space. To address this issue, point clouds are compressed before storage. Please refer to [link / reference needed]. Figure 1 , Figure 1 This application provides a schematic diagram of a communication system scenario, which includes at least one terminal (such as...). Figure 1 The terminals 111 to 113 shown, the network, and the data center 130. The terminals and the data center 130 can communicate via a network, which can be an interconnected network.

[0062] A terminal can also be called a terminal device, user equipment (UE), mobile station (MS), mobile terminal (MT), etc.

[0063] In some embodiments, the terminal may be a mobile phone (e.g., Figure 1 The terminal 111 shown), tablet computer (such as Figure 1 The terminal 112 shown), and a computer with wireless transceiver capabilities (such as...) Figure 1 The terminal 113 shown is an example of a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, and a wireless terminal in self-driving technology (such as those integrated into...). Figure 1 The examples shown include lidar on vehicles 121 and 122, wireless terminals in transportation safety, wireless terminals in smart cities, and so on.

[0064] In other embodiments, the terminal can also be a smart home terminal, such as a smart screen, installed in a residence. The terminal can also be a terminal installed in a hospital for remote medical surgery; for example, during non-invasive surgery, the terminal can be used to collect internal patient information (such as point clouds of internal organs).

[0065] Data center 130 can be a server cluster including at least one application server 131, or a cloud data center built from application servers 131. For example, multiple application servers can be independent and different physical devices, or the functions of multiple application servers can be integrated on the same physical device (e.g., multiple application servers under the jurisdiction of a cloud service provider), or some application server functions can be integrated on a single physical device.

[0066] The terminal connects to the application server 131 wirelessly or via a wired connection. The terminal can be fixed in location or mobile. Embodiments of this application do not limit the number of terminals and application servers included in the communication system.

[0067] The terminal can perform functions such as collecting point clouds, compressing point clouds, or decompressing point clouds, and the data center 130 can perform functions such as compressing point clouds or decompressing point clouds. For example... Figure 1 As shown, taking the example of a vehicle-mounted terminal capable of collecting and compressing point clouds, and a data center capable of decompressing point clouds, the vehicle-mounted terminal may include a lidar installed on vehicle 122 and processing equipment communicating with the lidar. For example, when vehicle 122 is traveling straight, the lidar collects point clouds of office buildings, residences, and basketball courts along the roadside; when turning right, the lidar collects point clouds of vegetation on both sides of the road. Figure 1 The point cloud (shown as a tree) is processed by a device that transmits it to a data center 130. However, current mainstream 64-line LiDAR systems collect over 100 gigabytes (GB) of point cloud data per hour, posing a significant challenge to the storage and transmission of the point cloud by the processing device. Therefore, efficient compression algorithms are essential to reduce transmission latency and storage capacity requirements.

[0068] In current technical solutions, point cloud compression approaches can be broadly divided into two categories: traditional point cloud compression algorithms and point cloud compression algorithms based on artificial intelligence (AI) technology.

[0069] In one example, traditional point cloud compression algorithms can be divided into two categories: The first category involves the point cloud compression device converting the point cloud into a two-dimensional signal, and then using existing image compression algorithms for further compression, such as the planar projection-based compression algorithm (MPEG V-PCC) provided by the Moving Pictures Experts Group (MPEG). The second category involves the point cloud compression device converting the point cloud into a tree structure and then performing entropy coding, such as the kd-tree-based Draco algorithm and the octree-based geometry-based compression algorithm (MPEG G-PCC).

[0070] In this paper, a point cloud compression device can be any device that compresses point clouds. For example, a point cloud compression device can be such as... Figure 1 Any one of terminals 111 to 113 shown.

[0071] In another example, AI-based point cloud compression algorithms can be divided into two categories: one is a hybrid algorithm based on traditional algorithm frameworks, which uses AI algorithms to replace the context coding module, such as the Oct Squeeze algorithm. The other category is end-to-end AI (full AI) compression algorithms, such as autoencoder algorithms based on "Point net++" features.

[0072] In general, both traditional compression algorithms and AI-based compression algorithms involve point cloud compression devices decomposing and compressing point clouds based on a tree structure to obtain compressed point cloud data. For example... Figure 2 As shown, Figure 2 This application provides a schematic diagram of a point cloud compression and decompression system. The system includes a transmitter 210 and a receiver 220. The transmitter 210 establishes a communication connection with the receiver 220 through a communication channel 230.

[0073] The aforementioned transmitter 210 can perform data compression, such as... Figure 1 As shown, the transmitter 210 can be any one of the terminals 111 to 113. The transmitter 210 can also be a point cloud compression system installed on vehicle 121 or vehicle 122. For example, the point cloud compression system includes a lidar and a processing device that is communicatively connected to the lidar.

[0074] The transmitter 210 may include a data source 211, a preprocessing module 212, an encoder 213, and a communication interface 214.

[0075] Data source 211 may include or can be any type of electronic device for capturing point clouds, and / or any type of point data generation device, such as a computer graphics processor for generating computer animation scenes or any type of device for acquiring and / or providing real-world point clouds or computer-generated point clouds. Data source 211 can be any type of memory or storage device for storing any point data in the aforementioned point cloud.

[0076] To distinguish the processing performed by the preprocessing module 212, the point cloud 241 can also be referred to as the raw data (or raw point cloud data) 241.

[0077] The preprocessing module 212 is used to receive the point cloud 241 and preprocess the point cloud 241 to obtain preprocessed data 242. For example, the preprocessing performed by the preprocessing module 212 may include color format conversion (e.g., from RGB to YCbCr), octree structuring, etc.

[0078] Encoder 213 is used to receive preprocessed data 242 and compress the preprocessed data 242 after performing context prediction to obtain point cloud compressed data 243.

[0079] The communication interface 214 in the transmitter 210 can be used to: receive point cloud compressed data 243, and send point cloud compressed data 243 (or a version of the point cloud compressed data 243 after any other processing) to another device such as the receiver 220 or any other device through the communication channel 230, so as to store or directly reconstruct it.

[0080] The aforementioned receiver 220 can perform data decompression, such as... Figure 1 As shown, the receiver 220 can be Figure 1 Any one or more servers in the data center 130 shown, such as application server 131, implement the data decompression function of receiving end 220.

[0081] The receiver 220 may include a display device 221, a post-processing module 222, a decoder 223, and a communication interface 224.

[0082] The communication interface 224 in the receiver 220 is used to receive point cloud compressed data 243 (or other processed versions) from the sender 210 or any other sender such as a storage device, for example, the storage device is a point cloud data storage device, and to provide the point cloud compressed data 243 to the decoder 223.

[0083] Communication interfaces 214 and 224 can be used to send or receive point cloud compressed data 243 through a direct communication link between the sender 210 and the receiver 220, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0084] For example, the communication interface 214 can be used to encapsulate the point cloud compressed data 243 into a suitable format such as a message, and / or process the encoded point cloud compressed data 243 using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0085] Communication interface 224 corresponds to communication interface 214. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain point cloud compressed data 243.

[0086] Both communication interface 224 and communication interface 214 can be configured as follows: Figure 2 The arrow pointing from the sender 210 to the corresponding communication channel 230 of the receiver 220 indicates a one-way or two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded compressed data transmission, etc.

[0087] Decoder 223 is used to receive point cloud compressed data 243 and perform context prediction on the point cloud compressed data 243 to obtain decoded data 244.

[0088] The post-processing module 222 is used to post-process the decoded data 244 to obtain post-processed data 245. The post-processing performed by the post-processing module 222 may include, for example, color format conversion (e.g., from YCbCr to RGB), octree reconstruction, or any other processing to generate data for display device 221, etc.

[0089] Display device 221 is used to receive post-processed data 245 for display to a user or viewer. Display device 221 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0090] As an optional implementation, the sender 210 and receiver 220 can transmit the point cloud compressed data 243 via a data forwarding device. For example, the data forwarding device can be a router or a switch.

[0091] Figure 3 This application provides a schematic diagram of a point cloud compression and decompression process, where the data in the point cloud is stored in the form of coordinate points, such as... Figure 3 The given value is “[(x1,y1,z1),(x2,y2,z2)…]”.

[0092] During point cloud compression, the octreeification module 31 can achieve... Figure 2 The preprocessing module 212 shown has the following functions: for example, the octreeification module 31 converts the point cloud into an octree to obtain data to be compressed, which includes a root node and eight child nodes. Figure 3 As shown, an octree is a tree-like data structure used to describe three-dimensional space. Each node of an octree represents a volume element (voxel) of a cube. Each root node has eight child nodes (obtained by dividing the voxels into front-back, left-right, and top-bottom sections). The sum of the volumes of the voxels represented by the eight child nodes is equal to the volume of the voxel represented by the parent node.

[0093] When using an octree to represent a point cloud, the octreeification module 31 can assign a value of "1" or "0" to a child node based on whether the voxel corresponding to the child node contains data. For example, "1" indicates that the voxel corresponding to the child node contains data, and "0" indicates that the voxel corresponding to the child node does not contain data. The octreeification module 31 integrates the values ​​of the eight child nodes and represents them with 8 bits to obtain the occupancy byte of the root node. This occupancy byte indicates the data distribution of the root node. Figure 3As shown, black nodes indicate that there is data in the voxel corresponding to the child node, and white nodes indicate that there is no data in the voxel corresponding to the child node. Therefore, the occupancy status of the root node is: 1011 1100 bytes.

[0094] During point cloud compression, the context prediction device 32 and the entropy encoder 33 can achieve... Figure 2 The encoder 213 shown functions, for example, after the context prediction device 32 predicts the occupancy status byte of the root node, the entropy encoder 33 compresses the point cloud corresponding to the root node based on the occupancy status byte to achieve point cloud compression and obtain compressed point cloud data. The entropy coding algorithm used by the entropy encoder 33 and the entropy decoder 35 can be any one or a combination of the following: Shannon coding, Huffman coding, and arithmetic coding, etc. For specific implementation methods of entropy coding, please refer to the relevant descriptions in the prior art, which will not be elaborated here.

[0095] During point cloud decompression, the context prediction device 34 and the entropy decoder 35 can achieve... Figure 2 The decoder 223 shown has the following functions: for example, it uses the same context prediction module 34 as the point cloud compression module to predict the occupancy byte of the root node in the compressed data; the entropy decoder 35 decompresses the compressed data corresponding to the node to be decoded based on the occupancy byte of the root node; the octree reconstruction module 36 reconstructs the compressed data corresponding to the node to be decoded after decompression; after each layer of nodes has been decompressed and reconstructed, the decompressed point cloud is obtained.

[0096] During point cloud compression, the entropy encoder uses the occupancy bytes predicted by the context prediction module. The closer the prediction result of the occupancy bytes is to the actual data distribution of the point cloud, the greater the data compression ratio (the ratio of the amount of point cloud to the amount of compressed data) of the entropy encoder, and the better the point cloud compression effect.

[0097] In current technical solutions, taking the context prediction method provided by the Oct Squeeze algorithm as an example, the Oct Squeeze algorithm converts the point cloud into an octree to obtain multi-level nodes. Assume that these multi-level nodes include at least four levels of nodes, such as... Figure 4 As shown, Figure 4 This diagram illustrates a context prediction method for the Oct Squeeze algorithm provided by existing technology. The point cloud after octreeification includes a great-grandfather node, a grandfather node, a parent node, and a node to be encoded. The depths of the aforementioned nodes are i-3, i-2, i-1, and i, respectively, where i is a positive integer greater than or equal to 3.

[0098] The depth of a node represents the number of levels from the root node to the current position. For example, the root node of an octree has a depth of 0, and the depth of its child nodes is 1. Figure 3 As shown, the root node can be called the level 0 node, and the child nodes can be called the level 1 nodes.

[0099] Figure 4 The context prediction method shown includes a feature extraction process. For example, it uses an MLP network (such as a 5-layer MLPlayer) to extract features from each layer node. Figure 4 As shown, the MLP network obtains the features of each node in each layer, which are high-dimensional feature vectors.

[0100] Figure 4 The context prediction method shown also includes a feature fusion process. For example, it utilizes an MLP network (such as 4 MLP layers) to perform waveNet-like step-by-step fusion of features from each layer node. Figure 4 As shown, the MLP network fuses the features of the node to be encoded (e.g., a node in layer i-1) with the features of its parent node (e.g., a node in layer i-2). After multiple levels of feature fusion, the occupancy prediction result of the node to be encoded is obtained. This occupancy prediction result indicates the data distribution of the voxel midpoints corresponding to the node to be encoded. For a detailed explanation of the principles of WaveNet, please refer to existing technologies; this application will not elaborate further.

[0101] Therefore, since feature extraction at each node layer requires a separate MLP network, and the fusion of two adjacent features also requires a separate MLP network, the point cloud compression device consumes a significant amount of computing resources for MLP network computations, resulting in wasted computing resources. Furthermore, as the depth of the nodes to be encoded increases, the point cloud compression device will repeatedly extract already extracted features using the MLP network, increasing redundant computation. To reduce the computing resources required for context prediction, the point cloud compression device stores each feature and the fused features; however, this also consumes a large amount of storage space.

[0102] To reduce the computational resources and storage space occupied by context prediction, this application provides a data processing method, comprising: a sending end generating tree-structured data to be compressed based on the original data, and using a recurrent network layer included in the data compression model to determine data occupancy information in the tree structure, the data occupancy information indicating the data distribution of the original data in the tree structure; and the sending end compressing the data to be compressed based on the data occupancy information to obtain compressed data. This application uses a single recurrent network layer instead of the multi-layered MLP network in the prior art for context prediction, reducing the complexity of the data compression model and the computational resources required for obtaining data occupancy information in context prediction. Furthermore, compared to the prior art which stores the features of each node and intermediate features obtained by fusing features, the data processing method provided in this application does not require storing features and intermediate features, reducing the storage space occupied by data compression.

[0103] Here Figure 2 The following explanation uses the example of the transmitter 210 (shown as an example) capable of data compression and the receiver 220 capable of data decompression. Figure 5 As shown, Figure 5 This is a flowchart illustrating a data processing method provided in this application, which includes the following steps.

[0104] S510: The sending end generates tree-structured data to be compressed based on the original data.

[0105] The transmitter can be a mobile terminal (e.g., a mobile phone, tablet, etc.) deployed on the vehicle, or it can be a point cloud compression system mounted on the vehicle. For example, the point cloud compression system includes a lidar and a processing device that communicates with the lidar.

[0106] In the first possible scenario, the original data could be three-dimensional data, such as a point cloud.

[0107] The three-dimensional data can be acquired by a sensor. For example, the sensor may include at least one of lidar, millimeter-wave radar, and sonar. The sensor can be integrated into the transmitter or deployed separately on a vehicle belonging to the same category as the transmitter (e.g., Figure 1 The sensor is located on vehicle 122 (as shown) and establishes a communication connection with the transmitting end. Alternatively, the sensor can be a sensor array, where each sensor can be deployed on vehicle 122 and the reference object to be collected (e.g., [reference object]). Figure 1 (The tree shown).

[0108] The point cloud described above can include various types of information. For example, if the point cloud is acquired by a laser measuring device, it includes three-dimensional coordinates and laser reflection intensity. Similarly, if the point cloud is acquired by a photogrammetric device, it can include three-dimensional coordinates and color information. Furthermore, if the point cloud is obtained by combining data acquired by both laser measuring and photogrammetric devices, it can include three-dimensional coordinates, laser reflection intensity, and color information.

[0109] In the second possible scenario, the original data can be two-dimensional data, such as image data.

[0110] The two-dimensional data can be acquired by a camera. This camera can be integrated into the transmitting end or deployed on vehicle 122, and establish a communication connection with the transmitting end. For example, the camera can include one or more cameras, such as those located on the windshield, rear windshield, roof, and body of vehicle 122.

[0111] The tree structure described above can be an M-ary tree, where M is a positive integer greater than or equal to 2, such as an octree, a quadtree, or a binary tree.

[0112] S520: The sending end uses a data compression model to determine the data occupancy information in the tree structure.

[0113] The data occupancy information mentioned above is used to indicate the distribution of the original data within the tree structure. For example... Figure 2 The voxels corresponding to the point cloud shown can be identified by data occupancy information, which indicates the distribution of points within the voxel. For example, this data occupancy information indicates that the occupancy of the root node in the data to be compressed has a prediction probability of 99% for bytes "1000 1000".

[0114] In the first example, such as Figure 6 As shown, assuming i = 3, the node at depth i-3 is the root node, as follows: Figure 6 As shown in (a), the root node has 8 child nodes, numbered 1 to 8. The data occupancy information of the root node is as follows: Figure 6 As shown in (b), the data possession information indicates that the voxels corresponding to the two child nodes with serial numbers "1" and "5" in the voxel corresponding to the root node contain data.

[0115] In the second example, for Figure 6 The node with index "1" and depth i-2 shown in (b) can be used for context prediction, and the resulting data occupancy information is as follows: Figure 6 As shown in (c), the data occupancy information indicates that the voxels corresponding to the nodes at depth i-2 and at depth i-1, with serial numbers "1" and "5", contain data.

[0116] In the third example, for Figure 6 The node with index "5" and depth i-2 shown in (b) can be used for context prediction, and the resulting data occupancy information is as follows: Figure 6 As shown in (d), the data occupancy information indicates that the voxel corresponding to the node with index "3" at depth i-2 contains data in the voxel corresponding to the node with depth i-1.

[0117] In the fourth example, for Figure 6 The node with index "5" and depth i-1 shown in (c) can be used for context prediction, and the resulting data occupancy information is as follows: Figure 6 As shown in (e), the data occupancy information indicates that the voxels corresponding to the nodes at depth i, with serial numbers "1" and "5", contain data.

[0118] The data compression model described above includes a recurrent network layer used to determine the data occupancy information. This recurrent network layer may include at least one convolutional neural network layer, such as a recurrent neural network (RNN), an LSTM layer, a gated recurrent unit (GRU), etc. As an optional implementation, the recurrent network layer may include any combination of the above-mentioned convolutional neural network layers.

[0119] The data compression model also includes a feature extraction layer. This layer determines the features of the node to be encoded—a high-dimensional feature vector—based on at least one of the following: the node's position, depth, child node numbers, and the occupancy bytes of its parent node. The number of feature extraction layers and recurrent network layers can be determined based on the node's depth.

[0120] In one example, Figure 7 This is a schematic diagram of a data compression model provided in this application. The data compression model 700 includes a feature extraction layer 710 and a recurrent network layer 720.

[0121] The feature extraction layer 710 includes a first extraction layer 711, a second extraction layer 712, a third extraction layer 713, and a fourth extraction layer 714. For example, the first extraction layer 711 to the fourth extraction layer 714 can all be multi-layer MLPs.

[0122] In one possible scenario, the first extraction layers 711 to the fourth extraction layers 714 described above can be different MLP networks. For example, the number of MLP layers included in the feature extraction layers may differ for nodes with different depths. For instance, the second extraction layer 712 includes 3 MLP layers, and the third extraction layer 713 includes 4 MLP layers.

[0123] In another possible scenario, the first extraction layer 711 through the fourth extraction layer 714 described above can be the same MLP network. For example, extraction layers can be reused for nodes with different depths. For instance, the first extraction layer 711 through the fourth extraction layer 714 could each consist of four identical MLP layers.

[0124] The recurrent network layer 720 includes a first network layer 721, a second network layer 722, a third network layer 723, and a fourth network layer 724. For example, the first network layer 721 to the fourth network layer 724 can all be LSTM. For instance, each of the first network layer 721 to the fourth network layer 724 includes three identical LSTM layers.

[0125] As an optional implementation, the sending end can input the data possession information of the (i-1)th layer node in the tree structure into the recurrent network layer to obtain the data possession information of the i-th layer node.

[0126] Where i is a positive integer, and the (i-1)th layer node is the parent node of the i-th layer node.

[0127] This example uses i≥3. In this paper, the great-grandfather node 731 is... Figure 6 The node shown has a depth of i-3, and its grandparent node is 732. Figure 6 The first black node shown has a depth of i-2, and its parent node is 733. Figure 6 The second black node shown has a depth of i-1, and the node to be encoded is 734. Figure 6 The first black node with depth i is shown.

[0128] In one possible example, the data occupancy information of parent node 733 is as follows: Figure 6 As shown in (e), for example, the data possession information indicates that the probability that the voxels corresponding to the two child nodes with indices "1" and "5" in the voxel corresponding to the parent node 733 contain data is 98%.

[0129] Compared to existing point cloud compression devices that store each feature and the fused features, the data processing method provided in this application can utilize the data occupancy information of the parent node of the node to be encoded to obtain the data occupancy information of the node to be encoded when performing context prediction. This eliminates the need for feature fusion between the features of the node to be encoded and the features of its parent node, reducing the use of MLP networks and lowering the complexity of the data compression model. Furthermore, since the recurrent network layer does not need to use the intermediate features obtained by fusing the features of the node to be encoded and the features of its parent node, the sending end does not need to store the features of the node to be encoded, the features of the parent node, and the intermediate features, which reduces the storage space required for context prediction at the sending end.

[0130] Regarding the process of determining the data occupancy information of the i-th layer node, here we take... Figure 7 The data compression model shown is 700, which will be used as an example for explanation. Figure 8 As shown, Figure 8 The flowchart of another data processing method provided in this application is shown. S520 may include the following steps.

[0131] S5201. The sending end extracts at least one of the following into the input feature extraction layer: the position, depth, and child node number of the i-th layer node, and the occupancy status byte of the (i-1)-th layer node, to obtain the features of the i-th layer node.

[0132] The position, depth, and child node numbers of the i-th layer node, as well as the occupancy information of the (i-1)-th layer node, can also be referred to as the explicit information of the i-th layer node. For example... Figure 7 As shown, the feature extraction layer 710 can obtain the features of a node based on its explicit information. For example, the fourth extraction layer 714 obtains the feature X of the node to be encoded 734 based on the position, depth, and child node numbers of the node to be encoded 734, as well as the occupancy bytes of the parent node 733. t This feature, also known as a latent feature or implicit information, is a high-dimensional feature vector. For example, the feature of each node in a layer is represented by X, such as... Figure 7 As shown, at time t, the feature of node 734 to be encoded is X. t The characteristic of parent node 733 is X. t-1 The characteristic of grandfather node 732 is X. t-2 The characteristic of the great-grandfather node 731 is X. t-3 .

[0133] In one example, the position mentioned above indicates the position of the voxel corresponding to the node to be encoded 734 within the voxel corresponding to the parent node 733; the depth mentioned above indicates that the depth of the node to be encoded 734 is i.

[0134] In another example, the aforementioned child node number indicates the relative position of the voxel corresponding to the child node of the node to be encoded 734 within the voxel of the node to be encoded 734. For example, this relative position refers to which of the eight child nodes of the node to be encoded 734 the child node is. For example, the child node number can be 1 to 8, with each of the eight numbers representing eight different child nodes. Alternatively, the child node number can also be 0 to 7, etc. The child node number can be adjusted according to the compression requirements of the point cloud, and this application does not limit this adjustment.

[0135] As an optional implementation, S5201 specifically includes: the sending end inputs the position, depth, and child node number of the i-th layer node, the occupancy status byte of the (i-1)-th layer node, and the occupancy status byte of at least one sibling node of the i-th layer node into the feature extraction layer to obtain the features of the i-th layer node. The sibling node of the i-th layer node refers to other nodes that share the same parent node (the (i-1)-th layer node) as the i-th layer node. For example, the node to be encoded, 734, is... Figure 6 The first black node (left side) with depth i is shown, and its sibling node can be the second black node (right side) with depth i. For example, the occupancy information of the sibling node can be determined by the sender using the data occupancy information of the sibling node and compressing the data in the voxel corresponding to the sibling node.

[0136] Because the characteristics of the i-th layer node include the occupancy status bytes of its parent and sibling nodes, the sending end reduces the prediction probability of some obviously impossible occupancy status bytes of the i-th layer node when performing context prediction for that i-th layer node, thereby improving the prediction accuracy of the data occupancy information of the i-th layer node. For example, as... Figure 6 As shown in (b), if the features of the i-th layer node do not include the occupancy byte of its sibling node, the probability that the occupancy byte of the i-th layer node predicted by the context is "1111 1111" is 0.05%; if the features of the i-th layer node include the occupancy byte of its sibling node, and the occupancy byte of the sibling node is "1111 1111", the probability that the occupancy byte of the i-th layer node predicted by the context is "1111 1111" will be less than 0.05%, such as 0.01%.

[0137] S5202, The sending end inputs at least one of the data possession information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node into the recurrent network layer to obtain the data possession information of the i-th layer node.

[0138] The summary information of the (i-1)th layer node is used to describe all the prediction information from the ancestor nodes of the (i-1)th layer node to the (i-1)th layer node. For example... Figure 7 As shown, the summary information C of parent node 733t-1 Used to describe all prediction information from the ancestor nodes of parent node 733 to parent node 733.

[0139] In one example, the ancestor node of a node at level i-1 can be any node from the parent node of the node at level i-1 to the root node. For example... Figure 7 As shown, the ancestor node of parent node 733 can be Figure 7 The grandfather node 732 or the great-grandfather node 731 shown can also be any ancestor node of the great-grandfather node 731, such as the root node of the octree where the parent node 733 is located.

[0140] In one possible implementation, such as Figure 7 As shown, the fourth network layer 724 in the recurrent network layer 720 can be based on the data possession information h of the parent node 733. t-1 Summary information C of parent node 733 t-1 and the feature X of node 734 to be encoded t Obtain the data occupancy information h of node 734 to be encoded t .

[0141] S5203. The sending end inputs at least one of the data possession information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node into the recurrent network layer to obtain the summary information of the i-th layer node.

[0142] The summary information of the i-th layer node is used to describe all the prediction information from the ancestor nodes of the i-th layer node to the i-th layer node. For example... Figure 7 As shown, at time t, the summarized information of node 734 (the node at layer i) to be encoded can be represented by C. t express.

[0143] As an optional implementation method, the above summary information C t It can be obtained iteratively. Here, we take an LSTM layer as an example to illustrate this. Figure 9 As shown, Figure 9 The diagram below illustrates a network structure of an LSTM provided in this application. The LSTM includes two hyperbolic tangent functions (tanh) and three gate structures (σ functions).

[0144] Both the Tanh and σ functions are used to selectively utilize the preceding information of the i-th layer node to obtain the data occupancy and summary information of the node to be encoded. For example... Figure 7 As shown, the preceding information may include the data occupancy information h of parent node 733. t-1 Summary of Information C t-1 and the feature X of node 734 to be encoded t LSTM layers can utilize ht-1 C t-1 and X t To obtain the summary information C of node 734 to be encoded t and data possession information h t .

[0145] In an LSTM network, the summary information C of the i-th layer node t It can be used to predict the data occupancy information of the next layer node (the (i+1)th layer node), just like the summary information C of the (i-1)th layer node. t-1 It participated in the prediction of data occupancy information for the i-th layer node, which will not be elaborated here. For the principles of Tanh and σ functions in the LSTM layer, please refer to existing explanations of LSTM; they will not be elaborated here.

[0146] As an optional implementation method, Figure 9 The figure shown and The operation can be replaced by convolution.

[0147] Compared to existing point cloud compression devices that require starting context prediction from the root node for each node, the data processing method provided in this application utilizes the features of the node to be encoded and the summary information extracted from its parent node (C). t-1 ), to obtain the data occupancy information and summary information (C) of the node to be encoded. t This allows the data processing method provided in this application to avoid starting from the root node, which reduces the computational resources required for context prediction.

[0148] Please see Figure 7 The fourth network layer 724 receives the output of the third network layer using h. t-1 and C t-1 And by utilizing the feature X of node 734 to be encoded t Predict the data occupancy information h of node 734 to be encoded t At that time, h t-1 C t-1 and X t After integration, the summary information C of node 734 to be encoded is obtained. t .

[0149] It is worth noting that existing point cloud compression devices need to store the features of each node as well as the fused features. However, this application uses a recurrent network layer to replace the multi-layer MLP network required for feature aggregation, which reduces the complexity of the data compression model. In addition, the recurrent network layer can extract and transmit the information of ancestor nodes, so that the sending end does not need to store a large number of features, which saves the storage space of the sending end.

[0150] S530: The sending end compresses the data to be compressed based on the data occupancy information to obtain compressed data.

[0151] In one example, the entropy encoder at the sending end compresses the data to be compressed by performing multi-level node-by-level compression on the data to be compressed, thus obtaining the compressed data corresponding to the original data. Please continue to see... Figure 7 Assuming i = 3, then the great-grandfather node 731 with depth i-3 is the root node. The point cloud compression process includes the following multi-level compression steps.

[0152] First-level compression: The entropy encoder uses the data occupancy information of the root node to compress the data in the voxel corresponding to the root node. For example, the data occupancy information of the root node can be obtained by using a data compression model for context prediction, or it can be a preset data occupancy information (e.g., the probability distribution of the occupancy bytes of the root node) provided according to different data compression requirements.

[0153] This example uses variable-length encoding in the entropy encoder. If the tree structure is an octree, the data occupancy information of the root node is shown in Table 1. The predicted probability of the root node's occupancy byte being "0001 1000" is 89%. The compression of the data to be compressed includes the following steps: ① Sort the occupancy bytes according to their predicted probabilities from largest to smallest; ② Group the occupancy bytes corresponding to the two lowest probabilities into a group and divide it into two branch fields, marking them as "0" and "1" respectively. For example, the case where the occupancy byte is "0000 0000" is marked as "0", and the case where the occupancy byte is "1111 1111" is marked as "1". The cases where the occupancy bytes are "0000 0000" and "1111" are also... The two branch fields of "1111" are merged into one branch field, with a probability equal to the sum of the two occupancy bytes (0.01% + 0.003% = 0.013%). ③ All occupancy bytes are combined in a similar manner to ② until the probability of the resulting branch field is 1. ④ The path from the branch field with a probability sum of 1 to each occupancy byte is queried, and the occupancy bytes corresponding to "0" and "1" in each branch field from right to left are recorded to obtain the codeword (binary file) corresponding to each occupancy byte, thus completing the data compression of the root node.

[0154] Table 1

[0155]

[0156]

[0157] Second-stage compression: The entropy encoder uses the data occupancy information of the grandfather node 732 to compress the data in the voxel corresponding to the grandfather node 732.

[0158] Third-level compression: The entropy encoder uses the data occupancy information of parent node 733 to compress the data in the voxel corresponding to parent node 733.

[0159] Fourth-level compression: The entropy encoder uses the data occupancy information of the node to be encoded 734 to compress the data in the voxel corresponding to the node to be encoded 734.

[0160] It is worth noting that the entropy coding method used in the second to fourth levels of compression is the same as that used in the first level of compression, and will not be elaborated here. Furthermore, the entropy coding method used in the first to fourth levels of compression is illustrated using variable-length coding (Huffman coding) as an example, but the data compression process in the data processing method provided in this application can also use arithmetic coding. This application does not limit the entropy coding method used for data compression.

[0161] Figure 6 and Figure 7 The example shown uses a point cloud octree to obtain a 4-level node. However, when the amount of point cloud data is smaller, fewer layers of nodes can be used to compress the point cloud; when the amount of point cloud data is larger, more layers of nodes can be used to compress the point cloud. This application does not limit this.

[0162] The sending end utilizes an entropy encoder to perform multi-level compression on the data to be compressed based on data occupancy information, resulting in compressed data. Compared to existing technologies that use multi-layered MLPs for context prediction, this application uses a recurrent network layer to replace the multi-layered MLP network required for feature aggregation, reducing the complexity of the data compression model, reducing the computational resources required for data compression, and improving the efficiency of data compression. Furthermore, compared to existing technologies where the sending end needs to store n-1 sets of features, in the data processing method provided in this application, the sending end only needs to store the data occupancy information of the node to be encoded, reducing the storage space consumption of the sending end.

[0163] S540: The sending end sends compressed data to the receiving end.

[0164] In one example, the sender can transmit the compressed data to the receiver via a communication link. For instance, if the compressed data is a binary file, the sender encapsulates the compressed data into a suitable format such as a message, and / or processes the compressed data using any type of transmission encoding or processing, so that it can be transmitted over a communication link or communication network.

[0165] Compared to existing technologies that store the features of each node and the intermediate features obtained by fusing features, the data processing method provided in this application does not require storing features and intermediate features, which reduces the storage space occupied by data compression, reduces the amount of compressed data transmitted between the sending and receiving ends, and reduces the latency of transmitting point clouds.

[0166] The S550 receiver uses a data compression model to determine the data occupancy information in the tree structure.

[0167] This data occupancy information is used to indicate the distribution of compressed data within the tree structure. For details regarding the data compression model and the specific implementation of the recurrent network layer, please refer to the above explanation of S520; it will not be repeated here.

[0168] During data compression, the sending end can send the predicted data occupancy information and the occupancy status bytes of the parent node into the entropy encoder together. This reduces the number of bits required to record the actual occupancy status bytes, thereby reducing the space required to store the data to be compressed, achieving the effect of data compression. Conversely, during data decompression, the receiving end can use the same context prediction method as in the data compression process to recover the tree structure from the compressed data, achieving the effect of data decompression.

[0169] In the above embodiments of this application, the process of context prediction is described using the sending end that implements data compression as an example. The context prediction method in the data decompression process is the same as that in the data compression process, and will not be described in detail here.

[0170] S560: The receiving end decompresses the compressed data based on the data occupancy information to obtain the decompressed data.

[0171] In one example, the entropy decoder at the receiving end obtains decompressed data by performing multi-level decompression of the compressed data. Please see below. Figure 7 Assuming i = 3, then the great-grandfather node 731 at depth i-3 is the root node. The point cloud decompression process includes the following multi-level decompression steps.

[0172] First-level decompression: The entropy decoder uses the data occupancy information of the root node to decompress the compressed data in the voxel corresponding to the root node. For example, the data occupancy information of the root node can be obtained by using a data compression model for context prediction, or it can be based on different data compression requirements, providing preset data occupancy information for the data compression and decompression of the root node (e.g., the probability distribution of the occupancy bytes of the root node).

[0173] This example uses variable-length encoding in the entropy decoder. If the tree structure is an octree, the data occupancy information of the root node is shown in Table 2. The predicted probability of the root node's occupancy byte being "0001 1000" is 89%. Decompressing the compressed data involves the following steps: ① Sort the occupancy bytes according to their predicted probabilities from largest to smallest; ② Group the occupancy bytes corresponding to the two lowest probabilities into a group and divide it into two branch fields, marking them as "0" and "1" respectively. For example, the case where the occupancy byte is "0000 0000" is marked as "0", and the case where the occupancy byte is "11111111" is marked as "1". The cases where the occupancy bytes are "0000 0000" and "1111" are also separated. The two branch fields of "1111" are merged into one branch field, with a probability equal to the sum of the two occupancy bytes (0.01% + 0.003% = 0.013%). ③ All occupancy bytes are combined in a similar manner to ② until the probability of the resulting branch field is 1. ④ The path from the branch field with a probability sum of 1 to each occupancy byte is queried, and the occupancy bytes corresponding to "0" and "1" in each branch field from right to left are recorded to obtain the codeword (binary file) corresponding to each occupancy byte, thus completing the decompression of the compressed data corresponding to the root node.

[0174] Table 2

[0175] Occupation status bytes Predicted probability 0000 0000 0.01％ ··· ··· 0001 1000 89％ ··· ··· 1111 1111 0.003％

[0176] Second-stage decompression: The entropy decoder uses the data occupancy information of grandfather node 732 to decompress the data in the voxel corresponding to grandfather node 732.

[0177] The third level of decompression: The entropy decoder uses the data occupancy information of parent node 733 to decompress the data in the voxel corresponding to parent node 733.

[0178] Fourth-level decompression: The entropy decoder uses the data occupancy information of the node to be encoded 734 to decompress the data in the voxel corresponding to the node to be encoded 734.

[0179] It is worth noting that the encoding method used in the second to fourth stages of decompression is the same as that used in the first stage of decompression, and will not be elaborated here. Furthermore, the encoding method used in the first to fourth stages of decompression is illustrated using variable-length coding (Huffman coding) as an example. The data decompression process in the data processing method provided in this application can also use arithmetic coding. This application does not limit the encoding method used for data decompression, but the encoding method used for data decompression and data compression should be consistent.

[0180] Figure 6 and Figure 7The example shown is a 4-level node obtained by reconstructing an octree from compressed data. However, when the amount of compressed data is smaller, fewer levels of nodes can be used to decompress the compressed data; when the amount of compressed data is larger, more levels of nodes can be used to decompress the compressed data. This application does not limit this.

[0181] This application uses a recurrent network layer instead of the multi-layer MLP network in the prior art, which reduces the complexity of the data compression model and reduces the computational resources required to obtain data occupancy information through context prediction. In addition, compared with the prior art which stores the features of each node and the intermediate features obtained by fusing features, the data processing method provided in this application does not need to store features and intermediate features, which reduces the storage space occupied by data decompression.

[0182] Typically, point cloud data is very large, and the data compression process is also quite lengthy. Various unexpected situations can occur during data compression (such as data compression stalling or stopping). To facilitate monitoring of the data compression process, this section uses... Figure 5 Taking the example of a transmitter with a display unit, the data processing method also includes the following steps.

[0183] S570, The sending end displays at least one of the following: a tree structure and data occupancy information.

[0184] The aforementioned display unit can be a display screen. For example, if the transmitter is a mobile phone, the display screen can be a touch screen. Alternatively, if the transmitter is the control device of an autonomous driving system, the display screen can be a head-up display (HUD) located inside the vehicle near the driver's side. The display screen can also be the projection area of ​​a projector located inside the vehicle.

[0185] In the first possible design, such as Figure 10 As shown in (a), the sending end can display the tree structure of the data to be compressed, which indicates the data distribution of the (i-1)th layer node and the i-th layer node.

[0186] In the second possible design, the sending end can display data occupancy information. For example... Figure 10 As shown in (b), it indicates the predicted probability of the occupancy byte of the (i-1)th layer node. For example, the predicted probability of the occupancy byte of the (i-1)th layer node being "00000000" is 0.1%, the predicted probability of the occupancy byte of the (i-1)th layer node being "1000 0100" is 99%, and the predicted probability of the occupancy byte of the (i-1)th layer node being "1111 1111" is 0.05%.

[0187] In the third possible design, such as Figure 10As shown in (c), the sending end can simultaneously display the tree structure of the data to be compressed and the data occupancy information of the nodes to be encoded (such as the nodes in the (i-1)th layer).

[0188] When the data compression process is lengthy, displaying at least one of the tree structure and data occupancy information helps users monitor the data compression process, locate errors in data compression (such as data compression stalls or stops), and improve the accuracy of data compression.

[0189] Data occupancy information may be a set of data that is unrecognizable to operations personnel. During data compression or decompression, if there are obvious errors in context prediction, it is difficult to determine the cause of the problem. To address this issue, this section uses a data compression model that also includes a dimension adjustment layer as an example. Please continue reading... Figure 5 The data processing method may also include the following steps.

[0190] S580. The sending end inputs the data occupancy information of the i-th layer node into the dimension adjustment layer to obtain the occupancy prediction table.

[0191] The aforementioned dimension adjustment layer may include at least one MLP layer, which is used to adjust the output dimension of data occupancy information. For example, if the tree structure is an octree, the output dimension of the MLP can be 256, outputting the predicted probability of each occupancy byte corresponding to "0000 0000" to "11111111". Alternatively, if the tree structure is an octree, the output dimension of the MLP can also be 260. In addition to outputting the probability of each occupancy byte, four dimensions can be reserved as options. These options can indicate the device type of the sending end (such as a mobile phone or computer), and can also serve as the storage address for compressed data, indicating the storage location of the compressed data, etc.

[0192] This occupancy prediction table indicates the predicted probability of each occupancy byte of the i-th level node. For example, if the tree structure is a binary tree, the occupancy prediction table can be as shown in Table 3, where the predicted probability of the occupancy byte of the i-th level node being "00" is 5%, the predicted probability of the occupancy byte of the i-th level node being "01" is 80%, the predicted probability of the occupancy byte of the i-th level node being "10" is 12%, and the predicted probability of the occupancy byte of the i-th level node being "11" is 3%.

[0193] Table 3

[0194] Serial Number Occupation status bytes Predicted probability 1 00 5％ 2 01 80％ 3 10 12％ 4 11 3％

[0195] During data compression or decompression, the dimension adjustment layer is used to adjust the dimensions of the data occupancy information to obtain a visualized occupancy prediction table. This is beneficial for operators to monitor the data compression or decompression process. In the event of obvious errors in the context prediction, operators can use this occupancy prediction table as a reference to quickly identify problems that occur during the data compression or decompression process, thereby improving the robustness of the data compression and decompression process.

[0196] As an optional implementation, if the tree structure of the data to be compressed is an octree, this application provides a possible specific implementation method for the above data compression model. Here, we will illustrate this using an example where the feature extraction layer includes 3 MLP layers, the recurrent network layer includes 3 LSTM layers, and the dimension adjustment layer includes 1 MLP layer. Figure 11 As shown, Figure 11 This is a schematic diagram of another data compression model provided in this application. The data compression model 1100 includes a feature extraction layer 1110, a recurrent network layer 1120, and a dimension adjustment layer 1130.

[0197] The feature extraction layer 1110 includes a first MLP 1111 (input dimension m, output dimension 128), a second MLP 1112 (input dimension 128, output dimension 128), and a third MLP 1113 (input dimension 128, output dimension 128). The feature extraction layer 1110 can achieve... Figure 7 The function of the fourth extraction layer 714 shown is, for example, that the feature extraction layer 1110 is used to obtain the feature X of the i-th layer node based on at least one of the position, depth and child node number of the i-th layer node, the occupancy byte of the (i-1)-th layer node, and the occupancy byte of at least one sibling node of the i-th layer node. t Regarding the characteristics and feature X of the i-th layer node. t Please refer to the relevant explanation in S5201; it will not be repeated here.

[0198] The recurrent network layer 1120 includes a first LSTM 1121 (input dimension 128, output dimension 128, stride 1×1), a second LSTM 1122 (input dimension 128, output dimension 128, stride 1×1), and a third LSTM 1123 (input dimension 128, output dimension 128, stride 1×1). The recurrent network layer 1120 can implement... Figure 7 The functions of the fourth network layer 724 shown are as follows: for example, the recurrent network layer 1120 can utilize the data occupancy information h of the (i-1)th layer nodes. t-1 Summary information C of the (i-1)th layer nodes t-1 and the feature X of the i-th layer node t Obtain the data occupancy information h of the i-th layer node.t And the summary information C of the i-th layer nodes. t .

[0199] The dimension adjustment layer 1130 includes a fourth MLP 1131 (input dimension 128, output dimension 256). The dimension adjustment layer 1130 can adjust the output dimension of the data occupancy information; for example, the fourth MLP 1131 can adjust the output dimension of the data occupancy information h of the i-th layer node. t The output dimension is adjusted to 256 to obtain the occupancy prediction table. For example, the occupancy prediction table can include the predicted probability of 256 occupancy bytes from "00000000" to "1111 1111".

[0200] by Figure 11 Taking the data compression model shown as an example, the data processing method provided in this application requires a total of 4 MLP layers and 3 LSTM layers to predict the data occupancy information of the node to be encoded, as well as to store the summary information and data occupancy information output by the 3 LSTM layers. The computational amount and storage space required for data compression and decompression are constant.

[0201] In contrast, Figure 4 In the provided context prediction methods, regardless of the layer to which the node to be encoded belongs, the OctSqueeze algorithm can only utilize information from a maximum of n-1 ancestor nodes (e.g., n=4). This results in low accuracy of the occupancy prediction results obtained from context prediction of the node to be encoded when the depth i (n≤i) is large. For example, predicting the data occupancy information of a node requires calculating 4n+1 MLP layers and storing n-1 sets of features. In other words, as the value of n increases, the computational cost of context prediction and the storage space required for features increase linearly.

[0202] Therefore, the data processing method provided in this application uses a recurrent network layer instead of multiple MLP layers, which reduces the complexity of the data compression model. Moreover, the computational resources occupied by the data compression model are constant, so that as the depth of the node to be encoded or the number of ancestor nodes increases, context prediction of the node to be encoded will not consume more computational resources. In addition, the space required to store summary information and data occupancy information is also fixed for each additional layer of depth of the node to be encoded, which reduces the storage space required for data compression and decompression.

[0203] Furthermore, the Oct Squeeze algorithm can only use information from a fixed set of n-1 ancestor nodes when performing context prediction. In the data processing method provided in this application, the recurrent network layer in the data compression model can utilize all prediction information from the root node to the parent node of the node to be encoded. Moreover, this prediction information can be selectively memorized and retained by the recurrent network layer. This allows the recurrent network layer to utilize information from all ancestor nodes of the node to be encoded when the data compression model predicts the data occupancy information of the node to be encoded, thereby improving the accuracy of context prediction and increasing the data compression ratio.

[0204] It is understood that, in order to achieve the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0205] The above text combines Figures 1 to 11 The data processing method provided according to this embodiment is described in detail below, and will be combined with Figure 12 and Figure 13 This describes the data processing apparatus and computing device provided according to this embodiment.

[0206] Figure 12 This is a schematic diagram of a data processing apparatus provided in this application. These data processing apparatuses can be used to implement the functions of the sending end and receiving end in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the data processing apparatus can be as follows: Figure 1 The terminals 111 to 113 or data center 130 shown can also be modules (such as chips) applied to the application server 131.

[0207] The following is combined Figure 12 The structure and function of the first data processing device 1210 and the second data processing device 1220 are described below. The first data processing device 1210 can realize... Figure 2 The functions of the transmitting end shown can be implemented by the second data processing device 1220. Figure 2 The functions of the receiving end are shown. It should be understood that this embodiment only provides an exemplary division of the structure and functional modules of the first data processing device 1210 and the second data processing device 1220, and this application does not limit its specific division in any way.

[0208] like Figure 12As shown, the first data processing device 1210 establishes a communication connection with the second data processing device 1220 through the communication channel 1230. The communication channel 1230 can transmit the compressed data sent from the sending end to the receiving end, and the communication channel 1230 can also transmit other information sent from the receiving end to the sending end.

[0209] like Figure 12 As shown, the first data processing device 1210 includes an acquisition unit 1211, a display unit 1212, a preprocessing unit 1213, a context prediction unit 1214, an encoding unit 1215, and a transceiver unit 1216. These units can be used to implement the above-mentioned... Figure 5 or Figure 8 The method shown in the embodiment corresponds to each operation step performed by the sending end.

[0210] When the first data processing device 1210 is used to implement Figure 5 In the method embodiment shown, the display unit 1212 is used to execute S570, the preprocessing unit 1213 is used to implement S510, the context prediction unit 1214 is used to implement S520 and S580, the encoding unit 1215 is used to implement S530, and the transceiver unit 1216 is used to implement S540.

[0211] Optionally, when the first data processing device 1210 is used to implement Figure 8 When implementing the functions shown in the method embodiment, the context prediction unit 1214 is also used to implement S5201 to S5203.

[0212] For example, the acquisition unit 1211 can implement the functions of the sensors (such as lidar, millimeter-wave radar, and sonar) and cameras provided in the above embodiments. The display unit 1212 may include a display screen; for example, if the first data processing device 1210 is a mobile phone, the display screen may be a touch screen. Or, if the first data processing device 1210 is a control device for an autonomous driving system, the display screen may be a HUD, etc.

[0213] like Figure 12 As shown, the second data processing device 1220 includes an acquisition unit 1221, a context prediction unit 1222, and a decompression unit 1223. These units can be used to implement the above-mentioned... Figure 5 or Figure 8 The method shown in the embodiment corresponds to each operation step performed by the receiving end.

[0214] When the second data processing device 1220 is used to implement Figure 5 or Figure 8In the method embodiment shown, the acquisition unit 1221 is used to acquire compressed data, the context prediction unit 1222 is used to execute S550, and the decompression unit 1223 is used to execute S560.

[0215] For a more detailed description of the first data processing device 1210 and the second data processing device 1220, please refer directly to the above description. Figure 5 or Figure 8 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0216] Figure 13 This application provides a schematic diagram of a computing device 1300, which includes a processor 1310 and a communication interface 1320. The processor 1310 and the communication interface 1320 are coupled together. It is understood that the communication interface 1320 can be a transceiver or an input / output interface. Optionally, the computing device 1300 may further include a memory 1330 for storing instructions executed by the processor 1310, or storing input data required by the processor 1310 to execute instructions, or storing data generated after the processor 1310 executes instructions.

[0217] As one possible implementation, processor 1310 can generate tree-structured data to be compressed based on the original data, and use the recurrent network layer included in the data compression model to determine data occupancy information in the tree structure. This data occupancy information is used to indicate the data distribution of the original data in the tree structure. Then, processor 1310 compresses the aforementioned data to be compressed based on this data occupancy information to obtain compressed data.

[0218] When computing device 1300 is used to implement Figure 5 or Figure 8 In the method shown, the processor 1310, communication interface 1320, and memory 1330 can also collaboratively implement the various operational steps in the data processing method executed by the sending and receiving ends. The computing device 1300 can also execute... Figure 12 The functions of the first data processing device 1210 and the second data processing device 1220 shown are not described in detail here.

[0219] This application embodiment does not limit the specific connection medium between the communication interface 1320, processor 1310, and memory 1330. This application embodiment... Figure 13 The communication interface 1320, processor 1310, and memory 1330 are connected via a bus 1340. Figure 13 The connections between other components are shown in bold and are for illustrative purposes only, not as limiting information. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 13The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0220] The memory 1330 can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method provided in the embodiments of this application. The processor 1310 executes the software programs and modules stored in the memory 1330 to perform various functional applications and data processing. The communication interface 1320 can be used to communicate with other devices for signaling or data. In this application, the computing device 1300 may have multiple communication interfaces 1320.

[0221] The aforementioned memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0222] The aforementioned processor can be an integrated circuit chip with signal processing capabilities. This processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0223] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, portable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in the computing device.

[0224] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a communication device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0225] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0226] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates an "or" relationship between the preceding and following related objects; in the formulas of this application, the character " / " indicates a "division" relationship between the preceding and following related objects.

[0227] The terms "first," "second," and "third," etc., used in this application specification, claims, and the aforementioned drawings are used to distinguish different objects, not to limit a specific order.

[0228] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0229] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

Claims

1. A data processing method, characterized in that, include: The tree-structured data to be compressed is generated based on the original data; the original data includes at least one of the following: three-dimensional data and two-dimensional data, wherein the three-dimensional data is collected by a sensor and the two-dimensional data is collected by a camera; The data compression model includes a recurrent network layer and a feature extraction layer. The recurrent network layer is used to determine data occupancy information, which is used to indicate the data distribution of the original data in the tree structure. Input at least one of the following into the feature extraction layer: the position, depth, and child node number of the i-th layer node, and the occupancy status byte of the (i-1)-th layer node, to obtain the features of the i-th layer node; The data occupancy information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node are input into the recurrent network layer to obtain the data occupancy information of the i-th layer node, where i is a positive integer, the (i-1)th layer node is the parent node of the i-th layer node, and the summary information of the (i-1)th layer node is used to describe all prediction information from the ancestor node of the (i-1)th layer node to the (i-1)th layer node. The data to be compressed is obtained by compressing the data based on the data occupancy information.

2. The method according to claim 1, characterized in that, The method further includes: The raw data is acquired by sensors, which are three-dimensional data, and the sensors include at least one of lidar, millimeter-wave radar and sonar.

3. The method according to claim 1, characterized in that, The method further includes: The raw data is collected by a camera and is in the form of two-dimensional data.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Display the tree structure and / or the data occupancy information.

5. The method according to claim 1, characterized in that, The method further includes: The recurrent network layer is input with at least one of the following: the data possession information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node. The summary information of the i-th layer node is used to describe all prediction information from the ancestor node of the i-th layer node to the i-th layer node.

6. The method according to claim 1, characterized in that, Input at least one of the following into the feature extraction layer: the position, depth, and child node number of the i-th layer node, and the occupancy status byte of the (i-1)-th layer node, to obtain the features of the i-th layer node, including: The position, depth, and child node number of the i-th layer node, the occupancy status byte of the (i-1)-th layer node, and the occupancy status byte of at least one sibling node of the i-th layer node are input into the feature extraction layer to obtain the features of the i-th layer node.

7. The method according to claim 1, characterized in that, The data compression model further includes a dimension adjustment layer, and the method further includes: The data occupancy information of the i-th layer node is input into the dimension adjustment layer to obtain an occupancy prediction table, which indicates the predicted probability of each occupancy byte of the i-th layer node.

8. The method according to claim 7, characterized in that, The dimension adjustment layer includes at least one multilayer perceptron (MLP).

9. The method according to claim 1, characterized in that, The feature extraction layer includes at least one MLP layer.

10. The method according to any one of claims 1-3, characterized in that, The recurrent network layer includes at least one Long Short-Term Memory (LSTM) network layer.

11. A data processing method, characterized in that, include: Acquire compressed data; wherein the compressed data is obtained by compressing tree-structured data generated from the original data, and the original data includes at least one of the following: three-dimensional data and two-dimensional data, wherein the three-dimensional data is collected by a sensor and the two-dimensional data is collected by a camera; The data compression model includes a recurrent network layer and a feature extraction layer. The recurrent network layer is used to determine data occupancy information, which is used to indicate the data distribution of the original data in the tree structure. Input at least one of the following into the feature extraction layer: the position, depth, and child node number of the i-th layer node, and the occupancy status byte of the (i-1)-th layer node, to obtain the features of the i-th layer node; The data occupancy information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node are input into the recurrent network layer to obtain the data occupancy information of the i-th layer node, where i is a positive integer, the (i-1)th layer node is the parent node of the i-th layer node, and the summary information of the (i-1)th layer node is used to describe all prediction information from the ancestor node of the (i-1)th layer node to the (i-1)th layer node. The compressed data is decompressed based on the data occupancy information to obtain decompressed data.

12. The method according to claim 11, characterized in that, The method further includes: The recurrent network layer is input with at least one of the following: the data possession information of the (i-1)th layer node, the summary information of the (i-1)th layer node, and the features of the i-th layer node. The summary information of the i-th layer node is used to describe all prediction information from the ancestor node of the i-th layer node to the i-th layer node.

13. The method according to claim 11, characterized in that, Input at least one of the following into the feature extraction layer: the position, depth, and child node number of the i-th layer node, and the occupancy status byte of the (i-1)-th layer node, to obtain the features of the i-th layer node, including: The position, depth, and child node number of the i-th layer node, the occupancy status byte of the (i-1)-th layer node, and the occupancy status byte of at least one sibling node of the i-th layer node are input into the feature extraction layer to obtain the features of the i-th layer node.

14. The method according to any one of claims 11-13, characterized in that, The data compression model further includes a dimension adjustment layer, and the method further includes: The data occupancy information of the i-th layer node is input into the dimension adjustment layer to obtain an occupancy prediction table, which indicates the predicted probability of each occupancy byte of the i-th layer node.

15. The method according to claim 14, characterized in that, The dimension adjustment layer includes at least one multilayer perceptron (MLP).

16. The method according to claim 11 or 13, characterized in that, The feature extraction layer includes at least one MLP layer.

17. The method according to any one of claims 11-13, characterized in that, The recurrent network layer includes at least one Long Short-Term Memory (LSTM) network layer.

18. A data processing apparatus, characterized in that, include: A preprocessing unit is used to generate a tree-structured data to be compressed based on the original data; the original data includes at least one of the following: three-dimensional data and two-dimensional data, wherein the three-dimensional data is collected by a sensor and the two-dimensional data is collected by a camera; a context prediction unit is used to determine data occupancy information in the data compression model, wherein the recurrent network layer is used to determine data occupancy information; the data occupancy information is used to indicate the data distribution of the original data in the tree structure; at least one of the position, depth, and child node number of the i-th layer node, and the occupancy status byte of the (i-1)-th layer node is input into the feature extraction layer to obtain the features of the i-th layer node; at least one of the data occupancy information of the (i-1)-th layer node, the summary information of the (i-1)-th layer node, and the features of the i-th layer node is input into the recurrent network layer to obtain the data occupancy information of the i-th layer node, where i is a positive integer, the (i-1)-th layer node is the parent node of the i-th layer node, and the summary information of the (i-1)-th layer node is used to describe all predicted information from the ancestor node of the (i-1)-th layer node to the (i-1)-th layer node; An encoding unit is used to compress the data to be compressed based on the data occupancy information to obtain compressed data.

19. The apparatus according to claim 18, characterized in that, The context prediction unit is further configured to input at least one of the data possession information of the (i-1)th layer node in the tree structure, the summary information of the (i-1)th layer node, and the features of the i-th layer node into the recurrent network layer to obtain the summary information of the i-th layer node. The summary information of the i-th layer node is used to describe all prediction information from the ancestor node of the i-th layer node to the i-th layer node.

20. The apparatus according to claim 18 or 19, characterized in that, The recurrent network layer includes at least one Long Short-Term Memory (LSTM) network layer.

21. A data processing apparatus, characterized in that, include: An acquisition unit is used to acquire compressed data; wherein the compressed data is obtained by compressing tree-structured data generated from the original data, and the original data includes at least one of the following: three-dimensional data and two-dimensional data, wherein the three-dimensional data is acquired by a sensor and the two-dimensional data is acquired by a camera; A context prediction unit is used in a data compression model comprising a recurrent network layer and a feature extraction layer. The recurrent network layer is used to determine data occupancy information, which indicates the data distribution of the original data in the tree structure. At least one of the following is input into the feature extraction layer: the position, depth, and child node number of the i-th layer node, and the occupancy status byte of the (i-1)-th layer node, to obtain the features of the i-th layer node. At least one of the following is input into the recurrent network layer: the data occupancy information of the (i-1)-th layer node, the summary information of the (i-1)-th layer node, and the features of the i-th layer node, to obtain the data occupancy information of the i-th layer node, where i is a positive integer, the (i-1)-th layer node is the parent node of the i-th layer node, and the summary information of the (i-1)-th layer node describes all prediction information from the ancestor node to the (i-1)-th layer node. The decompression unit is used to decompress the compressed data according to the data occupancy information to obtain decompressed data.

22. A computing device, characterized in that, The method includes a memory and a processor, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions, it performs the operation steps of the method according to any one of claims 1 to 10, or the operation steps of the method according to any one of claims 11 to 17.

23. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by a computing device, implement the method as described in any one of claims 1 to 10, or the method as described in any one of claims 11 to 17.

Images