Three-dimensional point cloud coding method and device, three-dimensional point cloud decoding method and device and electronic equipment

Abstract

The invention provides a three-dimensional point cloud coding method and device, a three-dimensional point cloud decoding method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The coding method comprises the following steps: down-sampling an original point cloud according to a down-sampling parameter to obtain a down-sampling point cloud and target information of each point in the down-sampling point cloud, the down-sampling parameter being a non-integer greater than 1; determining neighbor occupation information of each point in the down-sampling point cloud; clustering the down-sampling point clouds according to the down-sampling parameters to obtain a point cloud cluster; training an interpolation network according to the clustering clusters according to the target information and the neighbor occupation information of each point in each point cloud clustering cluster to obtain optimized interpolation network parameters of each clustering cluster; and performing lossless coding on the down-sampling point cloud to obtain down-sampling point cloud coding data, and coding the interpolation network parameters of each cluster to obtain network coding data, so that point cloud coding of fractional down-sampling can be effectively supported, and more flexible and efficient point cloud compression can be realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a three-dimensional point cloud encoding method, a three-dimensional point cloud decoding method, a device, and an electronic device. Background Art

[0002] Point cloud is a common format for representing three-dimensional data, which represents three-dimensional objects in the real world as an unordered set of points. In related technologies, point cloud encoding is usually performed using lossy compression or lossless compression. Although lossless compression can maintain the integrity of the point cloud, the compression rate is limited, so that the point cloud still requires a large storage space and network transmission bandwidth after compression. Although lossy compression can improve the compression rate and reduce storage and transmission costs, it will cause data distortion and loss of details. Therefore, how to achieve high-quality point cloud encoding has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the present invention is to propose a three-dimensional point cloud encoding method, a three-dimensional point cloud decoding method, a device, and an electronic device, aiming to improve the quality of point cloud coding.

[0004] To achieve the above object, the present invention provides a three-dimensional point cloud encoding method in a first aspect, the method comprising: Obtaining an original point cloud and a downsampling parameter; the downsampling parameter is a non-integer greater than 1; Down-sampling the original point cloud according to the down-sampling parameters to obtain a down-sampled point cloud and target information of each point in the down-sampled point cloud; Determine neighbor occupancy information of each point in the downsampled point cloud; Clustering the downsampled point cloud according to the downsampled parameters to obtain point cloud clusters; According to the target information and neighbor occupancy information of each point in each point cloud cluster, an interpolation network is trained according to the cluster to obtain optimized interpolation network parameters of each cluster; The downsampled point cloud is losslessly encoded to obtain downsampled point cloud encoding data, and the interpolation network parameters of each cluster are encoded to obtain network encoding data.

[0005] The downsampling the original point cloud according to the downsampling parameters to obtain the downsampled point cloud and target information of each point in the downsampled point cloud includes: quantizing the coordinate component of each point in the original point cloud according to the downsampling parameter to obtain the downsampled point cloud; Each point in the downsampled point cloud corresponds to one or more points in the original point cloud, and the target information of each point in the downsampled point cloud is the placeholder information of the corresponding point in the original point cloud.

[0006] The determining of neighbor occupancy information of each point in the downsampled point cloud includes: Determine a set of neighbor coordinate information of each point in the downsampled point cloud according to the geometric coordinates of each point in the downsampled point cloud and a preset distance parameter; Performing voxel processing on the downsampled point cloud to obtain a downsampled point cloud represented by voxelization; According to the downsampled point cloud represented by voxelization and the neighbor coordinate information set, the occupancy status of all neighbors in the neighbor coordinate information set of each point in the downsampled point cloud is queried to obtain the neighbor occupancy information of each point in the downsampled point cloud.

[0007] The step of clustering the downsampled point cloud according to the downsampled parameter to obtain a point cloud cluster includes: The downsampling parameters are , where n is a natural number, p and q are mutually prime positive integers, and q <p<2q; Determine a correspondence between coordinate components of the downsampled point cloud and coordinate components of the original point cloud according to p and q, wherein the correspondence includes a one-to-one or one-to-two relationship; According to the correspondence between the three coordinate components of each point of the downsampled point cloud, the points with the same correspondence are clustered into one category to obtain 8 point cloud clusters.

[0008] The step of clustering the downsampled point cloud according to the downsampled parameter to obtain a point cloud cluster includes: The downsampling parameters are , where n is a positive integer, p and q are mutually prime positive integers, and q <p<2q; Determine the correspondence between the coordinate components of the downsampled point cloud and the coordinate components of the original point cloud according to the n, p and q, wherein the correspondence includes a pair of n+1 or a pair of n+2; According to the correspondence between the three coordinate components of each point of the downsampled point cloud, the points with the same correspondence are clustered into one category to obtain 8 point cloud clusters.

[0009] The interpolation network is trained according to the target information and neighbor occupancy information of each point in each point cloud cluster to obtain optimized interpolation network parameters of each cluster, including: According to the neighbor occupancy information of each point in each point cloud cluster, the interpolation network of each cluster is input to obtain the predicted target information of each point; Calculating the loss function of the interpolation network of each cluster according to the predicted target information and the target information of each point in each point cloud cluster; The interpolation network parameters of each cluster are adjusted by minimizing the loss function to obtain optimized interpolation network parameters of each cluster.

[0010] To achieve the above object, the second aspect of the present invention proposes a three-dimensional point cloud decoding method, the method comprising: Get downsampling parameters, downsampling point cloud and interpolation network parameters; Determine neighbor occupancy information of each point in the downsampled point cloud; Clustering the downsampled point cloud according to the downsampled parameters to obtain downsampled point cloud clusters; Inputting the neighbor occupancy information of each point in each of the downsampled point cloud clusters into the interpolation network corresponding to the interpolation network parameters of each cluster to obtain the predicted target information of each point; A reconstructed point cloud is obtained according to the predicted target information of each point of the downsampled point cloud.

[0011] To achieve the above object, the third aspect of the present invention provides a three-dimensional point cloud encoding device, the device comprising: A first acquisition module is used to acquire an original point cloud and a downsampling parameter; the downsampling parameter is a non-integer greater than 1; A downsampling module, used to downsample the original point cloud according to the downsampling parameters to obtain a downsampled point cloud and target information of each point in the downsampled point cloud; A first determination module is used to determine neighbor occupancy information of each point in the downsampled point cloud; A first clustering module, used for clustering the down-sampled point cloud according to the down-sampling parameters to obtain point cloud clusters; A training module, used to train an interpolation network according to the target information and neighbor occupancy information of each point in each point cloud cluster, so as to obtain optimized interpolation network parameters of each cluster; The coding data generation module is used to losslessly encode the downsampled point cloud to obtain downsampled point cloud coding data, and encode the interpolation network parameters of each cluster to obtain network coding data.

[0012] To achieve the above object, the fourth aspect of the present invention provides a three-dimensional point cloud decoding device, the device comprising: The second acquisition module is used to obtain downsampling parameters, downsampling point clouds and interpolation network parameters; A second determination module is used to determine neighbor occupancy information of each point in the downsampled point cloud; A second clustering module, used for clustering the down-sampled point cloud according to the down-sampling parameters to obtain down-sampled point cloud clusters; An inference module, used for inputting the neighbor occupancy information of each point in each of the downsampled point cloud clusters into the interpolation network corresponding to the interpolation network parameters of each cluster to obtain the predicted target information of each point; The reconstruction module is used to obtain a reconstructed point cloud according to the predicted target information of each point of the down-sampled point cloud.

[0013] To achieve the above-mentioned purpose, the fifth aspect of the present invention proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the three-dimensional point cloud encoding method described in the first aspect or the three-dimensional point cloud decoding method described in the second aspect.

[0014] The three-dimensional point cloud encoding method, three-dimensional point cloud decoding method, three-dimensional point cloud encoding device, three-dimensional point cloud decoding device and electronic device of the embodiment of the present application obtain the original point cloud and the downsampling parameters, downsample the original point cloud according to the downsampling parameters, and obtain the downsampled point cloud and the target information of each point in the downsampled point cloud. The downsampling operation can reduce the data volume of the original point cloud, and indicate the one-to-one or one-to-many mapping relationship between the original point cloud and the downsampled point cloud through the target information, so as to assist the point cloud reconstruction process of the downsampled point cloud according to the target information. When the downsampling parameter is an integer, the original point cloud will be sampled at a fixed interval. When processing the original point cloud with a complex geometric structure, the downsampled point cloud will cause the key features and detail information in the original point cloud to be missing. The embodiment of the present application sets the downsampling parameter to a non-integer greater than 1, and realizes more refined and flexible sampling control through fractional downsampling, which can support the encoding of point clouds of any bit rate, and can better retain the key features and detail information in the original point cloud, thereby realizing flexible and efficient point cloud encoding. In order to restore the original point cloud from the point cloud encoding data with high quality, other information besides the downsampled point cloud is also needed to determine the neighbor occupancy information of each point in the downsampled point cloud, so as to use the neighbor occupancy information as supplementary information to assist point cloud reconstruction, thereby reducing visual distortion and improving the quality of point cloud encoding. During the downsampling process, the original point cloud will undergo non-uniform quantization, resulting in multiple quantization conditions in the original point cloud. In order to distinguish the original point clouds with different quantization conditions, the downsampled point clouds are clustered according to the downsampling parameters to obtain point cloud clusters. In order to ensure that the original point cloud corresponding to each non-uniform quantization condition can be restored with high quality, the interpolation network is trained according to the cluster cluster according to the target information and neighbor occupancy information of each point in each point cloud cluster cluster, and the optimized interpolation network parameters of each cluster cluster are obtained, so that the corresponding cluster clusters can be enhanced with different interpolation networks. In order to reduce the storage space consumed by storing the downsampled point cloud and interpolation network parameters, as well as the time and network overhead consumed by transmitting the downsampled point cloud and interpolation network parameters, the downsampled point cloud is losslessly encoded to obtain the downsampled point cloud encoding data, and the interpolation network parameters of each cluster are encoded to obtain the network encoding data. Lossless encoding can further reduce the bit rate and ensure the data integrity of the downsampled point cloud. The network coding data is used to reconstruct the point cloud of the downsampled point cloud encoding data, so that the downsampled point cloud can be restored to the original point cloud with high quality, thereby achieving high-quality point cloud encoding.

[0015] As can be seen from the above, by clustering the downsampled point cloud according to the downsampling parameters and training the interpolation network according to the clustering clusters, it is possible to effectively support the encoding of point clouds with fractional downsampling, and achieve more flexible and efficient point cloud compression. By supporting point cloud encoding when the downsampling parameter is a non-integer greater than 1, that is, supporting point cloud encoding with fractional downsampling, it is possible to support the encoding of point clouds with any bit rate, and achieve more flexible and precise compression control. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the three-dimensional point cloud encoding method provided by the present invention; Figure 2 yes Figure 1 Flow chart of step S120 in FIG. Figure 3 yes Figure 1 Flow chart of step S130 in FIG. Figure 4 yes Figure 1 Flow chart of step S140 in FIG. Figure 5 It is a schematic diagram of the quantization relationship of the horizontal axis components provided by the present invention; Figure 6 is another schematic diagram of the quantization relationship of the abscissa components provided by the present invention; Figure 7 yes Figure 1 Another flow chart of step S140 in FIG. Figure 8 is another schematic diagram of the quantization relationship of the abscissa components provided by the present invention; Fig. 9 yes Figure 1 Flow chart of step S150 in FIG. Fig.10 is a flow chart of the three-dimensional point cloud decoding method provided by the present invention; Fig.11 It is a structural schematic diagram of the three-dimensional point cloud encoding device provided by the present invention; Fig.12 It is a structural schematic diagram of a three-dimensional point cloud decoding device provided by the present invention; Fig.13 It is a schematic diagram of the hardware structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] Point cloud is a common format for representing three-dimensional data, which represents three-dimensional objects in the real world as an unordered set of points. In related technologies, point cloud encoding is usually performed using lossy compression or lossless compression. Although lossless compression can maintain the integrity of the point cloud, the compression rate is limited, so that the point cloud still requires a large storage space and network transmission bandwidth after compression. Although lossy compression can improve the compression rate and reduce storage and transmission costs, it will cause data distortion and loss of details. Lossy compression requires a balance between compression rate and visual quality, that is, rate-distortion optimization (RDO). Therefore, how to achieve high-quality point cloud encoding has become an urgent problem to be solved.

[0021] Based on this, the present invention provides a three-dimensional point cloud encoding method, a three-dimensional point cloud decoding method, a three-dimensional point cloud encoding device, a three-dimensional point cloud decoding device and an electronic device, which aim to effectively support fractional downsampling point cloud encoding and achieve more flexible and efficient point cloud compression.

[0022] The 3D point cloud encoding method, 3D point cloud decoding method, 3D point cloud encoding device, 3D point cloud decoding device and electronic device provided by the present invention are specifically described through the following embodiments. First, the 3D point cloud encoding method in the present invention is described.

[0023] The three-dimensional point cloud coding method provided by the present invention relates to the field of artificial intelligence technology. The three-dimensional point cloud coding method provided by the present invention can be applied to a terminal, can be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the three-dimensional point cloud coding method, etc., but is not limited to the above forms.

[0024] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0025] Figure 1 It is an optional flow chart of the three-dimensional point cloud encoding method provided by the present invention, which is applied to the encoding end and may include but is not limited to steps S110 to S160.

[0026] Step S110, obtaining the original point cloud and downsampling parameters; the downsampling parameters are non-integers greater than 1; Step S120, downsampling the original point cloud according to the downsampling parameters to obtain the downsampled point cloud and target information of each point in the downsampled point cloud; Step S130, determining the neighbor occupancy information of each point in the downsampled point cloud; Step S140, clustering the downsampled point cloud according to the downsampling parameters to obtain point cloud clusters; Step S150, training an interpolation network according to the target information and neighbor occupancy information of each point in each point cloud cluster, and obtaining optimized interpolation network parameters of each cluster; Step S160 , losslessly encode the downsampled point cloud to obtain downsampled point cloud encoding data, and encode the interpolation network parameters of each cluster to obtain network encoding data.

[0027] In step S110 of some embodiments, the surface of the object is scanned by a three-dimensional sensor such as a three-dimensional scanner, a laser radar, a depth camera, etc. to obtain three-dimensional data, and the three-dimensional data is recorded in a storage medium in the form of points, and the point cloud to be encoded is read from the storage medium to obtain the original point cloud. The original point cloud is a discrete data set composed of multiple three-dimensional coordinate points. The original point cloud includes multiple points, each point includes three-dimensional coordinates, and can carry other information related to the attributes of the point, such as color, reflectivity, and intensity. The storage medium can be an optical storage medium such as an optical disc, a semiconductor storage medium such as a dynamic random access memory, a magnetic storage medium such as a disk drive, etc.

[0028] Three-dimensional data usually contains rich geometric information (such as shape, size, position) and related attributes (such as color, reflectivity, etc.). Compared with two-dimensional data, three-dimensional data can better help machines understand the real environment and provide users with a more immersive visual experience. Point cloud is a common format for representing three-dimensional data. It represents three-dimensional objects in the real world as an unordered set of points (including spatial coordinates and attributes such as color and reflectivity). Since there is no need to store and maintain topological information, point clouds are flexible in the representation of non-manifold geometric structures and have the potential for real-time processing. In order to more realistically represent three-dimensional objects, the original point cloud usually contains a large number of points, which brings great challenges to the storage, transmission and processing of the original point cloud. Therefore, it is necessary to compress the original point cloud. The point cloud compression method based on integer point cloud downsampling does not support point cloud encoding of arbitrary bit rates, and the encoding flexibility is poor. In order to achieve more flexible and efficient point cloud compression, the embodiment of the present application targets non-integer downsampling parameters greater than 1 to support fractional downsampling point cloud compression, and provides support for point cloud encoding at any bit rate, thereby achieving finer compression control and more flexible and efficient point cloud compression.

[0029] Set the downsampling parameter. The downsampling parameter is used to downsample the original point cloud. Downsampling is the process of reducing the number of points in the original point cloud. Usually, certain rules are used to select which points to keep and which to remove. The downsampling parameter is represented by s, where s is any non-integer greater than 1. Under certain precision, s can be represented as , where n is a natural number, p and q are mutually prime positive integers, and q <p<2q。

[0030] See also Figure 2 In some embodiments, step S120 may include but is not limited to steps S210 to S220: Step S210, quantizing the coordinate component of each point in the original point cloud according to the downsampling parameter to obtain a downsampled point cloud; Step S220 , each point in the downsampled point cloud corresponds to one or more points in the original point cloud, and the target information of each point in the downsampled point cloud is the placeholder information of the corresponding point in the original point cloud.

[0031] In step S210 of some embodiments, at the encoding end, the coordinate components of each point in the original point cloud are downsampled s times according to the downsampling parameter s to obtain a downsampled point cloud. Specifically, for each point in the original point cloud, the three coordinate components of the point are divided by the downsampling parameter s to obtain three intermediate components. The three coordinate components are the horizontal coordinate component x, the vertical coordinate component y and the vertical coordinate component z. The three intermediate components of each point in the original point cloud are rounded to integers and then deduplicated to obtain a downsampled point cloud.

[0032] The original point cloud contains N points, and the i-th point is represented as , is the horizontal coordinate component of the ith point, is the ordinate component of the i-th point, is the vertical coordinate component of the i-th point, and the original point cloud is expressed as , the downsampled point cloud is represented as , M is the number of points in the downsampled point cloud, M is not greater than N, For downsampling point cloud The horizontal coordinate component of the jth point in , For downsampling point cloud The ordinate component of the jth point in , For downsampling point cloud The vertical coordinate component of the jth point in . The calculation method of the downsampled point cloud is expressed as: , Among them, unique indicates a deduplication operation, which is used to remove duplicate points; round indicates a rounding operation.

[0033] In step S220 of some embodiments, there is a one-to-one or one-to-many mapping relationship between the downsampled point cloud and the original point cloud, and this mapping relationship is the target information. Each point in the downsampled point cloud corresponds to one or more points in the original point cloud, and according to the correspondence between the midpoints in the downsampled point cloud and the midpoints in the original point cloud, the original point cloud is divided into multiple sub-point clouds, each of which corresponds to a point in the downsampled point cloud.

[0034] The jth point in the downsampled point cloud The corresponding sub-point cloud is , , M is the number of points in the downsampled point cloud, the original point cloud V and the sub-point cloud It can be expressed as: , , in, is the sub-point cloud corresponding to the j-th point in the downsampled point cloud; , and are the horizontal coordinate component, vertical coordinate component and vertical coordinate component of the i-th point in the original point cloud respectively; s is the downsampling parameter; , and are the horizontal coordinate component, vertical coordinate component and vertical coordinate component of the jth point in the downsampled point cloud respectively; Represents a union; round represents a rounding operation.

[0035] For example, the downsampling parameter s=4 / 3, the original point cloud V={(1,1,1),(1,1,3),(2,2,1),(3,3,1),(2,2,3),(2,3,3),(3,2,3),(3,3,3)}, and the downsampled point cloud is ={(1,1,1),(1,1,2),(2,2,1),(2,2,2)}, the original point cloud is divided into sub-point clouds according to the down-sampled point cloud. The down-sampled point cloud contains 4 points, and each point corresponds to a sub-point cloud for , , , .

[0036] For another example, the downsampling parameter s=7 / 5, the original point cloud V={(4,4,4),(4,4,5),(5,6,4),(6,6,4),(6,6,5),(5,5,6),(5,6,6),(6,5,6)}, and the downsampled point cloud is ={(3,3,3),(3,3,4),(4,4,3),(4,4,4)}, the original point cloud is divided into sub-point clouds according to the down-sampled point cloud. The down-sampled point cloud contains 4 points, and each point corresponds to a sub-point cloud for , , , .

[0037] For another example, the downsampling parameter s=5 / 2, the original point cloud V={(2,2,2),(2,2,3),(2,2,4),(2,2,0),(2,2,1),(2,0,0),(0,0,0)}, and the downsampled point cloud is ={(1,1,1),(1,1,0),(1,0,0),(0,0,0)}, the downsampled point cloud contains 4 points, and the original point cloud is divided into sub-point clouds according to the downsampled point cloud. The sub-point cloud corresponding to each point for , , , .

[0038] According to the geometric coordinates and downsampling parameters of each point in the downsampled point cloud, the coordinate information set of the potential original point cloud corresponding to each point in the downsampled point cloud is determined. Specifically, the coordinate information set of the point in the potential original point cloud corresponding to the jth point in the downsampled point cloud is expressed as: , in, is the coordinate information set of the points in the potential original point cloud corresponding to the jth point in the downsampled point cloud; s is the downsampling parameter; , and are the horizontal coordinate component, vertical coordinate component and vertical coordinate component of the jth point in the downsampled point cloud respectively; , and are the horizontal coordinate component, vertical coordinate component and vertical coordinate component of the i-th point in the potential original point cloud respectively; Represents the integer domain; round represents rounding operation.

[0039] For example, if the downsampling parameter s=4 / 3, the downsampling point cloud contains 4 points. ={(1,1,1),(1,1,2),(2,2,1),(2,2,2)}, the potential original point cloud coordinate information set corresponding to each point for , , , .

[0040] For another example, the downsampling parameter s=7 / 5, the downsampling point cloud contains 4 points, and the downsampling point cloud ={(3,3,3),(3,3,4),(4,4,3),(4,4,4)}, the potential original point cloud coordinate information set corresponding to each point for , , , .

[0041] For another example, the downsampling parameter s=5 / 2, the downsampling point cloud contains 4 points, and the downsampling point cloud ={(1,1,1),(1,1,0),(1,0,0),(0,0,0)}, the potential original point cloud coordinate information set corresponding to each point for , , , . There are 27 points in it. There are 18 points in it. There are 12 points in it. Contains 8 points.

[0042] For each point in the downsampled point cloud, according to the coordinate information set of the sub-point cloud and the potential original point cloud corresponding to the point, query the position information of all points in the coordinate information set of the potential original point cloud to obtain the target information. Specifically, determine whether each point in the coordinate information set of the corresponding potential original point cloud belongs to the sub-point cloud. If it belongs to the sub-point cloud, the position information of this point is 1, otherwise it is 0, until the judgment operation of each point in the coordinate information set of the corresponding potential original point cloud is completed, and the target information of the point in the downsampled point cloud is obtained.

[0043] For example, determine the kth point in the coordinate information set of the corresponding potential original point cloud Is it a sub-point cloud? , if the judgment result is yes, the occupancy information of the kth point is 1, if the judgment result is no, the occupancy information of the kth point is 0. The target information of the jth point in the downsampled point cloud is expressed as: , in, For target information; It is an indicator function, the function value is 1 when the event is true, otherwise it is 0; is the number of points in the coordinate information set of the potential original point cloud corresponding to the jth point in the downsampled point cloud; M is the number of points in the downsampled point cloud.

[0044] For example, the downsampling parameter s=4 / 3, the original point cloud V={(1,1,1),(1,1,3),(2,2,1),(3,3,1),(2,2,3),(2,3,3),(3,2,3),(3,3,3)}, and the downsampled point cloud is ={(1,1,1),(1,1,2),(2,2,1),(2,2,2)}, the target information corresponding to each point in the downsampled point cloud for , , , When the downsampling parameter s=4 / 3, the correspondence between the original point cloud, downsampled point cloud, potential original point cloud and target information is shown in Table 1.

[0045] For another example, the downsampling parameter s=7 / 5, the original point cloud V={(4,4,4),(4,4,5),(5,6,4),(6,6,4),(6,6,5),(5,5,6),(5,6,6),(6,5,6)}, and the downsampled point cloud is ={(3,3,3),(3,3,4),(4,4,3),(4,4,4)}, the target information corresponding to each point in the downsampled point cloud for , , , When the downsampling parameter s=7 / 5, the correspondence between the original point cloud, downsampled point cloud, potential original point cloud and target information is shown in Table 2.

[0046] For another example, with downsampling parameter s=5 / 2, the original point cloud V={(2,2,2),(2,2,3),(2,2,4),(2,2,0),(2,2,1),(2,0,0),(0,0,0)}, and the downsampled point cloud is ={(1,1,1),(1,1,0),(1,0,0),(0,0,0)}, the downsampled point cloud contains 4 points, and each point corresponds to the target information for , , , When the downsampling parameter s=5 / 2, the correspondence between the original point cloud, downsampled point cloud, potential original point cloud and target information is shown in Table 3.

[0047] Table 1

[0048] Table 2

[0049] Table 3

[0050] In the above steps S210 to S220, the original point cloud is downsampled by the downsampling parameters to reduce the number of points, and the downsampled point cloud and the target information of each point in the downsampled point cloud are obtained. The target information can reflect the one-to-one or one-to-many mapping information between the downsampled point cloud and the original point cloud, so that when reconstructing the point cloud, the downsampled point cloud is not directly scaled up, but the target information is used to perform point cloud geometric super-resolution, thereby achieving the purpose of reducing or even removing distortion.

[0051] See also Figure 3 In some embodiments, step S130 may include but is not limited to steps S310 to S330: Step S310, determining a neighbor coordinate information set of each point in the downsampled point cloud according to the geometric coordinates of each point in the downsampled point cloud and a preset distance parameter; Step S320, voxelizing the downsampled point cloud to obtain a voxelized downsampled point cloud; Step S330 , querying the occupancy status of all neighbors in the neighbor coordinate information set of each point in the down-sampled point cloud according to the voxelized down-sampled point cloud and the neighbor coordinate information set, and obtaining the neighbor occupancy information of each point in the down-sampled point cloud.

[0052] In step S310 of some embodiments, the preset distance parameters include distance parameters of three coordinate components, and the distance calculation is performed on the coordinate components of the geometric coordinates of each point in the downsampled point cloud according to the preset distance parameters to obtain a set of neighbor coordinate information of each point in the downsampled point cloud.

[0053] For example, if the jth point in the downsampled point cloud is represented by , the preset distance parameters are (0,-1,0) and (0,1,0), then The neighbor coordinate information set of .

[0054] In step S320 of some embodiments, voxelization is the process of converting three-dimensional point cloud data into a regular grid representation. In the voxelization process, each point of the downsampled point cloud is mapped to a fixed-size cubic cell (voxel) according to its coordinates in three-dimensional space, and the unit length is selected as the voxel size, that is, the side length of each voxel is 1 unit. In this case, each point in the downsampled point cloud is mapped to a voxel grid coordinate (x, y, z) according to its coordinates, that is, a 3D (three-dimensional) cube with a position of (x, y, z). There is a point with a value of 1 at the coordinate (x, y, z) position, and other coordinate positions that are not downsampled point clouds are all 0.

[0055] In step S330 of some embodiments, for the neighbor coordinate information set of each point in the downsampled point cloud, the downsampled point cloud represented by the voxelization is queried according to each point in the set, and the value is 1 if there is an occupied place, otherwise it is 0, to obtain the neighbor placeholder information of each point in the downsampled point cloud.

[0056] For example, the original point cloud V = {(1,1,1), (1,1,3), (2,2,1), (3,3,1), (2,2,3), (2,3,3), (3,2,3), (3,3,3)}, after downsampling V by 4 / 3, the downsampled point cloud ={(1,1,1),(1,1,2),(2,2,1),(2,2,2)}, for simple illustration, the neighbor coordinate information set of the downsampled point cloud . The neighbor coordinate information set of (1,1,1) is {(1,0,1), (1,2,1)}. The occupancy of the neighbor (1,0,1) is 0, and the occupancy of the neighbor (1,2,1) is 0. Neighbor placeholder information of (1,1,1) is [0,0]. The neighbor coordinate information set of (1,1,2) is {(1,0,2), (1,2,2)}. The occupancy of the neighbor (1,0,2) is 0, and the occupancy of the neighbor (1,2,2) is 0. Neighbor placeholder information of (1,1,2) is [0,0]. The neighbor coordinate information set of (2,2,1) is {(2,1,1), (2,3,1)}. The occupancy of the neighbor (2,1,1) is 0, and the occupancy of the neighbor (2,3,1) is 0. Neighbor occupancy information of (2,2,1) is [0,0]. The neighbor coordinate information set of (2,2,2) is {(2,1,2), (2,3,2)}. The occupancy of the neighbor (2,1,2) is 0, and the occupancy of the neighbor (2,3,2) is 0. Neighbor occupancy information of (2,2,2) is [0,0].

[0057] In the generated voxel grid, the value of each voxel is a binary signal, which is the event within the voxel. If the value of the voxel is 1, the event within the voxel is true, indicating that there is a point in the voxel and the voxel is occupied. In this way, the originally sparse point cloud data is converted into a regular and unified volume model, which is convenient for subsequent processing and analysis.

[0058] Through the above steps S310 to S330, the neighbor occupancy information of each point in the downsampled point cloud can be obtained.

[0059] See also Figure 4 In some embodiments, step S140 may include but is not limited to steps S410 to S430: Step S410, the downsampling parameters are , where n is a natural number, p and q are mutually prime positive integers, and q <p<2q; Step S420, determining a correspondence between coordinate components of the downsampled point cloud and coordinate components of the original point cloud according to p and q, wherein the correspondence includes a one-to-one or one-to-two relationship; Step S430 , based on the correspondence between the three coordinate components of each point in the downsampled point cloud, points with the same correspondence are grouped into one category, thereby obtaining eight point cloud clusters.

[0060] In step S410 of some embodiments, the downsampling parameter s is , where \(p\) and \(q\) are relatively prime positive integers, \(q < p < 2q\). When \(n = 0\), \(s = p / q\), and \(1 < s < 2\). When \(n\) is a natural number, \(s>2\), and the original point cloud can be downsampled once with a downsampling parameter of (the reconstruction method can be based on the disclosed method), and the downsampled point cloud is the new original point cloud to be compressed and the new downsampling parameter \(s\) n \(= p / q\), where \(p\) and \(q\) are relatively prime positive integers, \(q < p < 2q\).

[0061] In step S420 of some embodiments, for every \(p\) unit space of the downsampling parameter \(s = p / q\), the coordinates are quantized into \(q\) discrete values. During the quantization process, each coordinate value is mapped to the interval \([0, p - 1]\), and according to the ratio of \(p\) and \(q\), it is mapped to the new discrete value interval \([0, q - 1]\), so that the original continuous coordinate values are compressed into \(q\) discrete values. The coordinate components of the original point cloud include the horizontal coordinate component, the vertical coordinate component, and the vertical coordinate component. For each point of the original point cloud, the corresponding horizontal coordinate component, vertical coordinate component, and vertical coordinate component of the corresponding point are quantized according to the downsampling parameter, the corresponding horizontal coordinate component, vertical coordinate component, and vertical coordinate component are divided by the downsampling parameter respectively, and the calculation result of the division operation is rounded to obtain the discrete value of the horizontal coordinate component, the discrete value of the vertical coordinate component, and the discrete value of the vertical coordinate component of the corresponding point. Taking the quantization process of the horizontal coordinate component of the \(i\)-th point in the original point cloud as an example, the calculation method of quantization is expressed as: \(xd[i]=round(x[i]*q / p)\), where \(x[i]\) is the horizontal coordinate component of the \(i\)-th point in the original point cloud; \(xd[i]\) is the discrete value of the horizontal coordinate component of the \(i\)-th point in the original point cloud; \(round\) is the rounding operation; \(*\) represents the multiplication operation; \( / \) represents the division operation; \(q\) is the denominator of the downsampling parameter, and \(p\) is the numerator of the downsampling parameter.

[0062] Given an original point cloud, in the process of downsampling the original point cloud according to a downsampling parameter greater than 1 and less than 2, the horizontal coordinates, vertical coordinates and vertical coordinates of the points in the original point cloud will undergo non-uniform quantization, and there may be a one-to-one or one-to-two situation in the quantization process on each coordinate axis, that is, under each coordinate axis, one coordinate or two coordinates corresponding to the coordinate axis in the original point cloud can be matched to a coordinate of the coordinate axis in the downsampled point cloud. Specifically, for each point in the original point cloud, its coordinates may appear in the following eight situations: the horizontal coordinate, the vertical coordinate and the vertical coordinate are all one-to-one; the vertical coordinate and the vertical coordinate are one-to-one, and the horizontal coordinate is one-to-two; the horizontal coordinate and the vertical coordinate are one-to-one, and the vertical coordinate is one-to-two; the vertical coordinate is one-to-one, and the horizontal coordinate and the vertical coordinate are one-to-two; the vertical coordinate is one-to-one, and the horizontal coordinate and the vertical coordinate are one-to-two; the vertical coordinate is one-to-one, and the horizontal coordinate and the vertical coordinate are one-to-two; the horizontal coordinate is one-to-one, and the vertical coordinate and the vertical coordinate are one-to-two; the horizontal coordinate is one-to-one, and the vertical coordinate and the vertical coordinate are one-to-two; the horizontal coordinate is one-to-one, and the vertical coordinate and the vertical coordinate are one-to-two; the horizontal coordinate, the vertical coordinate and the vertical coordinate are one-to-one.

[0063] Specifically, the three coordinate components of each point in the original point cloud are quantized according to the downsampling parameters to obtain the discrete values ​​of the three coordinate components of each point (downsampled point cloud), and the frequency of occurrence of the discrete values ​​of each coordinate component is counted to identify repeated quantized values ​​according to the frequency, such as quantization collisions caused by the proximity of the original coordinates (the coordinates of the points in the original point cloud). The discrete values ​​of the coordinate components and the frequency of occurrence of the discrete values ​​are recorded in the histogram array hist to facilitate access to the data at any time. The correspondence between the coordinate components of the downsampled point cloud and the coordinate components of the original point cloud is determined according to the frequency. When n is 0, the frequency of the coordinate component is 1 or 2. If the frequency is 1, it means that the correspondence between the coordinate components of the downsampled point cloud and the coordinate components of the original point cloud is one-to-one. If the frequency is 2, it means that the correspondence between the coordinate components of the downsampled point cloud and the coordinate components of the original point cloud is one-to-two.

[0064] In step S430 of some embodiments, according to the correspondence between the three coordinate components of each point in the downsampled point cloud, the points with the same correspondence are clustered into one category, and the points with different correspondence are clustered into different categories, thereby obtaining 8 point cloud clusters, each of which has a cluster category value.

[0065] Select the discrete values ​​with a frequency greater than 1 from the discrete values ​​of each coordinate component to obtain the repeated discrete values ​​of the coordinate component. Cluster the original point cloud according to the repeated discrete values ​​of each coordinate component in the downsampled point cloud and the original point cloud, and map the classification results in the three-dimensional space to a unique cluster number based on these three-dimensional voxel coordinate information to obtain the cluster category value, which is any integer value from 0 to 7, so as to cluster the original point cloud into eight different categories.

[0066] Specifically, for repeated discrete values, the repeated discrete values ​​of the coordinate components are stored in the array dup, and the coordinate deviations corresponding to the repeated discrete values ​​are calculated, and the coordinate deviations are stored in the array x_ch. The calculation process of the coordinate deviation is as follows: the repeated discrete values ​​of the corresponding coordinate components of the points in the downsampled point cloud are upsampled according to the downsampling parameters, that is, the downsampling parameters and the repeated discrete values ​​of the corresponding coordinate components of the points in the downsampled point cloud are multiplied, and the multiplication result is rounded to obtain the intermediate component. Taking the horizontal coordinate component as an example, the repeated discrete value of the horizontal coordinate component of the point in the downsampled point cloud is x_d, and the intermediate component is expressed as round(p / q*x_d), where round represents the rounding operation. There is a mapping relationship between the points of the original point cloud and the points of the downsampled point cloud. The coordinate component of the corresponding point in the original point cloud is subtracted from the intermediate component to obtain the coordinate deviation, which is used to correct the quantization error of the downsampled point cloud.

[0067] For each point in the downsampled point cloud, each coordinate component of the point in the downsampled point cloud is modulo calculated according to the denominator q of the downsampled parameter to obtain the remainder of each coordinate component of the point in the downsampled point cloud. If the remainder of the coordinate component is not in the array dup, the coordinate component of the point in the downsampled point cloud is upsampled according to the downsampled parameter, and the three upsampled coordinate components are added to the coordinate deviations of the corresponding coordinate components to obtain the reference coordinate component. The correspondence between the coordinate component of the point in the downsampled point cloud and the reference coordinate component is determined. If the correspondence is one-to-one, the component clustering value of the coordinate component is 0. If the correspondence is one-to-two, the component clustering value of the coordinate component is 1. The component clustering values ​​of each coordinate component of the point in the downsampled point cloud are weighted to obtain the cluster category value. The component clustering values ​​of each coordinate component of the point in the downsampled point cloud form a three-dimensional classification vector child. Each element of the three-dimensional classification vector is weighted to determine the cluster category value. The calculation formula of the cluster category value is expressed as: c=abs(child[0])+2*abs(child[1])+4*abs(child[2]), Among them, c is the cluster category value; abs represents the absolute value; child[0] represents the component cluster value of the horizontal axis component; child[1] represents the component cluster value of the vertical axis component; child[2] represents the component cluster value of the vertical axis component; * represents the multiplication operation.

[0068] For example, the horizontal coordinate component of a point in the downsampled point cloud The horizontal coordinate component of the point in the original point cloud It is a one-to-two relationship, child[0] is 1, and the ordinate component The ordinate component of the point in the original point cloud If it is a one-to-two relationship, then child[1] is 1, and the vertical coordinate component The vertical coordinate component of the point in the original point cloud It is a one-to-one relationship, child[2] is 0, and the cluster category value is 1+2*1+4*0=3.

[0069] The cluster category value of the point cloud cluster is used to indicate the number of points in the original point cloud corresponding to the points in the downsampled point cloud, and the number of points in the original point cloud with a mapping relationship The relationship with the cluster category value c is as follows: .

[0070] For c=0, the downsampling of the original point cloud is a complete one-to-one quantization, and each downsampled point corresponds to one original point in the original point cloud. For c=1, 2, and 4, during the downsampling of the original point cloud, the quantization of one coordinate axis is one-to-two, and each point in the downsampled point cloud may correspond to two original points in the original point cloud. For c=3, 5, and 6, during the downsampling of the original point cloud, the quantization of two coordinate axes is one-to-two, and each point in the downsampled point cloud may correspond to four original points in the original point cloud. For c=7, during the downsampling of the original point cloud, the quantization of three coordinate axes is one-to-two, and each point in the downsampled point cloud corresponds to eight original points in the original point cloud. The actual number of corresponding original points is determined by the target information of the point.

[0071] Through the above steps S410 to S430, the point coordinates in the continuous space are non-uniformly quantized and mapped to the discrete space, which can distinguish different categories of point cloud data, so as to reconstruct the point cloud category by category according to different categories of point cloud data, and then restore the original point cloud with high quality, thereby improving the quality of point cloud compression.

[0072] See also Figure 5 , Figure 5 The figure is a schematic diagram of the correspondence between the horizontal coordinate components (x coordinate components) of the point cloud before and after quantization. The downsampling parameter is expressed as S=4 / 3. The upper layer is the x coordinate components of the original point clouds a, b, c and d, and the lower layer is the x coordinate components of the downsampled point clouds A, B and C. The arrows c and d point to the coordinate components of the same downsampled point cloud C, indicating that the original point clouds c and d have a quantization collision. The quantized repeated discrete value 2 in the downsampled point cloud C is recorded in the index array dup. The repeated discrete values ​​of point C are upsampled according to the downsampling parameters to obtain the middle component 3. The x coordinate component of the original point cloud corresponding to the repeated discrete value is [2,3]. The x coordinate component [2,3] is subtracted from the middle component 3 to obtain the coordinate deviation [-1,0], which is stored in the array x_ch.

[0073] According to q=3, the x-coordinate components of A, B, and C in the downsampled point cloud are modulo calculated, and the remainders of A, B, and C are 0, 1, and 2, respectively. The remainders of the x-coordinate components of points A and B are 0 and 1, respectively, and are not in the index array dup, so the component cluster value of the x-coordinate component is 0. The remainder of the x-coordinate component of point C is 2. In the index array dup, the coordinate component of point C in the downsampled point cloud is upsampled according to the downsampling parameters, that is, the horizontal coordinate component of point C is multiplied by s, and the upsampled x-coordinate component is added to the coordinate deviation in the array x_ch, and the reference coordinate components are obtained after rounding to 2 and 3. Since the correspondence between point C in the downsampled point cloud and the reference coordinate component is one-to-two, the component cluster value of point C is 1.

[0074] See also Figure 6 , Figure 6 The figure is a schematic diagram of the correspondence between the x-coordinate components of the point cloud before and after quantization, and the downsampling parameter is expressed as S=7 / 5. The upper layer is the x-coordinate components of the original point clouds a, b, c, d, e, f, and g, and the lower layer is the x-coordinate components of the downsampled point clouds A, B, C, D, and E. The two arrows b and c point to the x-coordinate components of the same downsampled point cloud B, and the two arrows f and g point to the x-coordinate components of the same downsampled point cloud E, indicating that the original point clouds b and c, and f and g have quantization collisions. Taking points B and E as examples, the repeated discrete values ​​of quantization are 1 and 4, and [1,4] is recorded in the index array dup. According to the downsampling parameters, the repeated discrete values ​​of points B and E are upsampled to obtain intermediate components of 1 and 6. The x-coordinate components of the points in the original point cloud corresponding to the repeated discrete values ​​are [1,2,5,6]. Subtract the x-coordinate component [1,2] from the intermediate component 1, and subtract the x-coordinate component [5,6] from the intermediate component 6 to obtain the coordinate deviation [0,1,-1,0]. The coordinate deviation is stored in the array x_ch.

[0075] Perform modulo calculations on the x - coordinate components of points A, B, C, D, and E in the down - sampled point cloud according to q = 5. The remainders of A, B, C, D, and E are 0, 1, 2, 3, and 4 respectively. The remainders of the x - coordinate components of A, C, and D are 0, 2, and 3 respectively, and these remainders are not in the index array dup, so the component clustering value of the x - coordinate component is 0. The remainder of the x - coordinate component of point B is 1, and the remainder of the x - coordinate component of point E is 4. These remainders are in the index array dup. Multiply the x - coordinate component of point B by s, add the coordinate deviation [0, 1] in the array x_ch and take the integer part to get the reference coordinate components as 1 and 2. Since the correspondence between the down - sampled point cloud point B and the reference coordinate components is one - to - two, the component clustering value of point B is 1. Multiply the x - coordinate component of point E by s, add the coordinate deviation [-1, 0] in the array x_ch and take the integer part to get the reference coordinate components as 5 and 6. Since the correspondence between the down - sampled point cloud point B and the reference coordinate components is one - to - two, the component clustering value of point E is 1.

[0076] Please refer to Figure 7 , in some embodiments, step S140 may include but is not limited to steps S710 to S730: Step S710, the down - sampling parameter is , where n is a positive integer, and p and q are relatively prime positive integers, and q < p < 2q; Step S720, according to n, p, and q, determine the correspondence between the coordinate components of the down - sampled point cloud and the coordinate components of the original point cloud. The correspondence includes a one - to - n + 1 or one - to - n + 2 relationship; Step S730, according to the correspondence of the 3 - coordinate components of each point in the down - sampled point cloud, cluster those with the same correspondence into one category to obtain 8 point - cloud clustering clusters.

[0077] In step S710 of some embodiments, the down - sampling parameter s is , p and q are relatively prime positive integers, q < p < 2q, and when n is a positive integer, s is a non - integer greater than 2.

[0078] In step S720 of some embodiments, according to n, p, and q, determine that the down - sampling parameter is a non - integer greater than 2. Refer to step S420 to determine the correspondence between the coordinate components of the down - sampled point cloud and the coordinate components of the original point cloud according to the down - sampling parameter. The correspondence includes a one - to - n + 1 or one - to - n + 2 relationship. Figure 8 Is a schematic diagram of the correspondence before and after quantization of the x - coordinate component of the point cloud, as Figure 8As shown, the downsampling parameter is expressed as S=5 / 2, and the downsampling parameter is a non-integer greater than 2. In the process of quantizing the original point cloud according to the downsampling parameter, the corresponding relationship between the coordinate components of the downsampled point cloud and the coordinate components of the original point cloud is one-to-many. For example, the coordinate components of point A correspond to the coordinate components of points a and b in the original point cloud, and the coordinate components of point B correspond to the coordinate components of points c, d, and e in the original point cloud.

[0079] In step S730 of some embodiments, according to the correspondence between the three coordinate components of each point of the downsampled point cloud, the points with the same correspondence are grouped into one category, and the points with different correspondence are grouped into different categories, thereby obtaining 8 point cloud clusters. The correspondence between the coordinate components of the downsampled point clouds in the 8 point cloud clusters is as follows: the horizontal coordinate, the vertical coordinate and the vertical coordinate are all a pair of n+1; the vertical coordinate and the vertical coordinate are a pair of n+1, and the horizontal coordinate is a pair of n+2; the horizontal coordinate and the vertical coordinate are a pair of n+1, and the vertical coordinate is a pair of n+2; the horizontal coordinate and the vertical coordinate are a pair of n+1, and the vertical coordinate is a pair of n+2; the vertical coordinate is a pair of n+1, and the horizontal coordinate and the vertical coordinate are a pair of n+2; the vertical coordinate is a pair of n+1, and the horizontal coordinate and the vertical coordinate are a pair of n+2; the vertical coordinate is a pair of n+1, and the horizontal coordinate and the vertical coordinate are a pair of n+2; the horizontal coordinate is a pair of n+1, and the horizontal coordinate and the vertical coordinate are a pair of n+2; the horizontal coordinate is a pair of n+1, and the vertical coordinate and the vertical coordinate are a pair of n+2; the horizontal coordinate is a pair of n+1, and the vertical coordinate and the vertical coordinate are a pair of n+2; the horizontal coordinate is a pair of n+1, and the vertical coordinate and the vertical coordinate are a pair of n+2; the horizontal coordinate, the vertical coordinate and the vertical coordinate are a pair of n+2.

[0080] The point cloud clusters have cluster category values, and the cluster category values ​​are any integer values ​​from 0 to 7, so as to cluster the original point cloud into eight different categories. The cluster category values ​​can be calculated with reference to step S430, which will not be described in detail here.

[0081] Through the above steps S710 to S730, different categories of downsampled point clouds can be distinguished, so that the downsampled point clouds can be reconstructed according to the characteristics of the point clouds of different categories, thereby improving the quality of point cloud encoding.

[0082] At the decoding end, after the decoded downsampled point cloud is geometrically scaled up, the above eight clustering situations may occur. Only when the downsampling parameter s is greater than 1 and less than 2 and the clustering category value c=0 will no distortion be introduced and the point cloud information can be fully restored. In other cases, distortion will be introduced due to the addition of erroneous points or the omission of correct points, and the original point cloud cannot be fully restored. If the original point cloud is to be fully restored, the clustering category value of each point in the downsampled point cloud needs to be 0, but this situation is extremely rare. In order to restore the original point cloud with high quality, other information other than the downsampled point cloud is required. The embodiment of the present application uses neighbor placeholder information as supplementary information. Point cloud reconstruction can be assisted according to neighbor placeholder information, so that point cloud reconstruction can not directly scale up the downsampled point cloud due to the lack of neighbor placeholder information, but can use these neighbor placeholder information to perform point cloud geometric super-resolution, thereby reducing or even removing distortion. In order to improve the quality of point cloud coding, the embodiment of the present application uses target information and neighbor placeholder information as information pairs for network training. An interpolation network is trained according to each category of information pairs to interpolate the downsampled point cloud of the corresponding category based on the interpolation network, thereby achieving high-quality point cloud reconstruction.

[0083] See also Fig. 9 In some embodiments, step S150 may include but is not limited to steps S910 to S930: Step S910, according to the neighbor occupancy information of each point in each point cloud cluster, input the interpolation network of each cluster to obtain the predicted target information of each point; Step S920, calculating the loss function of the interpolation network of each cluster according to the predicted target information and target information of each point in each point cloud cluster; Step S930 , minimizing the loss function to adjust the interpolation network of each cluster to obtain optimized interpolation network parameters of each cluster.

[0084] In step S910 of some embodiments, the original point cloud can be decomposed into a downsampled point cloud and neighbor placeholder information, different point cloud clustering clusters have different cluster category values, different cluster category values ​​correspond to different categories, and the neighbor placeholder information of the points in the downsampled point cloud with the same cluster category value is input into the interpolation network corresponding to the cluster category value. The downsampled point cloud is enhanced using the interpolation network to obtain the predicted target information of the points in the downsampled point cloud.

[0085] It should be noted that when the downsampling parameter is greater than 1 and less than 2, if the clustering category value of the point cloud clustering cluster is 0, no additional neighbor occupancy information is required, and the training of the interpolation network for this clustering cluster can be skipped. If the clustering category values of the point cloud clustering cluster are 1, 2, and 4, the interpolation network of this clustering cluster needs to be used to predict the occupancy information of two points. If the clustering category values of the point cloud clustering cluster are 3, 5, and 6, the interpolation network of this clustering cluster needs to be used to predict the occupancy information of four points. If the clustering category value corresponding to the category is 7, the interpolation network of this clustering cluster needs to be used to predict the occupancy information of eight points. When the downsampling parameter is greater than 2, if the clustering category value of the point cloud clustering cluster is 0, the interpolation network of this clustering cluster needs to be used to predict the occupancy information of points. If the clustering category values of the point cloud clustering cluster are 1, 2, and 4, the interpolation network of this clustering cluster needs to be used to predict the occupancy information of points. If the clustering category values of the point cloud clustering cluster are 3, 5, and 6, the interpolation network of this clustering cluster needs to be used to predict the occupancy information of points. If the clustering category value corresponding to the category is 7, the interpolation network of this clustering cluster needs to be used to predict the occupancy information of points. Point cloud geometry often has geometric similarity, that is, when the neighbor occupancy information of points is the same, the target information of this point is often also the same.

[0086] To improve the efficiency of point cloud compression, the interpolation network can adopt a lightweight network, which has fewer model parameters. It can be a multi-layer perceptron with only one hidden layer, and the number of neurons in the hidden layer can be set to a very small positive number such as 32. The multi-layer perceptron includes an input layer, a hidden layer, and an output layer. After the hidden layer, a Sine activation function is connected to provide non-linear fitting ability and learn the high-frequency information of the point cloud data. After the output layer, a Sigmoid activation function is connected to constrain the output value range to [0,1].

[0087] In step S920 of some embodiments, the downsampling parameter s is , where p and q are relatively prime positive integers, q < p < 2q, and when n is 0, the prediction of the target information can be regarded as =2, 4, 8 binary classification problems. The binary cross-entropy loss function is used for overfitting of the interpolation network. According to the predicted target information and the target information of each point in each point cloud clustering cluster, the loss function of the interpolation network of each clustering cluster is calculated. The calculation method of the loss function is expressed as: , where is the loss function; BCE is the binary cross-entropy loss function; is the number of points in the downsampled point cloud with the clustering category value of c; is the interpolation network; It is the neighbor occupancy information of the jth point in the point cloud cluster; It is the target information of the jth point in the point cloud cluster; are the network parameters of the interpolation network; It is the predicted target information of the jth point in the point cloud cluster output by the interpolation network.

[0088] In step S930 of some embodiments, the loss function of each point cloud cluster is minimized to adjust the network parameters of the interpolation network of the corresponding point cloud cluster, so as to obtain optimized interpolation network parameters of each point cloud cluster.

[0089] Through the above steps S910 to S930, a lightweight overfitting interpolation network can be obtained to accurately predict the target information at the decoding end based on the lightweight overfitting interpolation network, so as to restore the original point cloud from the downsampled point cloud with high quality at the decoding end.

[0090] In step S160 of some embodiments, in order to maintain the integrity of the point cloud data, the downsampled point cloud is losslessly compressed by a point cloud compression algorithm based on geometry (G-PCC), and the compressed bitstream file is used as the downsampled point cloud coding data. The interpolation network parameters of the interpolation network of each point cloud cluster are losslessly compressed by the fpzip compression library to obtain network coding data. The interpolation network constructed based on the network coding data is used to reconstruct the point cloud of the downsampled point cloud coding data at the decoding end to obtain high-precision three-dimensional coordinate information and restore the complete three-dimensional model of the object.

[0091] See also Fig.10 In some embodiments, the three-dimensional point cloud decoding method is applied to the decoding end, which may include but is not limited to steps S1010 to S1050: Step S1010, obtaining downsampling parameters, downsampling point cloud and interpolation network parameters; Step S1020, determining the neighbor occupancy information of each point in the downsampled point cloud; Step S1030, clustering the downsampled point cloud according to the downsampled parameters to obtain downsampled point cloud clusters; Step S1040, according to the neighbor occupancy information of each point in each downsampled point cloud cluster, input the interpolation network corresponding to the interpolation network parameters of each cluster to obtain the predicted target information of each point; Step S1050, obtaining a reconstructed point cloud according to the predicted target information of each point of the down-sampled point cloud.

[0092] In step S1010 of some embodiments, the encoder merges the downsampled point cloud coded data, the network coded data and the downsampled parameters, and forms a final compressed file for storage or transmission by merging the code stream file after lossless compression. The encoder sends the compressed file to the decoder, so that the decoder reconstructs the downsampled point cloud using the interpolation network. The decoder receives the compressed file, stores the compressed file in a storage medium, reads the compressed file from the storage medium, obtains the downsampled parameters from the compressed file, decodes the downsampled point cloud coded data of the compressed file through a geometry-based point cloud compression algorithm to obtain the downsampled point cloud, and decodes the network coded data through the fzip compression library to obtain the interpolation network parameters.

[0093] In step S1020 of some embodiments, referring to steps S310 to S330, a neighbor coordinate information set of each point in the downsampled point cloud is determined based on the geometric coordinates and preset distance parameters of each point in the downsampled point cloud, the downsampled point cloud is voxelized to obtain a downsampled point cloud represented by voxelization, and based on the downsampled point cloud represented by voxelization and the neighbor coordinate information set, the occupancy status of all neighbors in the neighbor coordinate information set of each point in the downsampled point cloud is queried to obtain the neighbor occupancy information of each point in the downsampled point cloud.

[0094] In step S1030 of some embodiments, according to steps S410 to S430 or steps S710 to S730, the downsampled point cloud is clustered according to the downsampled parameters to obtain downsampled point cloud clusters.

[0095] In step S1040 of some embodiments, the neighbor occupancy information of each point in each down-sampled point cloud cluster is input into the interpolation network corresponding to the interpolation network parameters of each down-sampled point cloud cluster to obtain the predicted target information of each point.

[0096] In step S1050 of some embodiments, point cloud reconstruction is performed on each point of the down-sampled point cloud according to the predicted target information of each point of the down-sampled point cloud, and point clouds reconstructed from different sampling point cloud clusters are merged and deduplicated to obtain a reconstructed point cloud.

[0097] Through the above steps S1010 to S1050, the original point cloud can be reconstructed from the downsampled point cloud with high quality, thereby achieving a smaller bit rate and reducing visual distortion. The interpolation network is very lightweight, so that the codeword overhead brought by transmitting these networks can achieve a better rate-distortion balance point compared to the improvement in point cloud quality, saving bit rate while reducing the distortion of the reconstructed point cloud. In addition, the interpolation network is trained at the encoding end, and the time overhead added at the decoding end is only the time overhead of the interpolation network acting on the decoded point cloud, which improves the efficiency of point cloud reconstruction.

[0098] To further improve performance, multiple categories of interpolation networks can be combined into an output dimension of This not only reduces computational and storage overhead, but also further reduces codeword overhead. It should be noted that since the merged network needs to predict the interpolation network outputs of multiple categories at the same time, its prediction accuracy may be reduced, resulting in reduced interpolation accuracy and increased visual distortion. Therefore, it is necessary to control the model training process more finely to achieve better performance and obtain a better rate-distortion balance point.

[0099] The three-dimensional point cloud encoding method of the embodiment of the present application includes: in the encoding stage, the input point cloud is clustered according to the fractional downsampling rate, and mapping data is constructed by category based on the clustered point cloud and the downsampled point cloud, and the data is used to train a lightweight interpolation network by category, and the network parameters of the trained interpolation network and the downsampled point cloud are respectively compressed and encoded and packaged into a bitstream file. In the decoding stage, the trained network parameters are decompressed from the bitstream file to reconstruct the interpolation network and decode the downsampled point cloud, the decoded downsampled point cloud is clustered according to the downsampling rate, and the interpolation network is applied to the clustered downsampled point cloud by category to enhance the point cloud, thereby achieving smaller visual distortion under the condition of lower bit rate and reaching the ideal rate-distortion equilibrium point.

[0100] The 3D point cloud encoding method of the embodiment of the present application was tested on the MPEG Cat1 dataset, using the octree-based G-PCC software MPEG-PCC-TMC13 v27.0 as the lossless encoder for the downsampled point cloud, and the Adam optimizer was used to train the interpolation network, with the learning rate set to 0.001 and a total of 150 training rounds. Compared with the Peak Signal-to-Noise Ratio (PSNR) of G-PCC, the 3D point cloud encoding method of the embodiment of the present application significantly improves the quality of the decoded point cloud, with an average PSNR of 72.38%, so that the downsampled point cloud can be restored to the original point cloud with high quality and reduced visual distortion.

[0101] See also Fig.11 The embodiment of the present application further provides a three-dimensional point cloud encoding device, which can implement the above three-dimensional point cloud encoding method, and the three-dimensional point cloud encoding device includes: The first acquisition module 1110 is used to acquire the original point cloud and the downsampling parameter; the downsampling parameter is a non-integer greater than 1; A downsampling module 1120 is used to downsample the original point cloud according to the downsampling parameters to obtain the downsampled point cloud and the target information of each point in the downsampled point cloud; A first determination module 1130 is used to determine neighbor occupancy information of each point in the downsampled point cloud; A first clustering module 1140 is used to cluster the downsampled point cloud according to the downsampling parameters to obtain point cloud clusters; The training module 1150 is used to train the interpolation network according to the target information and neighbor occupancy information of each point in each point cloud cluster, so as to obtain the optimized interpolation network parameters of each cluster; The coding data generation module 1160 is used to losslessly encode the downsampled point cloud to obtain the downsampled point cloud coding data, and encode the interpolation network parameters of each cluster to obtain the network coding data.

[0102] The specific implementation of the three-dimensional point cloud encoding device is basically the same as the specific implementation of the above-mentioned three-dimensional point cloud encoding method, and will not be repeated here.

[0103] See also Fig.12 The embodiment of the present application further provides a three-dimensional point cloud decoding device, which can implement the above three-dimensional point cloud decoding method, and the three-dimensional point cloud decoding device includes: A second acquisition module 1210 is used to acquire downsampling parameters, downsampling point clouds and interpolation network parameters; A second determination module 1220 is used to determine neighbor occupancy information of each point in the downsampled point cloud; A second clustering module 1230 is used to cluster the downsampled point cloud according to the downsampled parameters to obtain downsampled point cloud clusters; The inference module 1240 is used to obtain the predicted target information of each point according to the neighbor occupancy information of each point in each downsampled point cloud cluster, input the interpolation network corresponding to the interpolation network parameters of each cluster; The reconstruction module 1250 is used to obtain a reconstructed point cloud according to the predicted target information of each point of the down-sampled point cloud.

[0104] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned three-dimensional point cloud encoding method or three-dimensional point cloud decoding method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0105] See also Fig.13 , Fig.13 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 1310 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 1320 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1320 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1320, and the processor 1310 calls and executes the three-dimensional point cloud encoding method or three-dimensional point cloud decoding method of the embodiments of this application; Input / output interface 1330, used to implement information input and output; The communication interface 1340 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.); The bus 1350 transmits information between the various components of the device (e.g., the processor 1310, the memory 1320, the input / output interface 1330, and the communication interface 1340); The processor 1310 , the memory 1320 , the input / output interface 1330 , and the communication interface 1340 are connected to each other in communication within the device via a bus 1350 .

[0106] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0107] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0108] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0110] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0111] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0112] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0116] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A three-dimensional point cloud encoding method, characterized in that: The method comprises: Obtaining an original point cloud and a downsampling parameter; the downsampling parameter is a non-integer greater than 1; Down-sampling the original point cloud according to the down-sampling parameters to obtain a down-sampled point cloud and target information of each point in the down-sampled point cloud; Determine neighbor occupancy information of each point in the downsampled point cloud; Clustering the downsampled point cloud according to the downsampled parameters to obtain point cloud clusters; According to the target information and neighbor occupancy information of each point in each point cloud cluster, an interpolation network is trained according to the cluster to obtain optimized interpolation network parameters of each cluster; The downsampled point cloud is losslessly encoded to obtain downsampled point cloud encoding data, and the interpolation network parameters of each cluster are encoded to obtain network encoding data.

2. The method according to claim 1, characterized in that The downsampling the original point cloud according to the downsampling parameters to obtain the downsampled point cloud and target information of each point in the downsampled point cloud includes: quantizing the coordinate component of each point in the original point cloud according to the downsampling parameter to obtain the downsampled point cloud; Each point in the downsampled point cloud corresponds to one or more points in the original point cloud, and the target information of each point in the downsampled point cloud is the placeholder information of the corresponding point in the original point cloud.

3. The method according to claim 1, characterized in that: The determining of neighbor occupancy information of each point in the downsampled point cloud includes: Determine a set of neighbor coordinate information of each point in the downsampled point cloud according to the geometric coordinates of each point in the downsampled point cloud and a preset distance parameter; Performing voxel processing on the downsampled point cloud to obtain a downsampled point cloud represented by voxelization; According to the downsampled point cloud represented by voxelization and the neighbor coordinate information set, the occupancy status of all neighbors in the neighbor coordinate information set of each point in the downsampled point cloud is queried to obtain the neighbor occupancy information of each point in the downsampled point cloud.

4. The method according to claim 1, characterized in that The step of clustering the downsampled point cloud according to the downsampled parameter to obtain a point cloud cluster includes: The downsampling parameters are , where n is a natural number, p and q are mutually prime positive integers, and q <p<2q; Determine a correspondence between coordinate components of the downsampled point cloud and coordinate components of the original point cloud according to p and q, wherein the correspondence includes a one-to-one or one-to-two relationship; According to the correspondence between the three coordinate components of each point of the downsampled point cloud, the points with the same correspondence are clustered into one category to obtain 8 point cloud clusters.

5. The method according to claim 1, characterized in that The step of clustering the downsampled point cloud according to the downsampled parameter to obtain a point cloud cluster includes: The downsampling parameters are , where n is a positive integer, p and q are mutually prime positive integers, and q <p<2q; Determine the correspondence between the coordinate components of the downsampled point cloud and the coordinate components of the original point cloud according to the n, p and q, wherein the correspondence includes a pair of n+1 or a pair of n+2; According to the correspondence between the three coordinate components of each point of the downsampled point cloud, the points with the same correspondence are clustered into one category to obtain 8 point cloud clusters.

6. The method according to claim 1, characterized in that The interpolation network is trained according to the target information and neighbor occupancy information of each point in each point cloud cluster to obtain optimized interpolation network parameters of each cluster, including: According to the neighbor occupancy information of each point in each point cloud cluster, the interpolation network of each cluster is input to obtain the predicted target information of each point; Calculating the loss function of the interpolation network of each cluster according to the predicted target information and the target information of each point in each point cloud cluster; The interpolation network parameters of each cluster are adjusted by minimizing the loss function to obtain optimized interpolation network parameters of each cluster.

7. A three-dimensional point cloud decoding method, characterized in that: The method comprises: Get downsampling parameters, downsampling point cloud and interpolation network parameters; Determine neighbor occupancy information of each point in the downsampled point cloud; Clustering the downsampled point cloud according to the downsampled parameters to obtain downsampled point cloud clusters; Inputting the neighbor occupancy information of each point in each of the downsampled point cloud clusters into the interpolation network corresponding to the interpolation network parameters of each cluster to obtain the predicted target information of each point; A reconstructed point cloud is obtained according to the predicted target information of each point of the downsampled point cloud.

8. A three-dimensional point cloud encoding device, characterized in that: The device comprises: A first acquisition module is used to acquire an original point cloud and a downsampling parameter; the downsampling parameter is a non-integer greater than 1; A downsampling module, used to downsample the original point cloud according to the downsampling parameters to obtain a downsampled point cloud and target information of each point in the downsampled point cloud; A first determination module is used to determine neighbor occupancy information of each point in the downsampled point cloud; A first clustering module, used for clustering the down-sampled point cloud according to the down-sampling parameters to obtain point cloud clusters; A training module, used to train an interpolation network according to the target information and neighbor occupancy information of each point in each point cloud cluster, so as to obtain optimized interpolation network parameters of each cluster; The coding data generation module is used to losslessly encode the downsampled point cloud to obtain downsampled point cloud coding data, and encode the interpolation network parameters of each cluster to obtain network coding data.

9. A three-dimensional point cloud decoding device, characterized in that: The device comprises: The second acquisition module is used to obtain downsampling parameters, downsampling point clouds and interpolation network parameters; A second determination module is used to determine neighbor occupancy information of each point in the downsampled point cloud; A second clustering module, used for clustering the down-sampled point cloud according to the down-sampling parameters to obtain down-sampled point cloud clusters; An inference module, used for inputting the neighbor occupancy information of each point in each of the downsampled point cloud clusters into the interpolation network corresponding to the interpolation network parameters of each cluster to obtain the predicted target information of each point; The reconstruction module is used to obtain a reconstructed point cloud according to the predicted target information of each point of the down-sampled point cloud.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the three-dimensional point cloud encoding method according to any one of claims 1 to 6 or the three-dimensional point cloud decoding method according to claim 7 when executing the computer program.

Images