Method for restoring 3D point cloud data from voxel based feature data

Abstract

The 3D point cloud restoration method of the present invention is a method for restoring 3D point cloud data from voxel-based feature data extracted by a 3D feature extractor through a conduction model that is recombined in reverse order after training a 3D feature extractor to restore conduction blocks corresponding to blocks divided based on a downsampling layer at each step, with the output of each block being restored as the input of the block.

Description

Technology Field

[0001] The present invention relates to a vision data restoration technology using an artificial intelligence model, and more specifically, to a technology for restoring a 3D point cloud from 3D feature data extracted from a voxelized 3D point cloud. Background Technology

[0003] As the utilization of artificial intelligence models expands, the importance of data collection and sharing during model training and inference processes is being emphasized. Along with this, the issue of personal information leakage within data has emerged, leading to various studies and institutional efforts to prevent it. In the field of AI, attempts have been made to address this problem by sharing anonymized feature data as it passes through the model's backbone, rather than sharing the original data directly. However, model transfer attacks, which aim to reconstruct the original data from feature data, are currently being actively researched, and to counter this, attack and defense techniques are evolving in a complementary manner.

[0004] Conventional techniques regarding model transfer attacks have primarily focused on the 2D image domain. Representative methods include designing a regression model with the RGB channel values ​​of an image as the target for restoration, or using a conditional generative model that inputs feature data as a condition to a generative model. In regression models, the resolution of the given feature data is increased, and the original image is restored using the pixel-level root mean square error as the loss function. In conditional generative models, after training an image generation model, the given feature data is input as a condition to generate data that is close to the original.

[0005] In particular, the need for privacy protection in 3D point cloud data is rapidly increasing. This is because various types of personal information can be revealed through rich 3D shape information. For example, personal information can be identified through facial recognition and personal re-identification, and behavioral patterns can be identified through human pose estimation and activity recognition.

[0006] Unlike 2D images, data in 3D point clouds has sparsity, so it is difficult to properly restore the original if conventional techniques are applied as is. In particular, as the depth of the backbone increases, data sparsity decreases rapidly, making it even more difficult to accurately restore the sparsity of the original data. The problem to be solved

[0008] The present invention aims to provide a method for accurately restoring the original 3D point cloud from extracted feature data, even for 3D point clouds where data has sparsity unlike 2D images. means of solving the problem

[0010] A three-dimensional point cloud reconstruction method according to one aspect of the present invention is performed in a computing device comprising one or more processors and a memory that stores program instructions executable by said processors, and includes a step of determining the number of conduction blocks, a conduction block learning step, a reconstruction model configuration step, and a reconstruction step.

[0011] The step of determining the number of conduction blocks is a step of determining the number of conduction blocks N corresponding to each divided block by dividing a 3D feature extractor, which extracts 3D feature data based on convolution operations by voxelizing a 3D point cloud, into blocks based on a downsampling layer.

[0012] The conduction block learning step is a step of learning each of the N conduction blocks, and is a step of learning each conduction block to restore the form of input data of the corresponding 3D feature extractor block from the 3D feature data output by the block of the corresponding 3D feature extractor.

[0013] The reconstruction model construction step is the step of constructing a 3D point cloud reconstruction model by recombining N learned conduction blocks in reverse order.

[0014] The restoration step is a step of restoring a 3D point cloud by providing the 3D feature data finally output by the 3D feature extractor as input data to a 3D point cloud restoration model.

[0015] According to one aspect of the present invention, the conduction block learning step may include a voxel occupancy classification step that generates a binary encoding label by binary classifying whether a voxel is a voxel with an actual point for 3D feature data restored in the form of input data of a block of a 3D feature extractor corresponding to each conduction block.

[0016] In this case, the voxel occupancy classification step can use the sigmoid focal function as the loss function for voxel occupancy classification.

[0017] According to one aspect of the present invention, the conduction block learning step may further include a occupancy spread suppression step that suppresses the spread of voxel occupancy based on binary encoding labels generated in the voxel occupancy classification step, while performing regression on the channel values ​​of the 3D feature data when learning a conduction block other than the conduction block corresponding to the first block of the 3D feature extractor.

[0018] According to one aspect of the present invention, the restoration step may restore the output data of the restoration model into a three-dimensional point cloud, and may restore the three-dimensional point cloud by creating a point at the center coordinates of a voxel classified as having an actual point. Effects of the invention

[0020] According to the present invention, even for 3D point clouds where data has sparsity unlike 2D images, the original 3D point cloud can be accurately restored from the extracted feature data. Brief explanation of the drawing

[0022] Figure 1 conceptually illustrates the process of restoring a three-dimensional point cloud from voxel-based feature data according to the prior art. Figure 2 is a flowchart of a method for restoring a 3D point cloud from voxel-based feature data. Figure 3 conceptually illustrates the process of the present invention restoring a three-dimensional point cloud from voxel-based feature data. Figure 4 conceptually illustrates the voxel occupancy classification process. Figure 5 conceptually illustrates learning in the form of conduction blocks. Figure 6 illustrates the concept of the present invention restoring using a voxel center point. Specific details for implementing the invention

[0023] The foregoing and additional aspects are embodied in the embodiments described with reference to the attached drawings. It is understood that the components of each embodiment may be combined in various ways within the embodiment unless otherwise stated or contradictory. Each block in the block diagram may represent a physical part in some cases, but in others, it may be a logical representation of a part of the function of a single physical part or a function spanning multiple physical parts. Sometimes, the entity of a block or part thereof may be a set of program instructions. These blocks may be implemented in whole or in part by hardware, software, or a combination thereof.

[0025] Figure 1 conceptually illustrates the process of reconstructing a 3D point cloud from voxel-based feature data according to the prior art. Generally, in conventional conduction attacks, it is difficult to reconstruct 3D point cloud data from intermediate feature data output by a 3D feature extractor at an intermediate stage before the final result is output. This is because, due to the sparsity of 3D point cloud data, most voxels become empty when voxelization is performed; however, zero-padding empty voxels and voxels that do not contain valid points at the origin (0, 0, 0) have identical representations and cannot be distinguished. Furthermore, as the feature extractor proceeds with feature extraction, occupied voxels become increasingly dispersed, leading to increased confusion between empty and occupied voxels. Figure 1 conceptually illustrates this problem by showing the dispersion of occupied voxels while reconstructing a 3D point cloud from extracted features. In other words, Figure 1 indicates that a much larger number of points can be reconstructed compared to the original 3D point cloud. In addition, this restoration trend becomes more pronounced as the layer depth of the feature extractor increases.

[0026] When a 3D point cloud is restored from features extracted by a feature extractor using conventional technology, problems arise such as the restored points being biased toward a specific location, namely the origin (0, 0, 0), and a much larger number of points being restored compared to the original data.

[0027] FIG. 2 is a flowchart of a method for reconstructing a three-dimensional point cloud from voxel-based feature data, and FIG. 3 conceptually illustrates the process of reconstructing a three-dimensional point cloud from voxel-based feature data according to the present invention. A method for reconstructing a three-dimensional point cloud according to one aspect of the present invention is performed in a computing device comprising one or more processors and a memory storing program instructions executable by said processors, and includes a step of determining the number of conduction blocks, a conduction block learning step, a reconstruction model configuration step, and a reconstruction step.

[0028] Figure 3(a) conceptually illustrates the process of extracting features through a 3D feature extractor, and Figure 3(b) conceptually illustrates the process of restoring a 3D point cloud from the extracted 3D features.

[0029] The computing device on which the three-dimensional point cloud restoration method of the present invention is performed is also referred to as a three-dimensional point cloud restoration device and is a computing device comprising one or more processors and memory that stores program instructions executable by the processors, and includes software concepts in addition to hardware. The three-dimensional point cloud restoration device may be a computer device that additionally includes a GPU (Graphic Processor Unit), a storage device, a display, an input device, etc., in addition to the processor and memory. The processor is a processor that executes program instructions, and the memory is connected to the processor and stores program instructions executable by the processor, data that the processor will use for computation, and data processed by the processor.

[0030] The step of determining the number of conduction blocks, the conduction block learning step, the reconstruction model configuration step, and the reconstruction step of the 3D point cloud reconstruction method can be implemented as a set of computer program instructions that execute on a processor, at least some of which functions.

[0031] The step of determining the number of conduction blocks is a step of determining the number of conduction blocks N corresponding to each divided block by voxelizing a 3D point cloud and dividing a 3D feature extractor, which extracts 3D feature data based on convolution operations, into blocks based on a downsampling layer (S1000).

[0032] 3D feature extractors may refer to layers that extract features for classification in a model that detects and classifies 3D objects based on voxels from 3D point cloud data, or layers that extract features for semantic segmentation in a model that performs 3D semantic segmentation based on voxels from 3D point cloud data. However, 3D feature extractors are not limited to these and may be layers that extract features in a model composed of layers that extract features through 3D convolution operations.

[0033] A 3D feature extractor typically includes one or more 3D convolution layers, and at least some of the layers consist of downsampling layers.

[0034] The present invention can be viewed as an inversion attack model for existing 3D feature extractors. The inversion attack model aims to discover the inverse process of the 3D feature extractor targeted for attack.

[0035] In the case of a voxel-based 3D feature extractor targeted for restoration by the present invention, a voxelization process is included in the initial stage to convert point cloud data into a grid format. Voxelization is 3D point cloud data (where k is the number of points) voxel grid It is converted to (where H, W, and D are the spatial dimensions of the grid). At this stage, the x, y, and z coordinate information of the points is channelized, and empty voxels without points are zero-padded, so the channel value becomes (0, 0, 0). In particular, during the downsampling process of the 3D feature extractor, the spatial dimensions decrease and the channel size increases, so the feature data This is generated, where N is the number of downsampling layers, > 3, , , is smaller than H, W, and D. The conduction attack model is downsampled features From the original voxel grid The goal is to restore it.

[0036] The reconstruction model of the present invention, that is, the conduction attack model, has a symmetrical structure with the 3D feature extractor. That is, the reconstruction model of the present invention reconstructs the original form by performing upsampling at the location where the 3D feature extractor downsampled. Accordingly, the conduction blocks constituting the reconstruction model of the present invention are composed of the same number as the blocks of the 3D feature extractor. Here, the blocks of the 3D feature extractor are divided based on the downsampling layer. Accordingly, the number of conduction blocks constituting the reconstruction model is determined by the number of blocks divided based on the downsampling layer of the 3D feature extractor, and conduction blocks corresponding to N blocks are prepared (S1010). The present invention defines this number as N.

[0037] The conduction block learning step is a step of learning each of the N conduction blocks, and is a step of learning (S1020) each conduction block to restore the form of input data of the corresponding 3D feature extractor block from the 3D feature data output by the block of the corresponding 3D feature extractor.

[0038] While conventional conduction attack models are trained to reconstruct original 3D point cloud data from final feature data, the present invention uses a method of training a corresponding block for each step, namely the downsampling step. Accordingly, the present invention trains N conduction blocks individually, and each conduction block is trained to reconstruct the input data of the corresponding block from the output data of the 3D feature extractor block, using the input data of the block of the 3D feature extractor as the ground truth label. For example, the training for the i-th conduction block is the output feature of the i-th block of the feature extractor from It is performed to restore.

[0039] FIG. 4 conceptually illustrates a voxel occupancy classification process. According to one aspect of the present invention, the conduction block learning step may include a voxel occupancy classification step that generates a binary encoding label (S1021) by binary classifying whether a voxel is a voxel with an actual point for 3D feature data restored in the form of input data of a block of a 3D feature extractor corresponding to each conduction block.

[0040] In this case, the voxel occupancy classification step can use the sigmoid focal function as the loss function for voxel occupancy classification.

[0041] In the prior art, the restoration of 3D point cloud data from feature data was treated solely as a regression problem. The conduction attack model of the prior art directly restores the original data through regression on channel values. In the prior art, x, y, and z coordinates are channelized during the voxelization process, and coordinate values ​​are restored when regression is performed. However, in the voxelization of sparse point clouds, many voxels are zero-padded with channel values ​​(0, 0, 0), resulting in the restoration of many unnecessary points near the origin in the inversion attack results. To solve this problem, the present invention transforms the regression problem into a classification problem to resolve semantic ambiguity by classifying whether zero-padded voxels represent empty voxels or valid points at coordinates (0, 0, 0). Through the resolution of this classification problem, a simple binary encoding label can be generated that clarifies the meaning of zero padding by labeling each voxel as 0 (negative) or 1 (positive). The present invention addresses the occupancy classification step. Not continuous coordinate values ​​of the form It outputs a binary classification score of the form. If a voxel is determined to contain a point, the corresponding coordinates can be easily restored. This is because the range of coordinate values ​​is limited by the spatial location of the voxel, and the size of the voxel is generally sufficiently small.

[0042] In addition, due to the sparsity of the original point cloud data, the binary encoded labels have a higher proportion of 0s compared to 1s. This tends to become even more pronounced, especially as the depth of the layer increases. voxelized original 3D point cloud data, If we define it as reconstructed 3D point cloud data, the number of positive labels is |VoI of While fixed to |, the number of negative labels |VoI of | - |VoI of | increases exponentially as the depth of the layer increases (VoI stands for Voxels of Interest, which are non-empty voxels). To resolve this imbalance, the sigmoid focal (SF) loss, a variation of the conventional cross-entropy loss, is used as the loss function in the occupancy classification step. The mathematical expression for the SF loss is given by <Equation 1>.

[0043]

[0044] Here represents the model's predicted probability for the target class. Argument It is used to adjust the importance assigned to positive and negative classes.

[0045] According to one aspect of the present invention, the conduction block learning step may further include a occupancy spread suppression step (S1022) which performs regression on the channel values ​​of 3D feature data when learning a conduction block other than the conduction block corresponding to the first block of a 3D feature extractor, and suppresses the spread of voxel occupancy based on the binary encoding label generated in the voxel occupancy classification step. At this time, when regression on the channel values ​​is performed in the conduction block for a block other than the first block, a normalized form of feature data value is obtained.

[0046] As mentioned above, the voxelized original 3D point cloud data is sparse due to the large number of empty voxels. However, as mentioned above, a problem arises in which the density of non-empty voxels increases exponentially during the downsampling process of the 3D feature extractor, making it difficult to restore the 3D point cloud data to its original sparse state during the restoration stage.

[0047] The present invention performs reconstruction incrementally by training multiple conduction blocks, each divided based on a downsampling layer, of a 3D feature extractor. That is, as previously described, the present invention divides the feature extractor into multiple blocks based on the downsampling layer where variance of non-empty voxels occurs, generates conduction blocks corresponding to each block, and trains reconstruction on a block-by-block basis. Since the channel values ​​at the voxel level of the original 3D point cloud data represent point coordinates, regression is not required. However, since normalization is applied during the intermediate process, both classification and regression regarding the channel values ​​are necessary.

[0048] For example, the input to the (i+1)th block is and the output is If so, the (i+1)th conduction block of the conduction attack model is Receive as input It generates, and this It is the result value that restores. Specifically (Voxel occupancy classification score found by applying SF loss) and (Channel value found by applying L2 loss) is It is derived from. Then Using By masking Creates ( Binary encoded labels are derived from). Since unnecessary VoI values ​​are removed during the masking process, the variance of VoI can be suppressed. As mentioned above, in the first conduction block, which is the final stage of the conduction attack model, only classification is performed without additional regression. The loss function for each conduction block is as shown in <Equation 2>.

[0049]

[0050] The reconstruction model construction step is a step of constructing a 3D point cloud reconstruction model (S1030) by recombining N learned conduction blocks in reverse order. Since the reconstruction model is upsampled in response to downsampling of the 3D feature extractor, the conduction blocks, each separately learned and corresponding to the blocks of the 3D feature extractor, must be combined in reverse order.

[0051] The restoration step is a step of restoring the 3D point cloud data (S1040) by providing the 3D feature data finally output by the 3D feature extractor as input data to the 3D point cloud restoration model. That is, the restoration step is a step of restoring the original 3D point cloud through the restoration model using the 3D feature data extracted through the 3D feature extractor. Depending on the aspect of the invention, the 3D feature data may also be restored by providing the feature data extracted in the intermediate step, in addition to the final feature data extracted by the 3D feature extractor, to the conduction block of the restoration model corresponding to the feature extraction step.

[0052] FIG. 6 illustrates the concept of restoration using voxel center points according to the present invention. According to one aspect of the present invention, the restoration step restores the output data of the restoration model into a 3D point cloud, and can restore the 3D point cloud by creating points at the center coordinates of voxels classified as having actual points.

[0053] The final output after passing through all conduction blocks is It is a binary classification score of the form, and the restoration model completes the restoration of 3D point cloud data by generating a point at the center of the voxel corresponding to the occupancy of the positive.

[0054] Figure 6 illustrates an example of finding the center point of a voxel.

[0056] Although the present invention has been described above with reference to embodiments with reference to the accompanying drawings, it is not limited thereto and should be interpreted to encompass various modifications that can be obviously derived from them by those skilled in the art. The claims are intended to encompass such modifications.

Claims

Claim 1 A method for reconstructing a 3D point cloud from voxel-based feature data, comprising: a step of determining the number of conduction blocks, wherein layers constituting a 3D feature extractor that voxels a 3D point cloud and extracts 3D feature data based on convolution operations are divided into a plurality of blocks based on a downsampling layer; a step of learning each of the N conduction blocks, wherein each conduction block is learned to be restored in the form of input data for the corresponding 3D feature extractor block from the 3D feature data output by the block of the corresponding 3D feature extractor; a step of constructing a reconstruction model, wherein the N learned conduction blocks are recombined in reverse order to construct a 3D point cloud reconstruction model; and a reconstruction step, wherein the 3D feature data finally output by the 3D feature extractor is provided as input data to the 3D point cloud reconstruction model to reconstruct the 3D point cloud. Claim 2 A method for restoring a 3D point cloud from voxel-based feature data, wherein the conduction block learning step includes a voxel occupancy classification step that generates a binary encoded label by binary classifying whether a voxel is a voxel with an actual point for 3D feature data restored in the form of input data of a block of a 3D feature extractor corresponding to each conduction block. Claim 3 In claim 2, the voxel occupancy classification step is a method for reconstructing a 3D point cloud from voxel-based feature data, using a sigmoid focal function as the loss function for voxel occupancy classification. Claim 4 A method for restoring a 3D point cloud from voxel-based feature data, wherein the conduction block learning step performs regression on the channel values ​​of the 3D feature data when learning a conduction block other than the conduction block corresponding to the first block of the 3D feature extractor, and further includes a possession spread suppression step that suppresses the spread of voxel possession based on a binary encoding label generated in the voxel possession classification step. Claim 5 A method for restoring a 3D point cloud from voxel-based feature data, wherein the restoration step restores the output data of the restoration model into a 3D point cloud, and restores the 3D point cloud by generating points at the center coordinates of voxels classified as having actual points.

Images