Point cloud selection device, point cloud selection method, and program
The point cloud selection device and method enhance object surface reconstruction accuracy by selecting points of interest and optimizing memory usage, overcoming density-related challenges in INR-based deep learning models.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- KDDI CORP
- Filing Date
- 2023-06-06
- Publication Date
- 2026-06-10
AI Technical Summary
Existing techniques for restoring object surfaces using INR-based deep learning models face challenges in achieving high restoration accuracy due to differences in point cloud density during training and inference, leading to memory limitations and loss of local shape information.
A point cloud selection device and method that selects points of interest, extracts features, generates weighted features, and uses probability prediction to select points considering both global and local shapes while minimizing GPU memory usage.
The solution enables high restoration accuracy in object surface reconstruction by maintaining task performance and reducing GPU memory usage, addressing the limitations of existing methods.
Smart Images
Figure 0007872758000005 
Figure 0007872758000006 
Figure 0007872758000007
Abstract
Description
Technical Field
[0006] , , , ,
[0007] ,
[0001] The present invention relates to a point cloud selection device, a point cloud selection method, and a program.
Background Art
[0002] Conventionally, a technique for restoring the surface of an object from a point cloud using a deep learning model based on INR (Implicit Neural Representation) has been known (see, for example, Non-Patent Document 1). In such a technique, usually, the higher the density of the input point cloud, the more accurately the surface of the object can be restored.
[0003] However, if the density of the input point cloud is different during training and inference of the above deep learning model, high restoration accuracy may not be obtained due to the difference in data distribution.
[0004] Therefore, when inputting a high-density point cloud during inference of the above deep learning model, it is desirable to input a high-density point cloud during training of the above deep learning model as well.
[0005] However, as the density of the input point cloud increases, the amount of data increases, so there is a risk of exceeding the memory limit of the GPU used during training of the above deep learning model.
[0006] During training of a deep learning model, a method of processing a plurality of data together called the mini-batch learning method is generally used to stabilize the learning.
[0007] Also, during training of a deep learning model, in order to update the deep learning model based on the error backpropagation method, it is necessary to calculate the gradient of the model parameters with respect to the objective function. Therefore, during training of a deep learning model, the memory capacity available per point cloud may be further limited compared to during inference of the deep learning model.
Prior Art Documents
Non-Patent Documents
[0008] [Non-Patent Document 1] A. Boulch and M. Renaud, "Poco: Point convolution for surface reconstruction," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022 [Non-Patent Document 2] I. Lang, A. Manor, and S. Avidan, "Samplenet: Differentiable point cloud sampling," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020. [Non-Patent Document 3] RA Potamias, G. Bouritsas, and S. Zafeiriou, "Revisiting point cloud simplification: A learnable feature preserving approach," Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II. Cham: Springer Nature Switzerland, 2022. [Overview of the Initiative] [Problems that the invention aims to solve]
[0009] As a preprocessing step to reduce the amount of data in a point cloud, random sampling, which involves randomly selecting a specified number of points from the input point cloud, is widely used.
[0010] Similarly, the First Point Sampling (FPS) method, which selects a specified number of points spatially uniformly, is also widely used.
[0011] However, the point clouds selected using these methods lose local shape information. Therefore, there was a problem in that training an INR-based deep learning model using such point clouds may not yield high accuracy in reconstructing the surface of an object.
[0012] Non-patent document 2 discloses a technique for sampling point clouds while maintaining performance for any task.
[0013] However, the technology disclosed in Non-Patent Document 2 has the problem that it requires a pre-trained model for the task, and therefore cannot use sampled point clouds to train an INR-based deep learning model.
[0014] Non-patent document 3 discloses a technique for generating a point cloud consisting of fewer points from an input point cloud. By training an INR-based deep learning model using the point cloud obtained by this technique, it is possible to reduce the memory usage during training of the deep learning model.
[0015] However, the technology disclosed in Non-Patent Document 3 has the problem that it may not be possible to obtain high restoration accuracy in restoring the surface of an object because it does not take into consideration maintaining performance for the task.
[0016] Therefore, the present invention has been made in view of the above-mentioned problems, and aims to provide a point cloud selection device, a point cloud selection method, and a program that can generate a model capable of achieving high restoration accuracy in the restoration of an object's surface by selecting a point cloud that takes into consideration maintaining performance for the task while representing both the global and local shapes of the point cloud while suppressing the memory usage of the GPU. [Means for solving the problem]
[0017] The first feature of the present invention is a point cloud selection device comprising: a point of interest selection unit for selecting points of interest from a point cloud; a feature extraction unit for extracting features from the point cloud; a feature generation unit for generating features related to the points of interest from the features; and a point cloud selection unit for selecting some points from the point cloud based on the features related to the points of interest.
[0018] A second feature of the present invention is a point cloud selection device comprising a point of interest selection unit that selects points of interest from a point cloud, and a surface restoration unit that restores the surface of an object using a trained model, wherein the surface restoration unit selects prediction results for query points located near the points of interest when restoring the surface of the object.
[0019] The third feature of the present invention is a point cloud selection method comprising the steps of: selecting points of interest from a point cloud; extracting features from the point cloud; generating features related to the points of interest from the features; and selecting some points from the point cloud based on the features related to the points of interest.
[0020] A fourth feature of the present invention is a program that causes a computer to function as a point cloud selection device, wherein the point cloud selection device comprises: a point of interest selection unit that selects points of interest from a point cloud; a feature extraction unit that extracts features from the point cloud; a feature generation unit that generates features related to the points of interest from the features; and a point cloud selection unit that selects some points from the point cloud based on the features related to the points of interest. [Effects of the Invention]
[0021] According to the present invention, a point cloud selection device, a point cloud selection method, and a program are provided that can generate a model capable of achieving high restoration accuracy in the restoration of an object's surface by selecting a point cloud that takes into account maintaining performance for the task while suppressing GPU memory usage and representing both the global and local shapes of the point cloud. [Brief explanation of the drawing]
[0022] [Figure 1] FIG. 1 is a diagram showing an example of a functional block of a point group selection device 100 according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a functional block of a point group selection device 100 according to the second embodiment. [Figure 3] FIG. 3 shows a schematic diagram of a point group I and divided point groups {S_0, S_1, S_2, S_3} when d = 4. [Figure 4] FIG. 4 is a diagram showing an example of a functional block of a selected point group construction unit 21 of a point group selection device 100 according to the second embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a functional block of a point group selection device 100 according to the third embodiment.
Embodiments for Carrying Out the Invention
[0023] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.
[0024] (First Embodiment) Hereinafter, referring to FIG. 1, a point group selection device 100 according to the first embodiment of the present invention will be described. FIG. 1 is a diagram showing an example of a functional block of the point group selection device 100 according to the present embodiment.
[0025] As shown in FIG. 1, the point group selection device 100 according to the present embodiment includes an initial point group selection unit 11, an interest point selection unit 12, a distance weight calculation unit 13, a feature extraction unit 14, a weighted feature calculation unit 15, a feature generation unit 16, a probability prediction unit 17, a point group selection unit 18, and a model training unit 19.
[0026] Hereafter, the model for point cloud selection will be denoted as M_s, and the model for reconstructing the object surface based on INR will be denoted as M_r.
[0027] [Initial point cloud selection unit 11] The initial point cloud selection unit 11 is configured to select a portion of the input point cloud (hereinafter referred to as point cloud I).
[0028] Any non-learning-based method, such as random sampling or FPS, may be used as the method for selecting such points. Alternatively, any learning-based method, such as those described in Non-Patent Document 2 or Non-Patent Document 3, may be used as the method for selecting such points.
[0029] The initial point cloud selection unit 11 does not necessarily have to perform the above selection during the training of model M_s. It may also generate a point cloud obtained by selecting some points from point cloud I in advance as a preprocessing step before training model M_s, and then simply read those point clouds during the training of model M_s.
[0030] [Selection of points of interest section 12] The point of interest selection unit 12 is configured to select some points (hereinafter referred to as points of interest) from the set of points selected by the initial point cloud selection unit 11 (hereinafter referred to as point cloud X).
[0031] Here, when the point of interest selection unit 12 selects n points of interest, it may randomly select n points of interest from the point cloud X.
[0032] Alternatively, the point of interest selection unit 12 may pre-select m (≧n) points from the point cloud I using the FPS method, randomly select n points from the m points, and consider the nearest neighbor points in the point cloud X to those points as the selected points of interest.
[0033] Alternatively, the user may manually select m points in the point cloud I that are located in a region of interest, and the point of interest selection unit 12 may randomly select n points from these m points and consider the nearest neighbor points in the point cloud X to those points as the selected points of interest.
[0034] Alternatively, the point of interest selection unit 12 may first detect regions representing objects within the point cloud I using any object detection method or instance segmentation method, select m points located in the nearest neighbors within the point cloud I to the centroid of those regions, randomly select n points from the m points, and consider the nearest neighbors within the point cloud X to those points as the selected points of interest.
[0035] Furthermore, the point of interest selection unit 12 may select the above-mentioned points of interest by other means.
[0036] Furthermore, in the process of selecting n points from the m points described above, if the point of interest selection unit 12 performs iterative processing when training the model M_s, it may assign numbers to the points so that all m points are selected through such iterative processing, and then select the n points sequentially in each iterative process according to these numbers.
[0037] Here, the point of interest selection unit 12 does not necessarily have to perform the above processing during the training of model M_s. Instead, it may generate a point cloud obtained by selecting some points from the point cloud X in advance as a preprocessing step before training model M_s, and then simply read this point cloud during the training of model M_s. [Distance weight calculation unit 13] The distance weight calculation unit 13 is configured to calculate a point-specific weight according to the distance from the point of interest selected by the point of interest selection unit 12 to each point in the point cloud X.
[0038] Firstly, the distance weighting unit 13 calculates the distance from the point of interest to each point in the point cloud X. Here, the distance weighting unit 13 may use any distance measure, such as the Euclidean distance or the Mahalanobis distance.
[0039] Secondly, the distance weight calculation unit 13 uses the obtained distances to calculate weights corresponding to the distance from the point of interest. Here, the distance weight calculation unit 13 may use any function that outputs the reciprocal of the input or a Gaussian function, such as a Gaussian function, where the output increases as the input decreases.
[0040] Furthermore, if multiple points of interest are given, the distance weight calculation unit 13 may calculate the weight independently for each point of interest, or it may calculate a weight by combining the weights for multiple points of interest using a function that can represent a mixture of multiple probability distributions, such as a Gaussian mixture.
[0041] The distance weight calculation unit 13 does not necessarily have to perform the above processing during the training of the model M_s. Instead, as a preprocessing step before training the model M_s, it may calculate the weight for each point according to the distance from the point of interest to each point in the point cloud X, and then simply read that weight during the training of the model M_s.
[0042] From now on, the data obtained by arranging the weights of each point in a single line will be called a weight vector.
[0043] [Feature extraction unit 14] The feature extraction unit 14 is configured to extract feature vectors for each point from the point cloud X.
[0044] Here, the feature extraction unit 14 may use any DNN (Deep Neural Network) capable of extracting feature vectors for each point from the point cloud X. As an example, the feature extraction unit 14 may use the Projection Network described in Non-Patent Literature 3 as such a DNN. When the feature extraction unit 14 uses such a DNN, it can extract feature vectors for each point from the input point cloud using an MLP (Multi-Layer Perceptron), batch normalization, ReLU activation, and Graph Neural Networks.
[0045] From now on, data obtained by arranging the feature vectors for each point in a single line will be called a feature tensor.
[0046] [Weighted Feature Calculation Unit 15] The weighted feature calculation unit 15 is configured to calculate a weighted feature tensor (weighted feature) from the feature tensor extracted by the feature extraction unit 14 (hereinafter referred to as feature F_0) and the weight vector calculated by the distance weight calculation unit 13.
[0047] When there is only one point of interest, or when the weights for all points of interest are mixed together, only a single weight vector W_0 is given. In this case, the weighted feature calculation unit 15 can calculate the weighted feature tensor as W_0*F_0. Here, * represents element-wise multiplication, indicating that the scalar representing the weight for each point is multiplied by the feature vector.
[0048] When calculating weights independently for multiple points of interest, a set of weight vectors {W_0, W_1, ..., W_l} is given. In this case, the weighted feature calculation unit 15 can calculate the set of weighted feature tensors in a similar manner as {W_0*F_0, W_1*F_1, ..., W_I*F_l}.
[0049] [Feature generation unit 16] The feature generation unit 16 is configured to generate a feature tensor (hereinafter referred to as feature C) by combining the feature tensor extracted by the feature extraction unit 14 and the weighted feature tensor calculated by the weighted feature calculation unit 15. Feature C is a feature related to the point of interest. Here, the feature generation unit 16 generates the feature tensor by combining the feature vector for each point and the weighted feature vector.
[0050] Even when multiple weighted feature tensors are provided, the feature generation unit 16 combines all weighted feature tensors in the same manner.
[0051] Furthermore, the feature generation unit 16 may combine the feature tensor and the weighted feature tensor with the 3D coordinates of each point representing the coordinates of the point cloud X.
[0052] [Probability Prediction Unit 17] The probability prediction unit 17 is configured to use an arbitrary DNN to predict the selection probability (hereafter referred to as probability P) for each point, taking the feature C generated by the feature generation unit 16 as input. An example of a DNN is an MLP.
[0053] Furthermore, in order to limit the range of predicted values, the probability prediction unit 17 may consider the output of an arbitrary function such as a sigmoid function or a ReLU function, when the output of the DNN is input to it, as the predicted value.
[0054] [Point cloud selection unit 18] The point cloud selection unit 18 is configured to select some points from the point cloud X using the probability P predicted by the probability prediction unit 17.
[0055] For example, the point cloud selection unit 18 may select a specified number of points from the point cloud X in descending order of the selection probability for each point. Here, the specified number may be directly given by the user, or it may be a number obtained by multiplying the number of points in the point cloud X by a sampling rate specified by the user and converting the result to an integer.
[0056] Hereafter, the set of selected points will be referred to as the point cloud Y.
[0057] [Model Training Section 19] The model training unit 19 is configured to calculate the loss related to point cloud selection and the loss related to the reconstruction of the object surface, and to update the parameters of model M_s and model M_r using these losses.
[0058] As shown in Figure 1, the model training unit 19 may, when training only a model for point cloud selection, calculate only the loss related to point cloud selection and update only the parameters of the model M_s.
[0059] Here, the model training unit 19 may use the mean squared error (MSE) between the point of interest and each point in the point cloud Y as the loss function for selecting the point cloud.
[0060] If there are multiple points of interest, the model training unit 19 may calculate the MSE for each point of interest in relation to each point in the point cloud Y, and use the average of these MSEs as the loss function.
[0061] Here, the model training unit 19 may use a variation of the MSE that multiplies the predicted probability expressed by the following equation (1) in order to make the MSE a function of probability P.
[0062]
number
[0063] Here, the model training unit 19 may use a variation of the Chamfer distance in which each term expressed in the following equation (2) is multiplied by the predicted probability in order to make the Chamfer distance a function of probability P.
[0064]
number
[0065] Here, the model training unit 19 may use a variation of the repulsion loss that multiplies the repulsion loss by the predicted probability expressed in the following equation (3) in order to make the repulsion loss a function of probability P.
[0066]
number
number
[0067] The model training unit 19 may use the weighted sum of the above-mentioned MSE (and its variations), Chamfer distance (and its variations), and repulsion loss (and its variations) as the loss function for point cloud selection.
[0068] Here, the model training unit 19 is not required to use all of the above-mentioned loss functions as the loss function for point cloud selection. It may select any loss function and use the weighted sum of the selected loss functions as the loss function for point cloud selection.
[0069] Hereafter, the loss function related to point cloud selection will be denoted as L_s.
[0070] (Second Embodiment) Hereinafter, with reference to Figures 2 to 4, the point cloud selection device 100 according to the second embodiment of the present invention will be described, focusing on the differences from the point cloud selection device 100 according to the first embodiment described above. Figure 2 is a diagram showing an example of the functional block of the point cloud selection device 100 according to this embodiment.
[0071] As shown in Figure 2, the point cloud selection device 100 according to this embodiment has, in addition to the functions of the point cloud selection device 100 according to the first embodiment described above, a point cloud division unit 20, a selected point cloud construction unit 21, and a surface restoration unit 22.
[0072] [Point group division section 20] The point cloud division unit 20 is configured to generate a set of point clouds obtained by dividing the point cloud I into a specified number of points (hereinafter referred to as number d). One example of a method for dividing the point cloud I is to perform iterative random sampling while excluding points that have been selected in the past.
[0073] Here, the point cloud division unit 20 uses the integer value obtained by dividing the number of points in point cloud I by the number of points d as the number of points to sample, s.
[0074] Firstly, the point cloud division unit 20 selects s points from the point cloud I using a random sampling method. The resulting point cloud is denoted as S_0.
[0075] Secondly, the point cloud division unit 20 selects s points from the point cloud obtained by excluding points in S_0 from point cloud I using a random sampling method. The resulting point cloud is called S_1.
[0076] Similarly, the point cloud division unit 20 can obtain d point clouds {S_0, S_1, …, S_{d-1}} by iteratively selecting point clouds from the point cloud obtained by removing the already selected point clouds from point cloud I using a random sampling method.
[0077] Figure 3 shows schematic diagrams of the point group I and the divided point groups {S_0, S_1, S_2, S_3} when d=4.
[0078] Here, the point cloud division unit 20 may use other sampling methods such as the FPS method instead of the random sampling method. Furthermore, the point cloud division unit 20 may divide the point cloud I by other methods.
[0079] Furthermore, the point cloud division unit 20 does not necessarily have to perform the above processing during the training of model M_s. As a preprocessing step before training model M_s, it may generate a set of point clouds obtained by dividing point cloud I in advance, and then simply read these sets during the training of model M_s.
[0080] [Selected Point Cloud Construction Unit 21] The selected point cloud construction unit 21 is configured to select some points from the point cloud I using the points of interest and model M_s obtained by the point of interest selection unit 12. Figure 4 shows an example of the functional blocks of the selected point cloud construction unit 21.
[0081] As shown in Figure 4, the selected point cloud construction unit 21 includes a distance weight calculation unit 21A, a feature extraction unit 21B, a weighted feature calculation unit 21C, a feature generation unit 21D, a probability prediction unit 21E, and a point cloud selection unit 21F.
[0082] Here, the functions of the distance weight calculation unit 21A, the feature extraction unit 21B, the weighted feature calculation unit 21C, the feature generation unit 21D, the probability prediction unit 21E, and the point cloud selection unit 21F are the same as the functions of the distance weight calculation unit 13, the feature extraction unit 14, the weighted feature calculation unit 15, the feature generation unit 16, the probability prediction unit 17, and the point cloud selection unit 18 described above.
[0083] The selected point cloud construction unit 21 can select some points related to the points of interest (hereinafter referred to as point cloud Z) from point cloud I by processing the point cloud I and the points of interest as inputs using the functions described above.
[0084] However, in such cases, the large number of points in the point cloud I used as input to model M_s may lead to high memory usage.
[0085] Therefore, in order to reduce memory usage, instead of point cloud I, each of the d point clouds {S_0, S_1, …, S_{d-1}} obtained by dividing point cloud I in the point cloud division unit 20 may be used as input to model M_s.
[0086] In such a case, the selected point cloud construction unit 21 can take one point cloud (for example, S_0) and the point of interest from the d point clouds as input and perform the above processing to select some points related to the point of interest from the input point cloud.
[0087] The selected point cloud construction unit 21 can select some points from each point cloud by performing the same process on the remaining d point clouds.
[0088] Then, the selected point cloud construction unit 21 obtains a single point cloud Z by taking the union of some of the points obtained from each of the d point clouds {S_0, S_1, …, S_{d-1}}.
[0089] In this case, the number of points in the point cloud used as input to model M_s is smaller than the number of points in point cloud I, resulting in lower memory usage compared to when point cloud I is used as input.
[0090] [Surface restoration section 22] The surface restoration unit 22 is configured to restore the surface of an object from the point cloud Z and the query point cloud using the model M_r.
[0091] Here, the surface restoration unit 22 may use any surface restoration method and the corresponding model M_r. As an example, the surface restoration unit 22 can use the POCO method described in Non-Patent Literature 1.
[0092] The query point cloud is a point cloud used to train the model M_r for reconstructing the surface of an object based on INR, and is created, for example, by randomly sampling points from within the bounding box of the object to be reconstructed.
[0093] The query point cloud may be created using a method corresponding to the technique used, or it may be created by other methods.
[0094] [Model Training Section 19] In this embodiment, the model training unit 19 is configured to train not only the model M_s for point cloud selection, but also the model M_r for reconstructing the surface of an object based on INR.
[0095] Here, the model training unit 19 can train the model M_r by predicting the positional relationship between the query point cloud and the surface of the object.
[0096] For example, when using the POCO described above, the model training unit 19 can train the model M_r by predicting the occupancy rate, which represents the probability that the query point cloud is inside the target object, and calculating a loss between that occupancy rate and the true value.
[0097] The model training unit 19 uses a loss defined by the method used in the surface restoration unit 22 as the loss related to the restoration of the object's surface. An example of such a loss is the cross-entropy loss defined by the POCO method described in Non-Patent Literature 1.
[0098] Hereafter, the loss function related to the restoration of the object's surface will be denoted as L_r.
[0099] When updating the model parameters, the model training unit 19 may simultaneously update the parameters of models M_s and M_r (hereinafter referred to as simultaneous optimization).
[0100] Alternatively, the model training unit 19 may perform alternating updates (hereinafter referred to as alternating optimization) in which it updates one parameter while keeping the other parameter fixed, and then updates the other parameter while keeping the other parameter fixed.
[0101] In this case, the model training unit 19 may save the fixed model parameters, delete them from the GPU, and then reload them onto the GPU when updating, in order to reduce the memory usage of the GPU.
[0102] When performing simultaneous optimization, the model training unit 19 may use a weighted sum of L_s and L_r as the loss function.
[0103] On the other hand, when the model training unit 19 performs alternating optimization, it may use L_s as the loss function when updating the parameters of model M_s, and L_r as the loss function when updating the parameters of model M_r. It may also use a weighted sum of L_s and L_r as the loss function when updating the parameters of either or both of models M_s and M_r.
[0104] Furthermore, the model training unit 19 does not need to update the parameters of the model held by the point cloud selection device 100 if the point cloud selection device 100 holds a trained model M_s or M_r.
[0105] For example, if the point cloud selection device 100 holds a trained model M_s, the model training unit 19 may update only the parameters of model M_r while keeping the parameters of model M_s fixed at all times.
[0106] Furthermore, if the point cloud selection device 100 holds a trained model M_r, the model training unit 19 may update only the parameters of model M_s while keeping the parameters of model M_r fixed.
[0107] (Third embodiment) Hereinafter, with reference to Figure 5, the point cloud selection device 100 according to the third embodiment of the present invention will be described, focusing on the differences from the point cloud selection device 100 according to the first and second embodiments described above. Figure 5 is a diagram showing an example of the functional block of the point cloud selection device 100 according to this embodiment.
[0108] In this embodiment, the point cloud selection device 100 is used for inference using the trained model M_r.
[0109] Figure 5 shows an example of the functional blocks of the point cloud selection device 100 when performing inference using model M_r.
[0110] In this embodiment, the selected point cloud construction unit 21 is configured to construct a selected point cloud (point cloud Z) using the trained model M_s and the points of interest selected by the point of interest selection unit 12.
[0111] The surface restoration unit 22 is configured to restore the surface of an object using the point cloud Z constructed by the selected point cloud construction unit 21 and the trained model M_r.
[0112] In this context, the query point cloud used during inference is often a set of voxel center coordinates obtained by dividing the bounding box of the object to be reconstructed.
[0113] In this embodiment, the surface restoration unit 22 may select only the prediction results for the query point cloud that are located close to the point of interest.
[0114] The surface reconstruction unit 22 may obtain prediction results for all query point clouds by iteratively performing inference using various points of interest.
[0115] If the surface reconstruction unit 22 obtains multiple prediction results for the same query point cloud, it may consider statistical values such as the mean or median of those prediction results as the prediction result.
[0116] Furthermore, the processing in the point cloud division unit 20, the point of interest selection unit 12, and the selected point cloud construction unit 21 does not necessarily have to be performed during inference by the surface reconstruction unit 22. These processes can be performed in advance as pre-processing for inference, and during inference, the surface reconstruction unit 22 can simply read the results of these processes.
[0117] Furthermore, the point cloud selection device 100 described above may be implemented as a program that causes a computer to execute each function (each process).
[0118] In the above embodiments, the present invention was described using the application to a point cloud selection device 100 as an example. However, the present invention is not limited to such examples and can be similarly applied to a point cloud selection system equipped with the functions of the point cloud selection device 100. [Industrial applicability]
[0119] Furthermore, according to this embodiment, for example, it is possible to achieve an overall improvement in service quality in video communication, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]
[0120] 100... Point cloud selection device 11...Initial point cloud selection section 12…Selection of points of interest 13, 21A...Distance weight calculation section 14, 21B... Feature extraction section 15, 21C...Weighted Feature Calculation Unit 16, 21D... Feature generation unit 17, 21E... Probability prediction section 18, 21F... Point cloud selection section 19…Model Training Department 20...Point group division part 21…Selection Point Cloud Construction Section 22...Surface restoration section
Claims
1. A point cloud selection device, A point of interest selection unit that selects points of interest from the point cloud, A feature extraction unit that extracts features from the aforementioned point cloud, A feature generation unit that generates features related to the points of interest from the aforementioned features, A point cloud selection device characterized by having a point cloud selection unit that selects some points from the point cloud based on the characteristics of the points of interest.
2. A distance weighting calculation unit that calculates weights corresponding to the distance from the point of interest to each point in the point cloud, It comprises a weighted feature calculation unit that calculates weighted features from the weights and features, The point cloud selection device according to claim 1, characterized in that the feature generation unit generates features relating to the points of interest by combining the features and the weighted features.
3. The system further includes a probability prediction unit that predicts the selection probability based on the characteristics of the aforementioned points of interest, The point cloud selection device according to claim 1, characterized in that the point cloud selection unit selects a portion of the points from the point cloud using the selection probability.
4. A point cloud division unit that generates a point cloud divided from the aforementioned point cloud, A selected point cloud construction unit constructs a selected point cloud by taking the union of point clouds individually selected from the divided point clouds, The point cloud selection device according to claim 1, further comprising a surface restoration unit that restores the surface of an object using the selected point cloud.
5. The point cloud selection device according to claim 4, characterized in that the point cloud division unit generates the divided point cloud by iteratively selecting points from the point cloud.
6. A probability prediction unit that predicts the selection probability from the characteristics related to the aforementioned point of interest, The point cloud selection device according to claim 4, further comprising: a model training unit that calculates a loss using the selection probability predicted by the probability prediction unit, the partial points selected by the point cloud selection unit, the points of interest selected by the points of interest selection unit, and at least a portion of the information restored by the surface restoration unit, and trains a model.
7. The point cloud selection device according to claim 6, characterized in that the model training unit uses a model for selecting point clouds and a model for restoring the surface of an object as the model.
8. The point cloud selection device according to claim 7, characterized in that the model training unit simultaneously or alternately optimizes the model for selecting the point cloud and the model for restoring the surface of the object.
9. The point cloud selection device according to claim 7, characterized in that the model training unit optimizes both or only the model for point cloud selection and the model for restoring the surface of the object.
10. It further includes an initial point cloud selection unit, The point cloud selection device according to claim 1, characterized in that the point cloud input to the point of interest selection unit and the feature extraction unit is a point cloud selected by an arbitrary selection method using an initial point.
11. The model training unit calculates a loss using some of the points selected by the point cloud selection unit and at least some of the points of interest selected by the point of interest selection unit, and trains the model. The point cloud selection device according to claim 1, characterized in that the point of interest selection unit selects the points of interest before the model training unit trains the model, and reads the selection results of the points of interest during the model training.
12. The model training unit calculates a loss using some of the points selected by the point cloud selection unit and at least some of the points of interest selected by the point of interest selection unit, and trains the model. The point cloud selection device according to claim 2, characterized in that the distance weight calculation unit calculates the weights before the model training unit trains the model, and reads the weight calculation results when training the model.
13. The model training unit calculates a loss using at least a portion of the points selected by the point cloud selection unit, the points of interest selected by the point of interest selection unit, and the information restored by the surface restoration unit, and trains the model. The point cloud selection device according to claim 4, characterized in that the point cloud division unit generates the divided point cloud before the model training unit trains the model, and reads the generated result of the divided point cloud during the model training.
14. A point cloud selection method, The process of selecting points of interest from the point cloud, A step of extracting features from the aforementioned point cloud, A step of generating features related to the points of interest from the aforementioned features, A point cloud selection method characterized by comprising the step of selecting some points from the point cloud based on the characteristics of the points of interest.
15. A program that makes a computer function as a point cloud selection device, The point cloud selection device is A point of interest selection unit that selects points of interest from the point cloud, A feature extraction unit that extracts features from the aforementioned point cloud, A feature generation unit that generates features related to the points of interest from the aforementioned features, A program characterized by having a point cloud selection unit that selects some points from the point cloud based on the characteristics of the point of interest.