Point cloud identification device, learning device, point cloud identification method, and learning method

Abstract

A point-cloud identification device (300) is provided with: a point-cloud acquisition unit (320) for acquiring point-cloud information indicating N (N ≥ 2) points in k (k ≥ 2) dimensions; a model acquisition unit (310) for acquiring a model having training parameters; a rotation-invariant transformer (330) that orthogonalizes each of per-point basis vectors indicated in the point-cloud information and utilizes the orthogonalized data to compute rotation-invariant features; an inference unit (340) that utilizes the rotation-invariant features and the model to identify a point cloud indicated in the point-cloud information; and a result output unit (350) that outputs classification results from the identification by the inference unit.

Description

[Technical field]

[0001] The present disclosure relates to a point cloud identification technique for identifying a point cloud indicated in point cloud information. [Background technology]

[0002] For example, detection results by sensors such as LiDAR or radar may be represented in the form of point cloud information including a large number of points. There is a technique for extracting the features of a point cloud represented in such point cloud information. Patent Document 1 discloses a feature representation device that represents the features of a point cloud. Specifically, the feature representation device of Patent Document 1 is a feature representation device that represents the features of three-dimensional point cloud data (point cloud information), and represents a set of points by the coordinates s of spatial sample points s set around the points. x , s y , s z A distance field transformation unit transforms the spatial sample point s into a distance field indicating the nearest distance φ(s) from the spatial sample point s to the nearest point, and the coordinates s of the spatial sample point s x , s y , s z A canonical projection part obtains a transformation to the standard coordinate system by singular value decomposition of the matrix M consisting of the nearest neighbor distance φ(s), and the coordinates L of the spatial sample point s transformed into the standard coordinate system. in The present invention is equipped with a parameterization unit that trains an extreme learning machine that takes as input the vectors of the nearest neighbor distances φ(s) and outputs the weights β as feature vectors of the 3D point cloud data. The feature representation device of Patent Document 1 is configured to align, for example, a point cloud and a rotated point cloud thereof using the nearest neighbor distance to spatial sample points set in the vicinity of the point cloud. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-133545 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the feature representation device of Patent Document 1 has an issue in that, depending on the point cloud, an error may occur when aligning the point cloud with the spatial sample point using the nearest neighbor distance, and point clouds of the same shape may be identified as point clouds of different shapes, resulting in a tendency for the accuracy of point cloud identification to be low.

[0005] The present disclosure is intended to solve the above-mentioned problems, and has an object to improve the accuracy of identifying point clouds. [Means for solving the problem]

[0006] The point cloud identification device of the present disclosure is A point cloud acquisition unit that acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition unit for acquiring a model having learning parameters; a rotation invariant transformation unit that orthogonalizes basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the orthogonalized data; an inference unit that uses the rotation invariant features and the model to identify points represented in the point cloud information; a result output unit that outputs a classification result based on the identification by the inference unit; Equipped with: Effect of the Invention

[0007] According to the present disclosure, an effect is achieved in that it is possible to improve the accuracy of identifying point clouds. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a point cloud identification system including a point cloud identification device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a rotation invariant transformation unit in the point cloud identification device. [Diagram 3] FIG. 3 is a flowchart illustrating an example of processing performed by the point cloud identification device. [Figure 4]FIG. 4 is a flowchart showing a specific example of the rotation invariant transformation process in the processing of the point cloud identification device. [Diagram 5] FIG. 5 is a diagram illustrating the rotation of a point cloud. [Figure 6] Fig. 6A is a diagram for explaining the relationship between the rotation of point clouds showing the same shape and subspaces, and Fig. 6B is a diagram for explaining the relationship between point clouds showing different shapes and subspaces. [Figure 7] FIG. 7 is a diagram showing an example of the configuration of a point cloud learning system 2 including a point cloud learning device according to the second embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of processing by the point cloud learning device. [Figure 9] FIG. 9 is a diagram illustrating a first example of a hardware configuration for realizing the functions of the point cloud identification device or the point cloud learning device in the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating a second example of a hardware configuration for realizing the functions of the point cloud identification device or the point cloud learning device in the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0010] Embodiment 1 In the first embodiment, a configuration of a point cloud identification device will be described.

[0011] FIG. 1 is a diagram showing an example of the configuration of a point cloud identification system 1 including a point cloud identification apparatus 300 according to the first embodiment. FIG. 2 is a diagram showing an example of the configuration of the rotation invariant transformation unit 330 in the point cloud identification device 300. As shown in FIG. FIG. 3 is a flowchart showing an example of processing performed by the point cloud identification device 300. FIG. 4 is a flowchart showing a specific example of the rotation invariant transformation process in the processing of the point cloud identification device 300. FIG. 5 is a diagram for explaining the rotation of the point groups 1100 and 1200. Fig. 6A is a diagram for explaining the relationship between the rotation of point clouds showing the same shape and subspaces 2100, 2200. Fig. 6B is a diagram for explaining the relationship between point clouds showing different shapes and subspaces 3100, 3200.

[0012] An example of the configuration of a point cloud identification system 1 including a point cloud identification device 300 will be described. The point cloud identification system 1 is a system having a point cloud identification device 300 that identifies a point cloud. The point cloud identification system 1 shown in FIG. 1 includes a point cloud input device 100, a storage device 200, a point cloud identification device, and a result output device 900.

[0013] The point cloud input device 100 receives point cloud information (point cloud data) from, for example, a sensor (not shown), and outputs the point cloud information to the point cloud identification device 300. The sensor (not shown) is, for example, a LiDAR (Light Detection and Ranging) sensor or a radar. The point cloud information indicates a plurality of points (N≧2) that represent position coordinates or features detected by a sensor (not shown). The point cloud information represents each point included in the point cloud in the form of coordinate values ​​in k dimensions (k≧2), for example.

[0014] The storage device 200 includes a storage unit 210 . The storage unit 210 has information used in the classification process for classifying a group of points. Specifically, the storage unit 210 has, for example, learning parameters as a model for classification.

[0015] Point cloud classification device 300 classifies a point cloud using a trained model in k-dimensional point cloud classification. The trained model includes a model that is trained appropriately as in the second embodiment, for example. The k-dimensional point cloud classification refers to classifying point cloud data representing a point cloud expressed in k dimensions based on features such as the shape of the point cloud. An example of the configuration of the point cloud identification device 300 will be described. The point cloud identification device 300 includes a model acquisition unit 310, a point cloud acquisition unit 320, a rotation invariant transformation unit 330, an inference unit 340, and a result output unit 350.

[0016] The model acquisition unit 310 acquires a model having learning parameters. The learning parameters are parameters that have been learned by the learning device and are used when identifying a point cloud.

[0017] The point cloud acquisition unit 320 acquires k-dimensional point cloud information of N points. Specifically, the point cloud acquisition unit 320 acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions.

[0018] The rotation invariant transformation unit 330 orthogonalizes the basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the data after the orthogonalization. Rotation invariant features are features inherent to the shape of a point cloud that are not affected even when the point cloud is rotated.

[0019] The reasoner 340 uses the rotation invariant features and the model to identify points represented in the point cloud information. Specifically, the inference unit 340 calculates a point cloud feature indicating the characteristics of the point cloud using, for example, the rotation invariant feature and the model, and identifies the point cloud using the point cloud feature. The point cloud feature is a feature specific to each category of the point cloud and indicates a feature effective for classifying the point cloud. For example, when identifying by animal type, the point cloud feature is a feature that can identify one category for a horse point cloud and one category for a bird point cloud. The inference unit 340 may be configured to extract the shape of a point cloud and identify the point cloud using a method that uses a filter for extracting object-specific information from the shape of the point cloud. In this case, when identifying a horse from a bird, for example, the inference unit 340 designs a filter to extract features such as the shape of the head, the number of legs, the presence or absence of feathers, the presence or absence of a beak, etc., extracts point cloud features using the filter, and identifies the point cloud using the point cloud features.

[0020] The result output unit 350 outputs the classification result based on the discrimination by the inference unit 340 . Specifically, the result output section 350 outputs the classification result to the result output device 900 .

[0021] An example of the internal configuration of the rotation invariant transformation unit 330 will be described. The rotation invariant transformation unit 330 shown in FIG. 2 includes an orthogonalization layer unit 331 and a projection layer unit 332.

[0022] The orthogonalization layer unit 331 calculates basis vectors for each of k coordinates indicating a point (each of N points) included in the point cloud information, and converts the k basis vectors so that they are orthogonal to each other to generate an (N×k) orthonormal basis matrix.

[0023] The orthogonalization layer 331 performs the orthogonalization process, for example, in the following manner. A point group P represented by point group data (k-dimensional point group data) showing N (N≧2) points N in k (k≧2) dimensions is represented by an N×k matrix, for example, as shown in formula (1). In this case, each point in the point group P is represented by P N1 , , P Nk It is expressed using k coordinate values ​​such as: TIFF0007612122000001.tif20166 The point group P is assumed to have its center of gravity translated to the origin of the coordinate system. The orthogonalization layer unit 331 calculates a basis vector for each of the k coordinates. The orthogonalization layer unit 331 performs orthogonalization orth on the point group P as shown in equation (2) in order to transform the k basis vectors so that they are orthogonal to each other. The orthogonalization layer 331 obtains an orthonormal basis matrix X by orthogonalization orth. Specifically, the orthogonalization orth may use the Gram-Schmitt orthogonalization method X = GramSchmit(P), QR decomposition (X, r = QR(P)), or singular value decomposition (X, σ, V T=SVD(P)) or eigenvalue decomposition (X, Λ=EIG(P)) may be used. orth(P)=X, stX T X=I_ k (2) In equation (2), "Ik" represents a k×k unit matrix.

[0024] The projection layer 332 uses the orthonormal basis matrices and the model to calculate a projection matrix that represents rotation invariant features. The projection layer unit 332 extracts a projection matrix M, which is a k-dimensional rotation invariant feature (rotation invariant feature), from the orthonormal basis matrix obtained by the orthogonalization layer unit 331. The projection matrix M is calculated using the following formula (3). M=XX T (3) Here, when a projection matrix is ​​calculated using a matrix XR obtained by performing k-dimensional rotation (rotation R) on the orthonormal basis matrix X, the following formula (4) holds. XR(XR) T =XX T =M (4) For this reason, the projection matrix M is invariant to the rotation R (rotation invariant). The projection layer unit 332 calculates and outputs a matrix "Y" by multiplying the projection matrix M by a learning parameter W included in the model as shown in the following formula (5). In this disclosure, the learning parameter W shown in formula (5) indicates a first weight that has a geometric meaning, and is a parameter that ultimately becomes a constant through learning. Y = MW (5) Geometrically, the projection matrix M is a projection matrix onto the space spanned by the projection matrix M. "Y" in formula (5) represents the point cloud (shape) obtained by projecting "W" onto the space spanned by "M" by multiplying "W" from the right of the projection matrix M. As a result, "Y" becomes shape information that is geometrically independent of rotation and exhibits rotation-invariant characteristics. By using this, the projection layer section 332 can obtain the same "Y" from the point group P before rotation (orthonormal basis matrix X) and the point group PR after rotation (orthonormal basis matrix XR), as described below in Figures 6A and 6B. On the other hand, the projection layer unit 332 can obtain different "Y" and "Y_hat" (which indicates that the "Y" obtained by equation (5) is different) from the point group P1 (orthonormal basis matrix X1) and the point group P2 (orthonormal basis matrix X2) of different shapes.

[0025] The rotation invariant transformation unit 330 shown in FIG. 2 orthogonalizes the basis vectors for each point indicated in the point cloud information, for example, as described above, and calculates rotation invariant features using the orthogonalized data and model (learning parameter W indicating the first weight).

[0026] If the rotation invariant transform unit 330 is configured as described above, the inference unit 340 identifies the points represented in the point cloud information using a projection matrix and a model that indicate the rotation invariant features.

[0027] An example of the internal configuration of the inference unit 340 will be described. The inference unit 340 extracts point cloud features based on the result of orthogonal projection of the components indicated by the learning parameters into a subspace spanned by an orthonormal basis matrix in an N-dimensional space using a projection matrix and a model indicating the rotation invariant features, and identifies the point cloud using the point cloud features. Specifically, it performs class identification of the point cloud.

[0028] The point cloud features constructed by the inference unit 340 will be described with reference to FIGS. First, the rotation of a point cloud will be described. In FIG. 5, a group of open circles (◯) points 1100 represents the two-dimensional point group X before rotation, and a group of filled circles (●) points 1200 represents the two-dimensional point group XR(π) after rotation. The two-dimensional point group X in Fig. 5 is x 11 is -0.5, x 12 is -1.625, x 21 is 0.5, x 22 is -0.625, x31 is -1.5, x 32 is 3.375, x 41 is 1.5, x 42 This shows what is represented by the matrix X, where x is -1.125. The two-dimensional point group XR(π) in Figure 5 is xr 11 0.5, xr 12 is 1.625, xr 21 -0.5, xr 22 0.625, xr 31 1.5, XR 32 is -3.375, xr 41 -1.5, xr 42 This shows what is represented by the matrix XR(π) where π is 1.125. As shown in FIG. 5, when point cloud data is rotated, the position coordinates change, and the numerical data before and after the rotation are different information. Due to the rotation, the group of circles (white circle points) and the group of black circles (black circle points) are represented by different coordinate values, but the objects shown in the point cloud are the same before and after rotation, even if the coordinates change.Therefore, even with a rotated point cloud, the essential information is not changed, making it possible to identify objects with the same accuracy (objects of the same category).

[0029] In this regard, feature construction by inference unit 340 based on projection matrix M can be geometrically illustrated in FIG. The orthonormal basis matrix X is a matrix in which k basis vectors for each of N points are arranged, as shown in the following formula (6). TIFF0007612122000002.tif19166 This orthonormal basis matrix X spans a k-dimensional subspace spanX in an N-dimensional space. That is, the subspace spanX is the space generated by the orthonormal basis matrix X. In FIG. 6A, a subspace 2100 (spanX) represents the column space of an orthonormal basis matrix X obtained from a point set P. The orthonormal basis matrix X in FIG. 6A is x 11 is -0.5, x 12 is -1.625, x 21 is 0.5, x22 is -0.625, x 31 is -1.5, x 32 is 3.375, x 41 is 1.5, x 42 This shows that the value is -1.125. In FIG. 6A, the subspace 2200 (spanXR) represents the column space of the orthonormal basis matrix XR obtained from the point group PR. The orthonormal basis matrix XR(π) in FIG. 11 0.5, xr 12 is 1.625, xr 21 -0.5, xr 22 0.625, xr 31 1.5, XR 32 is -3.375, xr 41 -1.5, xr 42 This shows that the value is 1.125. The orthonormal basis matrix X and the orthonormal basis matrix XR are orthonormal basis matrices generated from point groups of the same shape. In this case, the subspace 2100 (spanX) and the subspace 2200 (spanXR) are the same. In FIG. 6B, a subspace 3100 (spanX1) represents the column space of an orthonormal basis matrix X1 obtained from a point group P1. The orthonormal basis matrix X1 in FIG. 6B is 11 -0.5, x1 12 is -1.625, x1 21 0.5, x1 22 is -0.625, x1 31 -1.5, x1 32 is 3.375, x1 41 1.5, x1 42 This shows that the value is -1.125. In FIG. 6B, subspace 3200 (spanX2) represents the column space of the orthonormal basis matrix X2 obtained from the point group P2. The orthonormal basis matrix X2 in FIG. 6B is 11 is -4.676, x2 12 is 3.1419615, x2 21 is 3.892, x2 22 is 5.1419615, x2 31 is 1.892, x2 32 is -7.4258845, x241 is -1.108, x2 42 The result shows that the value is -0.8580385. The orthonormal basis matrix X1 and the orthonormal basis matrix X2 are orthonormal basis matrices generated from point groups that indicate mutually different shapes. In this case, the subspace 3100 (spanX1) and the subspace 3200 (spanX2) do not match. Taking advantage of this, the inference unit 340 extracts point cloud features that are effective for identifying point clouds, using an orthonormal basis matrix for each point cloud.

[0030] The inference unit 340 extracts the point cloud feature Z using, for example, the following equation (7). Z = Φ(X) = σ(WY) + B (7) In equation (7), "Y" is obtained by multiplying the projection matrix M by the parameter W indicating the first weight, as shown in equation (5). In equation (7), the learning parameter W indicates a weight (second weight), and the learning parameter B (second learning parameter) indicates a bias. Equation (7) shows that the feature extraction function Φ as a feature extractor has learning parameters (parameter W indicating the first weight, parameter W indicating the second weight, and parameter B indicating the bias) and a nonlinear transformation function σ for nonlinear transformation, and that the point cloud feature Z is extracted by inputting the orthonormal basis matrix X to this feature extraction function Φ. In equation (7), extracting the point cloud feature Z by Φ(X) corresponds to performing orthogonal projection onto the subspace spanX (subspace 2100) spanned by the orthogonal basis matrix X. The orthogonal projection onto the subspace spanX (subspace 2100) and the subspace spanXR (subspace 2200) projects the parameter W onto the same coordinates because these subspaces are the same. This is equivalent to extracting rotation-invariant features specific to the point cloud shape. On the other hand, the orthogonal projection onto the subspace spanX1 (subspace 3100) and the subspace spanX2 (subspace 3200) projects the parameter W onto different coordinates because these subspaces do not coincide. In this way, point cloud information can be identified using features inherent to the point cloud shape that are independent of the coordinates of the point cloud. In the description, the inference unit 340 is configured to receive "Y" calculated by multiplying the projection matrix (projection matrix M) by a learning parameter (parameter W indicating the first weight) included in the model, calculated by the projection layer unit 332, but the same can be achieved by configuring the inference unit 340 to multiply the projection matrix (projection matrix M) by a learning parameter (parameter W indicating the first weight) included in the model. When configured in this way, the projection layer unit 332 outputs the projection matrix M as is.

[0031] The result output device 900 receives and outputs the point cloud classification result output from the point cloud identification device 300. The result output device 900 may be any device that uses the point cloud classification result, and may be, for example, a display device that simply displays the result, or may be a control device related to automatic driving of a vehicle.

[0032] The processing of the point cloud identification device 300 will be described with reference to FIGS. When starting the process, point cloud identification device 300 executes a model acquisition process (step ST110). Specifically, the model acquisition unit 310 in the point cloud identification device 300 acquires a model having learning parameters from the storage unit 210 of the storage device 200. The model acquisition unit 310 outputs the model having learning parameters.

[0033] Point cloud identification device 300 executes a point cloud acquisition process (step ST120). Specifically, the point cloud acquisition unit 320 in the point cloud identification device 300 acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions from the point cloud input device 100. The point cloud acquisition unit 320 outputs the acquired point cloud information.

[0034] Point cloud identification device 300 executes rotation invariant transformation processing (step ST130). Specifically, rotation invariant transformation section 330 in point cloud identification device 300 orthogonalizes the basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the data after the orthogonalization.

[0035] A specific example of the rotation invariant transformation process will be described. When starting the rotation invariant transformation process, the rotation invariant transformation unit 330 executes an orthogonalization process (step ST131) as shown in FIG. Specifically, the orthogonalization layer unit 331 in the rotational invariant transformation unit 330 calculates a basis vector for each of k coordinates indicating a point (each of N points) included in the point cloud information, and transforms the k basis vectors so that they are orthogonal to each other to generate an (N × k) orthonormal basis matrix. The orthogonalization layer 331 converts the basis vectors so that they are orthogonal to each other by using, for example, the Gram-Schmidt orthogonalization method, QR decomposition, singular value decomposition, or eigenvalue decomposition to generate an (N×k) orthogonal basis matrix. The orthogonalization layer 331 outputs the orthogonal basis matrix.

[0036] The rotation invariant transformation unit 330 executes a projection process (step ST132). Specifically, the projection layer unit 332 in the rotation invariant transformation unit 330 acquires an orthonormal basis matrix from the orthogonalization layer unit 331. The projection layer unit 332 calculates a projection matrix indicating a rotation invariant feature using the orthonormal basis matrix. The projection layer unit 332 acquires a model from the model acquisition unit 310, multiplies a learning parameter (learning parameter W indicating a first weight) included in the model by a projection matrix (projection matrix M), and outputs the result.

[0037] Point cloud identification device 300 executes an inference process (step ST140). Specifically, the inference unit 340 in the point cloud identification device 300 first acquires a projection matrix from the projection layer unit 332, and acquires a model (a learning parameter W indicating a second weight, and a learning parameter B indicating a bias) from the model acquisition unit 310. The inference unit 340 then identifies a point cloud indicated in the point cloud information using the rotation invariant feature and the model. More specifically, as described above, the inference unit 340 uses the projection matrix and model indicating the rotation invariant feature to extract point cloud features based on the result of orthogonal projection of components indicated in the learning parameters into a subspace spanned by an orthonormal basis matrix in an N-dimensional space, and identifies the point cloud using the point cloud features. When the projection layer unit 332 in the rotation invariant transformation unit 330 is configured to output the projection matrix without multiplying it by the learning parameter (parameter W), the inference unit 340 multiplies the projection matrix by the learning parameter (parameter W).

[0038] Point cloud identification device 300 executes a result output process (step ST150). Specifically, the result output section 350 in the point cloud identification device 300 outputs the classification result based on the identification by the inference section 340 to the result output device 900.

[0039] Point cloud identification device 300 determines whether to end the series of processes (step ST160). If point cloud identification device 300 determines not to end the process ("NO" in step ST160), it proceeds to the process of step ST110 and repeats the process from step ST110. If it is determined that the process should be ended ("YES" in step ST160), point cloud identification device 300 ends the process.

[0040] An example of the effect of the point cloud identification device of the present disclosure will be described. For example, in autonomous driving technology, point cloud data is used to identify other vehicles and obstacles in the vicinity in order to grasp the surrounding environment of the vehicle. In this case, the point cloud data is processed in real time and the point cloud is identified, so that the autonomous driving device can confirm the presence of obstacles to avoid collision with the vehicle ahead, or can grasp obstacles, rubble, etc. left in front. In this case, the point cloud may be rotated to recognize the state when viewed from various viewpoints, and conventionally, separate processing such as data expansion by rotation and attitude adjustment was required. In contrast, in the present disclosure, as described above, a configuration has been presented in which point cloud data is represented by orthonormal basis vectors, further treated as one subspace, and its projection matrix is ​​treated as data to obtain rotation invariance. In the present disclosure, since the point cloud is represented as rotation-invariant data, learning only a single posture is equivalent to learning for all postures, and it is possible to obtain a discrimination model that is robust to all rotations, which not only makes learning more efficient but also makes it possible to obtain the same discrimination accuracy before and after rotation.

[0041] The point cloud identification device of the present disclosure is configured as follows. A point cloud acquisition unit that acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition unit for acquiring a model having learning parameters; a rotation invariant transformation unit that orthogonalizes basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the orthogonalized data; an inference unit that uses the rotation invariant features and the model to identify points represented in the point cloud information; a result output unit that outputs a classification result based on the identification by the inference unit; A point cloud identification device comprising: As a result, the present disclosure has an effect of providing a point cloud identification device that improves the accuracy of identifying point clouds.

[0042] The point cloud identification method of the present disclosure is configured as follows. a point cloud acquisition step in which a point cloud acquisition unit acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition step in which a model acquisition unit acquires a model having learning parameters; a rotation invariant transformation step in which a rotation invariant transformation unit orthogonalizes each basis vector for each point indicated in the point cloud information and calculates a rotation invariant feature using the orthogonalized information; an inference step of identifying points represented in the point cloud information using the rotation invariant features and the model by an inference unit; a result output step in which a result output unit outputs a classification result based on the identification by the inference unit; The point cloud identification method includes: As a result, the present disclosure has an effect of providing a point cloud identification method that improves the accuracy of identifying point clouds.

[0043] The point cloud identification device of the present disclosure is further configured as follows. The rotation invariant transformation unit an orthogonalization layer that calculates a basis vector for each of k coordinates indicating a point included in the point cloud information, and converts the k basis vectors so that they are orthogonal to each other to generate an orthogonal basis matrix; a projection layer unit that calculates a projection matrix indicating the rotation invariant feature by using the orthonormal basis matrix; having The inference unit is identifying a point cloud represented in the point cloud information using a projection matrix representing the rotation invariant features and the model; A point cloud identification device characterized by: As a result, the present disclosure has an effect of providing a point cloud classification device that can efficiently calculate rotation invariant features of a point cloud. Furthermore, the present disclosure achieves the same effects as those described above by applying the above configuration to the point cloud identification method.

[0044] The point cloud identification device of the present disclosure is further configured as follows. The inference unit is Using the projection matrix representing the rotation invariant features and the model, extracting point cloud features based on a result of orthogonal projection of components indicated by the learning parameters onto a subspace spanned by the orthonormal basis matrix in an N-dimensional space, and identifying the point cloud using the point cloud features; A point cloud identification device characterized by: As a result, the present disclosure has an effect of providing a point cloud identification device that identifies point cloud information using features unique to the point cloud shape that are independent of the coordinates of the point cloud. Furthermore, the present disclosure achieves the same effects as those described above by applying the above configuration to the point cloud identification method.

[0045] Embodiment 2 In the second embodiment, a configuration of a learning device will be described. In the second embodiment, detailed explanations of the same contents as those already explained will be omitted as appropriate.

[0046] FIG. 7 is a diagram showing an example of the configuration of a point cloud learning system 2 including a point cloud learning device 400 according to the second embodiment. The point cloud learning system 2 includes a point cloud input device 100, a storage device 200A, and a point cloud learning device 400. The point cloud input device 100 is similar to the point cloud input device 100 already described, and therefore a detailed description thereof will be omitted here.

[0047] The storage device 200A includes a storage unit 210A. The storage unit 210A has information used in the classification process for classifying point clouds. Specifically, the storage unit 210A has, for example, learning parameters as a classification model. The model is appropriately learned and updated by the point cloud learning device 400.

[0048] The point cloud learning device 400 learns a rotation-invariant model for k-dimensional point cloud classification, and classifies the point cloud using the trained model. The k-dimensional point cloud classification refers to classifying point cloud data representing a point cloud expressed in k dimensions based on features such as the shape of the point cloud.

[0049] An example of the configuration of the point cloud learning device 400 will be described. The point cloud learning device 400 includes a model acquisition unit 410, a point cloud acquisition unit 420, a rotation invariant transformation unit 430, an inference unit 440, a result output unit 450, an evaluation unit 460, and a model update unit 470.

[0050] The model acquisition unit 410 acquires a model having learning parameters. The learning parameters are parameters that have been learned by the point cloud learning device 400, and are used when identifying point clouds.

[0051] The point cloud acquisition unit 420 acquires k-dimensional point cloud information of N points. Similar to the point cloud acquisition section 320 already described, the point cloud acquisition section 420 acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions.

[0052] Similar to the rotation invariant transformation section 330 already described, the rotation invariant transformation section 430 orthogonalizes the basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the data after the orthogonalization.

[0053] The inference unit 440, like the inference unit 340 already described, identifies the points represented in the point cloud information by using the rotation invariant features and the model. Specifically, the inference unit 440 performs class identification of the points.

[0054] The result output unit 450 outputs the classification result based on the identification by the inference unit. Specifically, the result output unit 450 outputs the classification result to the evaluation unit 460 . The result output unit 450 may be further configured to output the classification result to an external device. In this case, the external device may be, for example, the result output device 900 described in the first embodiment.

[0055] The evaluation unit 460 evaluates the model using the classification results. Various indexes that measure the error between the output label and the correct label can be used as the evaluation method in the evaluation unit 460. Basic evaluation methods include, for example, an evaluation method using a cross-entropy error function or a square error function. The evaluation unit 460 outputs the evaluation result to the model update unit 470 .

[0056] The model update unit 470 updates the model using the evaluation result by the evaluation unit 460 . Specifically, the model update unit 470 updates the model by rewriting the learning parameters stored as the model in the storage unit 210A of the storage device 200A. For example, when using an evaluation result based on a cross-entropy error function, the model updating unit 470 updates the parameter W in a direction in which the value of the error function becomes minimum. The direction in which the function value becomes minimum is the direction of the gradient in which the function decreases when the error function is differentiated with respect to the parameter W of the model. At this time, the model updating unit 470 may update the error function until the value converges, or may stop updating midway when a predetermined condition is satisfied. The model update unit 470 performs the update using a general optimization method such as the stochastic gradient descent method, the Newton method, etc. When calculating the gradient, the gradient of all parameters in the model is calculated sequentially using a method called backpropagation, for example.

[0057] An example of the internal configuration of the rotation invariant transformation unit 430 will be described. Here, the internal configuration of rotation invariant transformation unit 430 is such that orthogonalization layer unit 331 and projection layer unit 332 in rotation invariant transformation unit 330 already described in FIG. 2 are replaced with orthogonalization layer unit 431 and projection layer unit 432, and are not shown in the figure.

[0058] The rotation invariant transformation unit 430 includes an orthogonalization layer unit 431 and a projection layer unit 432 .

[0059] Similar to the orthogonalization layer unit 331 already described, the orthogonalization layer unit 431 calculates basis vectors for each of k coordinates indicating a point (each of N points) included in the point cloud information, and converts the k basis vectors so that they are orthogonal to each other to generate an (N × k) orthonormal basis matrix. The projection layer unit 432, like the projection layer unit 332 already described, calculates a projection matrix indicating rotation invariant features using an orthonormal basis matrix.

[0060] The orthogonalization process by orthogonalization layer unit 431 is similar to the orthogonalization process by orthogonalization layer unit 331 already described, and therefore a further detailed description thereof will be omitted here.

[0061] When the rotation invariant transformation unit 430 is configured in this manner, the inference unit 440 identifies the points represented in the point cloud information using the projection matrix and model that indicate the rotation invariant features.

[0062] An example of the internal configuration of the inference unit 440 will be described. The inference unit 440 extracts point cloud features based on the result of orthogonal projection of the components indicated by the learning parameters into a subspace spanned by an orthonormal basis matrix in an N-dimensional space using a projection matrix and a model indicating the rotation invariant features, and identifies the point cloud using the point cloud features. Specifically, it performs class identification of the point cloud.

[0063] The point cloud features constructed by inference unit 440 are similar to the point cloud features constructed by inference unit 340 already described and will not be described in further detail here.

[0064] This allows the present disclosure to identify point cloud information using features specific to the point cloud shape that are independent of the coordinates of the point cloud.

[0065] The processing of the point cloud identification device will be described with reference to FIG. FIG. 8 is a flowchart showing an example of processing by the point cloud learning device 400.

[0066] When starting processing, point cloud learning device 400 first determines whether to perform learning (step ST100). Specifically, the point cloud learning device 400 selects whether to learn the learning parameters from the beginning. The point cloud learning device 400 checks, for example, preset setting information to determine whether to learn from the beginning or to use the learning parameters of the previously stored model.

[0067] When it is determined that learning is to be performed (step ST100 "YES"), point cloud learning device 400 proceeds to the process of step ST120. When learning from scratch, point cloud learning device 400 may initialize learning parameters randomly, or may initialize using any commonly used initialization method.

[0068] When the point cloud learning device 400 determines not to perform learning (step ST100 "NO"), the point cloud learning device 400 executes a model acquisition process (step ST110).

[0069] After determining to perform learning (step ST100 "YES") or after executing model acquisition processing (step ST110), point cloud learning device 400 executes point cloud acquisition processing (step ST120). Specifically, the point cloud acquisition unit 420 in the point cloud learning device 400 acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions from the point cloud input device 100. The point cloud acquisition unit 420 outputs the acquired point cloud information.

[0070] Point cloud learning device 400 executes rotation invariant transformation processing (step ST130). Specifically, rotation invariant transformation unit 430 in point cloud learning device 400 orthogonalizes the basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the data after the orthogonalization.

[0071] A specific example of the rotation invariant transformation process will be described below. Note that the rotation invariant transformation process by point cloud learning device 400 is similar to the rotation invariant transformation process in Fig. 4, and therefore a new illustration will be omitted here. When starting the rotation invariant transformation process, the rotation invariant transformation unit 430 executes an orthogonalization process (step ST131) as shown in FIG. Specifically, the orthogonalization layer 431 in the rotation invariant transformation unit 430 calculates a basis vector for each of k coordinates indicating a point (each of N points) included in the point cloud information, and performs transformation so that the k basis vectors are orthogonal to each other to generate an (N×k) orthogonal basis matrix. The orthogonalization layer 431 outputs the orthogonal basis matrix.

[0072] The rotation invariant transformation unit 430 executes a projection process (step ST132). Specifically, the projection layer unit 432 in the rotation invariant transformation unit 430 acquires an orthonormal basis matrix from the orthogonalization layer unit 431. The projection layer unit 432 calculates a projection matrix indicating rotation invariant features using the orthonormal basis matrix.

[0073] Point cloud learning device 400 executes inference processing (step ST140). Specifically, the inference unit 440 in the point cloud learning device 400 first obtains a projection matrix from the projection layer unit 432, and obtains a model from the model acquisition unit 410. The inference unit 440 then uses the rotation invariant features and the model to identify the point cloud indicated in the point cloud information. More specifically, the inference unit 440 uses a projection matrix and model indicating rotation invariant features to extract point cloud features based on the results of orthogonal projection of the components indicated in the learning parameters into a subspace spanned by an orthonormal basis matrix in N-dimensional space, and identifies the point cloud using the point cloud features.

[0074] Point cloud learning device 400 executes a result output process (step ST150). Specifically, result output section 450 in point cloud learning device 400 outputs the classification result based on the identification by inference section 440 to result output device 900.

[0075] Point cloud learning device 400 determines whether to end a series of processes (step ST160). If it is determined that the process should be ended ("YES" in step ST160), the point cloud learning device 400 ends the process.

[0076] If it is determined not to end the process ("NO" in step ST160), the point cloud learning device 400 executes a model evaluation process (step ST170). Specifically, the evaluation unit 460 in the point cloud learning device 400 evaluates the model using the classification result. The evaluation unit 460 outputs the evaluation result to the model update unit 470.

[0077] Point cloud learning device 400 executes a model update process (step ST180). The model update unit 470 updates the model using the evaluation result by the evaluation unit 460 . Specifically, the model update unit 470 updates the model by rewriting the learning parameters stored as the model in the storage unit 210A of the storage device 200A.

[0078] After executing the model update process (step ST180), point cloud learning device 400 proceeds to the process of step ST110, and repeats the processes from step ST110.

[0079] The learning device (point cloud learning device) of the present disclosure is configured as follows. A point cloud acquisition unit that acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition unit for acquiring a model having learning parameters; a rotation invariant transformation unit that orthogonalizes basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the orthogonalized data; an inference unit that uses the rotation invariant features and the model to identify points represented in the point cloud information; a result output unit that outputs a classification result based on the identification by the inference unit; an evaluation unit for evaluating the model using the classification result; a model update unit that updates the model using an evaluation result by the evaluation unit; A learning device equipped with As a result, the present disclosure has an effect of providing a learning device that improves the accuracy of identifying point clouds.

[0080] The learning method (point cloud learning method) of the present disclosure is configured as follows. a point cloud acquisition step in which a point cloud acquisition unit acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition step in which a model acquisition unit acquires a model having learning parameters; a rotation invariant transformation step in which a rotation invariant transformation unit orthogonalizes each basis vector for each point indicated in the point cloud information and calculates a rotation invariant feature using the orthogonalized information; an inference step of identifying points represented in the point cloud information using the rotation invariant features and the model by an inference unit; a result output step in which a result output unit outputs a classification result based on the identification by the inference unit; an evaluation step in which an evaluation unit evaluates the model using the classification result; a model updating step in which a model updating unit updates the model using an evaluation result by the evaluation unit; A learning method that includes: As a result, the present disclosure has an effect of providing a learning method that improves the accuracy of identifying point clouds.

[0081] The learning device (point cloud learning device) of the present disclosure is further configured as follows. The rotation invariant transformation unit an orthogonalization layer that calculates a basis vector for each of k coordinates indicating a point included in the point cloud information, and converts the k basis vectors so that they are orthogonal to each other to generate an orthogonal basis matrix; a projection layer unit that calculates a projection matrix indicating the rotation invariant feature by using the orthonormal basis matrix; having The inference unit is identifying a point cloud represented in the point cloud information using a projection matrix representing the rotation invariant features and the model; A learning device characterized by: As a result, the present disclosure has the effect of providing a learning device that can efficiently calculate rotation invariant features of a point cloud. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the above learning method.

[0082] The learning device (point cloud learning device) of the present disclosure is further configured as follows. The inference unit is Using the projection matrix representing the rotation invariant features and the model, extracting point cloud features based on a result of orthogonal projection of components indicated by the learning parameters onto a subspace spanned by the orthonormal basis matrix in an N-dimensional space, and identifying the point cloud using the point cloud features; A learning device characterized by: As a result, the present disclosure has the effect of providing a learning device that identifies point cloud information using features unique to the point cloud shape that are independent of the coordinates of the point cloud. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the above learning method.

[0083] Here, a hardware configuration for realizing the functions of the point cloud identification device 300 and the point cloud learning device 400 of the present disclosure will be described. FIG. 9 is a diagram illustrating a first example of a hardware configuration for realizing the functions of point cloud identification device 300 and point cloud learning device 400 in this disclosure. FIG. 10 is a diagram illustrating a second example of a hardware configuration for realizing the functions of point cloud identification device 300 and point cloud learning device 400 in the present disclosure. The point cloud identification device 300 and the point cloud learning device 400 of the present disclosure are realized by hardware such as that shown in FIG. 9 or FIG.

[0084] As shown in FIG. 9, the point cloud identification device 300 and the point cloud learning device 400 are configured with, for example, a processor 10001, a memory 10002, an input / output interface 10003, and a communication circuit 10004. The processor 10001 and the memory 10002 are installed in, for example, a computer. The memory 10002 stores programs for making the computer function as the model acquisition unit 310, the point cloud acquisition unit 320, the rotation invariant conversion unit 330, the inference unit 340, the result output unit 350, the model acquisition unit 410, the point cloud acquisition unit 420, the rotation invariant conversion unit 430, the inference unit 440, the result output unit 450, the evaluation unit 460, the model update unit 470, and a control unit (not shown). The processor 10001 reads and executes the programs stored in the memory 10002, thereby realizing the functions of the model acquisition unit 310, the point cloud acquisition unit 320, the rotation invariant conversion unit 330, the inference unit 340, the result output unit 350, the model acquisition unit 410, the point cloud acquisition unit 420, the rotation invariant conversion unit 430, the inference unit 440, the result output unit 450, the evaluation unit 460, the model update unit 470, and a control unit (not shown). Furthermore, a storage unit (not shown) is realized by the memory 10002 or another memory (not shown). The processor 10001 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, or a digital signal processor (DSP). Memory 10002 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable Read Only Memory) or a flash memory, or a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk. The processor 10001 and the memory 10002 are connected in a state in which they can transmit data to each other. The processor 10001 and the memory 10002 are also connected to other hardware via an input / output interface 10003 in a state in which they can transmit data to each other. Furthermore, the communication circuit 10004 realizes a communication unit (not shown).

[0085] Alternatively, the functions of the model acquisition unit 310, the point cloud acquisition unit 320, the rotation invariant transformation unit 330, the inference unit 340, the result output unit 350, the model acquisition unit 410, the point cloud acquisition unit 420, the rotation invariant transformation unit 430, the inference unit 440, the result output unit 450, the evaluation unit 460, the model update unit 470, and a control unit (not shown) may be realized by a dedicated processing circuit 20001 as shown in FIG. 10. The processing circuit 20001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration), etc. Furthermore, the memory 20002 or another memory (not shown) implements a storage unit (not shown). The memory 20002 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable Read Only Memory) or a flash memory, or a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk. The processing circuit 20001 and the memory 20002 are connected in a state in which they can transmit data to each other. In addition, the processing circuit 20001 and the memory 20002 are connected via the input / output interface 20003 in a state in which they can transmit data to other hardware. Furthermore, the communication circuit 20004 realizes a communication unit (not shown). The functions of the model acquisition unit 310, the point cloud acquisition unit 320, the rotation invariant transformation unit 330, the inference unit 340, the result output unit 350, the model acquisition unit 410, the point cloud acquisition unit 420, the rotation invariant transformation unit 430, the inference unit 440, the result output unit 450, the evaluation unit 460, the model update unit 470, and a control unit (not shown) may be realized by separate processing circuits, or may be realized collectively by a processing circuit.

[0086] Alternatively, some of the functions of the model acquisition unit 310, point cloud acquisition unit 320, rotation invariant transformation unit 330, inference unit 340, result output unit 350, model acquisition unit 410, point cloud acquisition unit 420, rotation invariant transformation unit 430, inference unit 440, result output unit 450, evaluation unit 460, model update unit 470, and a control unit (not shown) may be realized by the processor 10001 and memory 10002, and the remaining functions may be realized by the processing circuit 20001.

[0087] In addition, the present disclosure allows free combination of each embodiment, modification of any component of each embodiment, or omission of any component of each embodiment within the scope of the invention. For example, the first embodiment and the second embodiment may be combined to output the point cloud identification result to the result output device by the result output unit, and to output the result to the model update unit via the evaluation unit. This allows the effects of both the first embodiment and the second embodiment to be achieved. [Industrial Applicability]

[0088] The point cloud identification device and learning device disclosed herein can each improve the accuracy of point cloud identification, and are therefore suitable for use in identifying point clouds in technologies such as vehicle control and driving assistance. [Explanation of symbols]

[0089] 1 Point cloud identification system, 2 Point cloud learning system, 100 Point cloud input device, 200, 200A Storage device, 210, 210A Storage unit, 300 Point cloud identification device, 310 Model acquisition unit, 320 Point cloud acquisition unit, 330 Rotation invariant transformation unit, 331 Orthogonalization layer unit, 332 Projection layer unit, 340 Inference unit, 350 Result output unit, 400 Point cloud learning device, 410 Model acquisition unit, 420 Point cloud acquisition unit, 430 Rotation invariant transformation unit, 431 Orthogonalization layer unit, 432 Projection layer unit, 440 Inference unit, 450 Result output unit, 460 Evaluation unit, 470 Model update unit, 900 Result output device.

Claims

1. a point cloud acquisition unit that acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition unit for acquiring a model having learning parameters; a rotation invariant transformation unit that orthogonalizes basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the orthogonalized data; an inference unit that uses the rotation invariant features and the model to identify points represented in the point cloud information; a result output unit that outputs a classification result based on the identification by the inference unit; A point cloud identification device comprising:

2. The rotation invariant transformation unit an orthogonalization layer unit that calculates a basis vector for each of k coordinates indicating a point included in the point cloud information, and converts the k basis vectors so that they are orthogonal to each other to generate an orthogonal basis matrix; a projection layer unit that calculates a projection matrix indicating the rotation invariant feature by using the orthonormal basis matrix; having The inference unit is identifying a point cloud represented in the point cloud information using a projection matrix representing the rotation invariant features and the model; The point cloud identification device according to claim 1 .

3. The inference unit is Using the projection matrix representing the rotation invariant features and the model, extracting point cloud features based on a result of orthogonal projection of components indicated by the learning parameters onto a subspace spanned by the orthonormal basis matrix in an N-dimensional space, and identifying the point cloud using the point cloud features; The point cloud identification device according to claim 2 .

4. a point cloud acquisition unit that acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition unit for acquiring a model having learning parameters; a rotation invariant transformation unit that orthogonalizes basis vectors for each point indicated in the point cloud information, and calculates rotation invariant features using the orthogonalized data; an inference unit that uses the rotation invariant features and the model to identify points represented in the point cloud information; a result output unit that outputs a classification result based on the identification by the inference unit; an evaluation unit for evaluating the model using the classification result; a model update unit that updates the model using an evaluation result by the evaluation unit; A learning device equipped with

5. The rotation invariant transformation unit an orthogonalization layer unit that calculates a basis vector for each of k coordinates indicating a point included in the point cloud information, and converts the k basis vectors so that they are orthogonal to each other to generate an orthogonal basis matrix; a projection layer unit that calculates a projection matrix indicating the rotation invariant feature by using the orthonormal basis matrix; having The inference unit is identifying a point cloud represented in the point cloud information using a projection matrix representing the rotation invariant features and the model; 5. The learning device according to claim 4.

6. The inference unit is Using the projection matrix representing the rotation invariant features and the model, extracting point cloud features based on a result of orthogonal projection of components indicated by the learning parameters onto a subspace spanned by the orthonormal basis matrix in an N-dimensional space, and identifying the point cloud using the point cloud features; 6. The learning device according to claim 5.

7. A point cloud identification method executed by a point cloud identification device, comprising: a point cloud acquisition step in which a point cloud acquisition unit of the point cloud identification device acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition step in which a model acquisition unit of the point cloud identification device acquires a model having learning parameters; a rotation invariant transformation step in which a rotation invariant transformation unit of the point cloud identification device orthogonalizes each basis vector for each point indicated in the point cloud information and calculates a rotation invariant feature using the orthogonalized information; an inference step of identifying a point cloud represented in the point cloud information by an inference unit of the point cloud identification device using the rotation invariant features and the model; a result output step in which a result output unit of the point cloud identification device outputs a classification result based on the identification by the inference unit; The point cloud identification method includes:

8. A learning method using a learning device, comprising: a point cloud acquisition step in which a point cloud acquisition unit of the learning device acquires point cloud information indicating N (N≧2) points in k (k≧2) dimensions; a model acquisition step in which a model acquisition unit of the learning device acquires a model having learning parameters; a rotation invariant transformation step in which a rotation invariant transformation unit of the learning device orthogonalizes each basis vector for each point indicated in the point cloud information and calculates a rotation invariant feature using the orthogonalized information; an inference step of identifying a point group represented in the point cloud information by an inference unit of the learning device using the rotation invariant features and the model; a result output step in which a result output unit of the learning device outputs a classification result based on the discrimination by the inference unit; an evaluation step in which an evaluation unit of the learning device evaluates the model using the classification result; a model updating step in which a model updating unit of the learning device updates the model using an evaluation result by the evaluation unit; A learning method that includes:

Images