Object detection device, object detection method, and object detection program

Abstract

In order to detect an object with high precision, an object detection device (1) comprises: a first calculation means that selects a seed point; a second calculation means that calculates a center position of an object to which the seed point belongs; a third calculation means that calculates an object accuracy of the seed point; and a generation means that selects temporary representative points and representative points and generates a cluster on the basis of each representative point.

Description

OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION PROGRAM

[0001] The present invention relates to an object detection device, an object detection method, and an object detection program.

[0002] Deep learning technologies such as deep learning have achieved remarkable results in various fields, including image and audio. Compared to images, 3D point clouds have accurate spatial positioning and are robust to lighting variations, which has led to active development of object detection technologies using 3D point clouds.

[0003] Under such circumstances, an example of a technology for estimating the type of an object, a three-dimensional circumscribing rectangle, and object-likeness based on seed points selected from an input three-dimensional point cloud is described in Non-Patent Document 1.

[0004] Charles R. Qi et. al., "Deep Hough Voting for 3D Object Detection in Point Clouds", IEEE / CVF International Conference on Computer Vision(ICCV), 2019, pp. 9276-9285

[0005] In Non-Patent Document 1, the representative points further selected from the seed points are selected so as not to bias the distribution of the representative points, which has led to the problem that the selected seed points do not belong to an object, or that seed points that belong to an object with few points are not selected as representative points. For this reason, there is room for improvement in the object detection accuracy of the technology described in Non-Patent Document 1.

[0006] One aspect of the present invention has been made in view of the above-described problems, and one object thereof is to provide a technology for performing object detection with high accuracy.

[0007] An object detection device according to one aspect of the present invention comprises: first calculation means for selecting a plurality of seed points from a three-dimensional point cloud representing at least one object, and performing a first calculation process to calculate a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed point; second calculation means for performing a second calculation process to calculate, for each seed point, a second feature amount that is a feature amount of the seed point by referring to the first feature amount of the seed point; third calculation means for performing a third calculation process to calculate, for each seed point, an object likelihood that indicates the likelihood of the seed point being an object by referring to the second feature amount of the seed point; and cluster generation means for selecting at least one temporary representative point from the plurality of seed points, and for each temporary representative point, re-selecting as a representative point a seed point with the highest object likelihood from among the seed points included in a neighborhood of the temporary representative point, generating, for each representative point, a cluster made up of the seed points included in a neighborhood of the representative point, and linking two clusters where the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold.

[0008] an object detection method according to one aspect of the present invention includes: selecting a plurality of seed points from a three-dimensional point cloud representing at least one object; performing a first calculation process to calculate a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed point; performing a second calculation process to calculate, for each seed point, a second feature amount that is a feature amount of the seed point by referring to the first feature amount of the seed point; and performing a third calculation process to calculate, for each seed point, an object likelihood that indicates a likelihood of the seed point being an object by referring to the second feature amount of the seed point; selecting at least one temporary representative point from the plurality of seed points; re-selecting, for each temporary representative point, the seed point with the highest object likelihood from among the seed points included in a neighborhood of the temporary representative point as the representative point; generating, for each representative point, a cluster made up of the seed points included in a neighborhood of the representative point; and linking two clusters where the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold.

[0009] An object detection program according to one aspect of the present invention is an object detection program for causing a computer to function as an object detection device, the program including at least one processor included in the computer, a first calculation means for executing a first calculation process of selecting a plurality of seed points from a three-dimensional point cloud representing at least one object, and calculating a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed points, a second calculation means for executing a second calculation process of calculating a second feature amount that is a feature amount of each seed point by referring to the first feature amount of the seed points, and a third calculation means for executing a second calculation process of calculating a second feature amount that is a feature amount of each seed point by referring to the first feature amount of the seed points. and a cluster generating means for selecting at least one temporary representative point from the plurality of seed points, and for each temporary representative point, reselecting the seed point with the highest object likelihood from among the seed points included in the neighborhood of the temporary representative point as the representative point, generating, for each representative point, a cluster made up of the seed points included in the neighborhood of the representative point, and linking two clusters where the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold.

[0010] According to one aspect of the present invention, it is possible to provide a technique for improving the accuracy of object detection.

[0011] FIG. 1 is a block diagram showing the configuration of an object detection device 1 according to exemplary embodiment 1 of the present invention. FIG. 2 is a flow diagram showing the flow of an object detection method S1 according to exemplary embodiment 1 of the present invention. FIG. 3 is a diagram showing an example of processing by a cluster generation unit 14 according to exemplary embodiment 1 of the present invention. FIG. 4 is a block diagram showing the configuration of an object detection device 1A according to exemplary embodiment 2 of the present invention. FIG. 5 is a flow diagram showing the flow of an object detection method S1A according to exemplary embodiment 2 of the present invention. FIG. 6 is a block diagram showing the configuration of an object detection device 1B according to exemplary embodiment 3 of the present invention. FIG. 7 is a flow diagram showing the flow of an object detection method S1B according to exemplary embodiment 3 of the present invention. FIG. 8 is a block diagram showing the configuration of an object detection program according to exemplary embodiment 4 of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0013] (Overview of Object Detection Device 1) The object detection device 1 is a device for detecting an object from a three-dimensional point cloud representing at least one object.

[0014] In order to detect an object, a technique is disclosed in which feature quantities of a three-dimensional point cloud are hierarchically extracted around a seed point selected from an input three-dimensional point cloud, an object center is estimated based on the extracted feature quantities, seed points are selected and clustered based on the estimated object centers, and the type of object, a three-dimensional circumscribing rectangle, and object likelihood are estimated for each cluster. The seed point selection uses a farthest point sampling (FPS) method to prevent bias in the distribution of the selected seed points (hereinafter also referred to as representative points). However, seed point selection using the FPS method has problems such as false detection (selection of seed points unrelated to the object) and non-detection (non-selection of seed points from objects with few points).

[0015] The object detection device 1 includes a means for executing a process of selecting, as a representative point, a seed point that has a high object-likeness (hereinafter also referred to as "object accuracy") from among seed points selected from a three-dimensional point cloud. By including a means for executing a process of selecting a representative point based on the object accuracy, it is possible to prevent erroneous detection and non-detection.

[0016] (Configuration of Object Detection Device 1) The configuration of the object detection device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the object detection device 1.

[0017] 1 , the object detection device 1 includes a first calculation unit 11, a second calculation unit 12, a third calculation unit 13, and a cluster generation unit 14. In this exemplary embodiment, the first calculation unit 11, the second calculation unit 12, the third calculation unit 13, and the cluster generation unit 14 are configured to realize a first calculation means, a second calculation means, a third calculation means, and a cluster generation means, respectively.

[0018] The first calculation unit 11 is configured to execute a first calculation process that selects multiple seed points from a 3D point cloud representing at least one object and calculates first feature amounts, which are feature amounts of each seed point. The first feature amounts are calculated by referring to the coordinate values ​​of the seed points. The first calculation unit 11 outputs the coordinates of the seed points and the first feature amounts to the second calculation unit 12. The first calculation unit 11 also outputs the coordinates of the seed points to the cluster generation unit 14.

[0019] The second calculation unit 12 is configured to execute a second calculation process for calculating, for each seed point, a second feature amount that is a feature amount of the seed point. The second feature amount of each seed point can be calculated by referring to the first feature amount of the seed point. The second calculation unit 12 outputs the second feature amount to the third calculation unit 13.

[0020] The third calculation unit 13 is configured to execute a third calculation process to calculate, for each seed point, an object probability that indicates the likelihood of the seed point being an object. The object probability of each seed point can be calculated by referring to the second feature amount of the seed point. The third calculation unit 13 outputs the object probability to the cluster generation unit 14.

[0021] The cluster generation unit 14 is configured to select at least one representative point from a plurality of seed points and generate a cluster based on the representative points. The cluster generation unit 14 selects a temporary representative point, reselects each representative point by referring to the neighborhood of each temporary representative point, generates a cluster consisting of seed points for each representative point, and connects two clusters in which the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold. At least one temporary representative point is selected from the plurality of seed points. Each representative point is selected as the seed point with the highest object likelihood among the seed points included in the neighborhood of the temporary representative point. The cluster generated for each representative point is made up of seed points included in the neighborhood of that representative point.

[0022] The first calculation unit 11, the second calculation unit 12, the third calculation unit 13, and the cluster generation unit 14 described above may be realized by, for example, a neural network.

[0023] (Flow of Object Detection Method S1) The flow of the object detection method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the object detection method S1. As shown in Fig. 2, the object detection method S1 includes a first calculation process S11, a second calculation process S12, a third calculation process S13, and a cluster generation process S14.

[0024] In the first calculation process S11, a plurality of seed points are selected from a three-dimensional point cloud representing at least one object, and a first feature amount, which is a feature amount of each seed point, is calculated by referring to the coordinate values ​​of the seed points.

[0025] The second calculation process S12 calculates, for each seed point, a second feature value that is a feature value of the seed point. The second feature value of each seed point can be calculated by referring to the first feature value of the seed point.

[0026] The third calculation process S13 calculates an object probability for each seed point, which indicates the likelihood of the seed point being an object. The object probability for each seed point can be calculated by referring to the second feature amount for the seed point.

[0027] The cluster generation process S14 selects at least one representative point from a plurality of seed points and generates a cluster based on the representative points. The cluster generation process S14 selects tentative representative points, reselects each representative point by referring to the neighborhood of each tentative representative point, generates a cluster consisting of seed points for each representative point, and connects two clusters in which the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold. At least one tentative representative point is selected from the plurality of seed points. Each representative point is selected as the seed point with the highest object likelihood among the seed points included in the neighborhood of the tentative representative point. The cluster generated for each representative point is made up of seed points included in the neighborhood of that representative point.

[0028] (Effects of Object Detection Device 1 and Object Detection Method S1) As described above, the object detection device 1 and object detection method S1 according to this exemplary embodiment employ a configuration in which the object certainty of the seed points is referenced when selecting a representative point. With this configuration, by selecting seed points with high object certainty as representative points, it is possible to prevent erroneous detection or non-detection in the selection of seed points. Therefore, the object detection device 1 and object detection method S1 according to this exemplary embodiment have the effect of improving object detection accuracy.

[0029] (Specific Example of First Calculation Unit 11) When calculating the first feature amount for each seed point, color information (RGB values), normal vectors, curvature, and the like may be referenced in addition to the coordinate values ​​of the seed point.

[0030] The first calculation process executed by the first calculation unit 11 may be a process of selecting seed points from a three-dimensional point cloud, generating local clusters whose elements are points included in local regions among the points constituting the three-dimensional point cloud, and performing a first convolution to calculate a first feature value as a feature value for each local cluster. The seed points are selected from the three-dimensional point cloud input to the first calculation unit 11. Local clusters are generated by determining local regions centered on each of the multiple seed points and using points of the three-dimensional point cloud included in the local regions as elements. The first convolution process is a process in which, for each local cluster, coordinate values ​​of the points constituting the local cluster are input and the feature value of the local cluster is calculated as a first feature value of the seed points belonging to the local cluster. This configuration is preferable in that it allows for calculating a feature value that aggregates the feature values ​​of points other than the seed points included in the local region as the first feature value.

[0031] (Specific Example of Second Calculation Unit 12) The second calculation unit 12 may further calculate, for each seed point, the center position of an object to which the seed point belongs. The second calculation process executed by the second calculation unit 12 may be a second convolution process that receives the first feature value of each seed point as input and outputs the center position of the object indicated by the seed point and the second feature value of the seed point.

[0032] (Specific Example of Third Calculation Unit 13) The third calculation unit 13 may further calculate, for each seed point, a third feature amount that is a feature amount of the seed point. The third calculation process executed by the third calculation unit 13 may be a third convolution process that receives the second feature amount of each seed point as an input and outputs the object likelihood of the seed point and the third feature amount of the seed point. The object likelihood may be a numerical value between 0 and 1.

[0033] (Specific Example of Cluster Generation Unit 14) The tentative representative point can be selected by FPS (Farthest Point Sampling). For example, an arbitrary point p1 is selected from a plurality of seed points, and point p2, which is the furthest from the selected point {p1}, is selected. Furthermore, point p3, which is the furthest from the selected point {p1, p2}, is selected. The furthest point from the already selected points is continued to be selected, and this process is repeated until a predetermined number of points have been selected. Here, as the distance measure, for example, the inter-feature distance relating to the first feature amount, the second feature amount, or the third feature amount can be used.

[0034] The neighborhood of the temporary representative point may be any neighborhood as long as it includes the temporary representative point. The size of the neighborhood of the temporary representative point may be a predetermined fixed size. If the third calculation unit 13 is configured to calculate the third feature amount and the cluster generation unit 14 is realized by a neural network, the size of the neighborhood of the temporary representative point may be a size determined based on the third feature amount of the temporary representative point.

[0035] The neighborhood of the representative point may be any neighborhood as long as it includes the representative point. The size of the neighborhood of the representative point may be a predetermined fixed size. If the third calculation unit 13 is configured to calculate the third feature amount and the cluster generation unit 14 is realized by a neural network, the size of the neighborhood of the representative point may be a size determined based on the third feature amount of the representative point.

[0036] An example of the processing performed by the cluster generation unit 14 will be described with reference to Fig. 3. Fig. 3 is a diagram schematically showing an example of the processing from determining a representative point A and its neighboring area from a point group made up of seed points.

[0037] The point cloud in the upper left of Fig. 3 is a point cloud made up of seed points input to the cluster generation unit 14. In the point cloud, the points indicated by black circles are seed points, and the rectangular area X is an area corresponding to object X. In the point cloud in the lower left, the points indicated by white circles are tentative representative points selected from the seed points by the cluster generation unit 14. Hereinafter, the tentative representative point A' will be described. The area indicated by a circle with a dashed line is the neighborhood area of ​​tentative representative point A'. In the cluster generation unit 14, in the upper right of Fig. 3, representative point A with the highest object certainty is reselected from the neighborhood area of ​​tentative representative point A'. Then, the seed points included in the neighborhood area of ​​representative point A indicated by the dashed line are generated as a cluster of representative point A.

[0038] In the neighborhood of the temporary representative point A', the region X corresponding to the object contains only one seed point, but in the neighborhood of the representative point A, the region X corresponding to the object contains three seed points. In this way, by reselecting the representative point A with a high object certainty, it is possible to cluster the seed points with a high object certainty in the neighborhood of the representative point A.

[0039] (Modification of First Calculation Unit 11) The object detection device 1 may include a first calculation unit 11' instead of the first calculation unit 11.

[0040] The first calculation unit 11′ is configured to perform a first calculation process including a selection process of selecting a plurality of points from the three-dimensional point cloud, a local cluster generation process of generating a local cluster for each of the plurality of points, and a first convolution process of calculating a feature amount of each local cluster. Furthermore, the first calculation unit 11′ repeatedly executes a cycle of processes included in the first calculation process using, as new inputs, the feature amounts of local clusters including each of the plurality of points, which are three-dimensional point clouds configured by each of the plurality of points.

[0041] A local cluster is a cluster generated based on a local region that includes each of a plurality of points. The size of the local region is determined by a fixed value or by convolution processing of the coordinate values ​​of each of the plurality of points. For each local region generated from each of a plurality of seed points, a cluster is generated whose elements are points included in the local region among the points that make up the three-dimensional point cloud, and this is called a local cluster. The feature amount of the local cluster is calculated by a first convolution processing that inputs the coordinate values ​​that make up the local cluster.

[0042] (Modification of Object Detection Device 1) The object detection device 1 may include a removal unit that removes points assigned to a background region from the three-dimensional point cloud. A background region refers to an asymmetric region that is not used for object detection, such as the ground and a wall. For example, points corresponding to the background region may be estimated from the three-dimensional point cloud by segmentation. Furthermore, points corresponding to the background region may be removed from the three-dimensional point cloud by referring to the segmentation results of a different modality, such as an image.

[0043]

[0033] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0044] (Configuration of object detection device 1A) The configuration of the object detection device 1A according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the object detection device 1A.

[0045] 4 , the object detection device 1A includes a first calculation unit 11, a second calculation unit 12, a third calculation unit 13A, a cluster generation unit 14, and a classification unit 15. The first calculation unit 11, the second calculation unit 12, and the cluster generation unit 14 of the object detection device 1A have the same configurations as the first calculation unit 11, the second calculation unit 12, and the cluster generation unit 14 of the object detection device 1. In this embodiment, the third calculation unit 13A and the classification unit 15 are configured to realize a third calculation means and a classification means, respectively.

[0046] The third calculation unit 13A is configured to execute a third calculation process to further calculate, for each seed point, a third feature amount, which is a feature amount of that point. The third feature amount for each seed point is calculated by referring to the object probability and the second feature amount of that seed point. The third feature amount calculated by the third calculation unit 13A is output to the classification unit 15.

[0047] The classification unit 15 is configured to classify each cluster generated by the cluster generation unit 14 into a class corresponding to an object. The classification unit 15 performs classification of each cluster by referring to the feature of the cluster. The feature of each cluster is calculated by referring to the third feature of the seed points included in the cluster. The calculation process of the feature of each cluster may be a convolution process in which the third feature of the seed points included in the cluster is used as an input and the feature of the cluster is used as an output.

[0048] In this embodiment, the classification unit 15 calculates a corresponding object score y for each object (each class) i. ｉ Calculate the object score y ｉ is defined as a probability value normalized so that the sum becomes 1 using a softmax function, for example, as in equation (1).

[0049] In the formula, x ｋ , x ｉ is the output of the neural network, and y ｉ is the object score, and i is a natural number between 1 and n.

[0050] Such a classifier 15 may be realized by a neural network, and in particular, by an MLP (Multi-layer Perceptron) network.

[0051] (Flow of Object Detection Method S1A) The flow of the object detection method S1A according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing the flow of the object detection method S1A. As shown in Fig. 5, the object detection method S1A includes a first calculation process S11, a second calculation process S12, a third calculation process S13A, a cluster generation process S14, and a classification process S15. The first calculation process S11, the second calculation process S12, and the cluster generation process S14 of the object detection method S1A have the same configuration as the first calculation process S11, the second calculation process S12, and the cluster generation process S14 of the object detection method S1.

[0052] The third calculation process S13A further calculates a third feature amount for each seed point, which is a feature amount of that point. The third feature amount for each seed point is calculated by referring to the object probability of that seed point and the second feature amount of that seed point.

[0053] The classification process S15 classifies the classes to which each cluster generated in the cluster generation process S14 belongs. The classification of each cluster is performed by referring to the feature of the cluster. The feature of each cluster is calculated by referring to the third feature of the seed points included in the cluster.

[0054] (Effects of Object Detection Device 1A) As described above, the object detection device 1A according to this exemplary embodiment employs a configuration for classifying objects based on the feature amounts of clusters. Therefore, in addition to the effects of the object detection device 1 according to the first exemplary embodiment, the object detection device 1A according to this exemplary embodiment can also achieve the effect of being able to classify objects with high accuracy.

[0055]

[0033] A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will not be repeated.

[0056] (Configuration of Object Detection Device 1B) The configuration of the object detection device 1B according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the configuration of the object detection device 1B.

[0057] 6 , the object detection device 1B includes a first calculation unit 11, a second calculation unit 12A, a third calculation unit 13A, a cluster generation unit 14, a classification unit 15, and an update unit 16. The first calculation unit 11 and the cluster generation unit 14 of the object detection device 1B have the same configuration as the first calculation unit 11 and the cluster generation unit 14 of the object detection device 1, and the third calculation unit 13A and the classification unit 15 have the same configuration as the third calculation unit 13A and the classification unit 15 of the object detection device 1A. In this embodiment, the third calculation unit 13A and the update unit 16 are configured to realize third calculation means and update means, respectively.

[0058] The second calculation unit 12A is configured to execute a second calculation process to further calculate, for each seed point, the center position of the object to which the seed point belongs. The center position of the object to which each seed point belongs is calculated by referring to the first feature value of the seed point as well as the second feature value that is the feature value of the seed point. The center position of the object calculated by the second calculation unit 12A is output to the update unit 16.

[0059] The update unit 16 is configured to execute an update process for updating the first loss, the second loss, and the third loss using parameters that define the first calculation process, parameters that define the second calculation process, and parameters that define the third calculation process. To calculate each loss, the update unit 16 refers to the center position of the object calculated by the second calculation unit 12A, the object accuracy calculated by the third calculation unit 13A, and the classification result calculated by the classification unit 15. The update is performed so as to minimize the sum of the first loss, the second loss, and the third loss.

[0060] The first loss is the sum of first deviations representing the deviation between the center position of the object to which each seed point belongs and the correct answer identified from the training data. The first loss calculation process calculates the first deviations and calculates the first loss, which is the sum of the first deviations. The training data for the center position of the object to which each seed point belongs can be calculated in advance using any method.

[0061] The second loss is the sum of second deviations representing the deviation between the object accuracy of each seed point and the correct answer identified from the training data. The second loss calculation process calculates the second deviations and calculates the second loss, which is the sum of the second deviations. The training data for the object accuracy of each seed point (training data indicating the correct answer as to whether each seed point belongs to an object) can be calculated in advance.

[0062] The third loss is the sum of third deviations representing the deviation between the classification result of each seed point in the classification unit 15 and the correct answer identified from the training data. The third loss calculation process calculates the third deviations and calculates the third loss, which is the sum of the third deviations. The correct answer identified from the training data may be a one-hot vector in which the probability value of the correct answer class to which each seed point belongs is 1 and the probability value of the other object is 0.

[0063] The first loss, the second loss, and the third loss may be calculated from the following formula:

[0064]

[0065] When calculating the first loss, in equations (2) and (3), L is the first loss, and y ｉ is the center position of the object to which each seed point belongs, and t ｉ is the correct answer identified from the training data. When calculating the second loss, in Equation (2) and Equation (3), L is the second loss, and y ｉ is the object probability of each seed point, and t ｉ is the correct answer identified from the training data. When calculating the third loss, in Equation (2) and Equation (3), L is the third loss, and y ｉ is the classification result for each seed point, and t ｉ is the correct answer identified from the training data.

[0066] (Flow of Object Detection Method S1B) The flow of object detection method S1B according to this exemplary embodiment will be described with reference to FIG. 7 . FIG. 7 is a flow diagram showing the flow of object detection method S1B. As shown in FIG. 7 , object detection method S1B includes a first calculation process S11, a second calculation process S12A, a third calculation process S13A, a cluster generation process S14, a classification process S15, and an update process S16. The first calculation process S11 and the cluster generation process S14 of object detection method S1B are similar to the first calculation process S11 and the cluster generation process S14 of object detection method S1, and the third calculation process S13A and the classification process S15 are similar to the third calculation process S13A and the classification process S15 of object detection method S1A.

[0067] The second calculation process S12A further calculates, for each seed point, the center position of the object to which the seed point belongs, by referring to the first feature value of the seed point together with the second feature value of the seed point.

[0068] The updating process S16 updates the parameters that define the first calculation process, the parameters that define the second calculation process, and the parameters that define the third calculation process based on the first loss, the second loss, and the third loss. To calculate each loss, the updating unit 16 refers to the center position of the object calculated by the second calculation unit 12, the object accuracy calculated by the third calculation unit 13, and the classification result calculated by the classification unit 15. The updating is performed so as to minimize the sum of the first loss, the second loss, and the third loss.

[0069] The first loss is a sum of first deviations representing the deviation between the center position of the object to which each seed point belongs and the correct answer identified from the training data. The first loss calculation process calculates the first deviations and calculates the first loss, which is the sum of the first deviations.

[0070] The second loss is the sum of second deviations representing the deviation between the object accuracy of each seed point and the correct answer identified from the training data. The second loss calculation process calculates the second deviations and calculates the second loss, which is the sum of the second deviations.

[0071] The third loss is the sum of third deviations representing the deviation between the classification result of each seed point and the correct answer identified from the training data. The third loss calculation process calculates the third deviations and calculates the third loss, which is the sum of the third deviations.

[0072] (Effects of Object Detection Device 1B) As described above, the object detection device 1B according to this exemplary embodiment is configured to update the parameters that define the first calculation process, the parameters that define the second calculation process, and the parameters that define the third calculation process based on the center position of the object to which each calculated seed point belongs, the second calculation process, and the classification result. Therefore, the object detection device 1B according to this exemplary embodiment has the effect of being able to detect objects with higher accuracy, in addition to the effects achieved by the object detection device 1 according to the first exemplary embodiment.

[0073] [Example of Software Implementation] Some or all of the functions of the object detection devices 1, 1A, and 1B may be implemented by hardware such as an integrated circuit (IC chip), or may be implemented by software.

[0074] In the latter case, the object detection devices 1, 1A, and 1B are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 8. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the object detection devices 1, 1A, and 1B. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the object detection devices 1, 1A, and 1B.

[0075] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0076] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0077] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0078] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0079] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the following described aspects. and a cluster generating means for selecting at least one temporary representative point from the plurality of seed points, and for each temporary representative point, reselecting, from among the seed points in a neighborhood of the temporary representative point, a seed point with the highest object likelihood as the representative point, generating, for each representative point, a cluster made up of the seed points in a neighborhood of the representative point, and connecting two clusters where the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold.

[0080] According to the above configuration, it is possible to perform object detection from a three-dimensional point cloud with high accuracy.

[0081] (Supplementary Note 2) The object detection device according to Supplementary Note 1, wherein the third calculation process further calculates, for each seed point, a third feature that is a feature of that seed point, and the object detection device further includes a classification means that calculates, for each cluster, the feature of that cluster by referring to the third feature of the seed points included in that cluster, and classifies a class to which each cluster belongs by referring to the feature of that cluster.

[0082] According to the above configuration, objects can be classified with high accuracy.

[0083] (Supplementary Note 3) The object detection device according to Supplementary Note 1 or 2, further comprising: first calculation means for performing a first calculation process, instead of the first calculation means, including: a selection process for selecting the plurality of seed points from the three-dimensional point cloud; a local cluster generation process for determining local regions including each of the plurality of seed points and generating local clusters having points included in the local regions among the points constituting the three-dimensional point cloud as elements; and a first convolution process for, for each local cluster, taking coordinate values ​​of the points constituting the local cluster as input and outputting a feature amount of the local cluster as a first feature amount; and further comprising: first calculation means for performing a first calculation process for repeatedly performing a cycle of processing included in the first calculation process using, as a new input, a three-dimensional point cloud constituted by each of a plurality of points, the feature amounts of local clusters including each of the plurality of points, and using the plurality of points selected in a final cycle as seed points and calculating the feature amounts of the seed points as first feature amounts.

[0084] According to the above configuration, points obtained by hierarchically aggregating features of local regions of a three-dimensional point cloud are used as seed points for object detection, thereby enabling object detection with higher accuracy.

[0085] (Supplementary Note 4) The object detection device according to Supplementary Note 2 or 3, wherein the second calculation process further calculates, for each seed point, a center position of an object to which the seed point belongs, and the second calculation process is a second convolution process that takes the first feature of each seed point as an input and outputs the center position of the object to which the seed point belongs and the second feature of the seed point, and the third calculation process is a third convolution process that takes the second feature of each seed point as an input and outputs the object accuracy of the seed point and the third feature of the seed point.

[0086] According to the above configuration, the second calculation process and the third calculation process can be performed efficiently and with high accuracy.

[0087] (Supplementary Note 5) A first loss calculation process for calculating a first loss, which is a sum of first deviations, from a first deviation representing a deviation between the center position of the object to which each seed point belongs and a correct answer identified from teacher data; a second loss calculation process for calculating a second loss, which is a sum of second deviations, from a second deviation representing a deviation between the object accuracy of each seed point and a correct answer identified from teacher data; and a second loss calculation process for calculating a second loss, which is a sum of second deviations, from a result of classification of each seed point in the classification means and a correct answer identified from teacher data. and an update means for executing an update process that updates parameters that define the first calculation process, parameters that define the second calculation process, and parameters that define the third calculation process so as to minimize the sum of the first loss, the second loss, and the third loss.

[0088] According to the above configuration, parameters are updated, so that more accurate object detection can be performed.

[0089] (Supplementary Note 6) The object detection device according to any one of Supplementary Notes 1 to 5, further comprising a removal unit that removes points assigned to a background region from the three-dimensional point cloud.

[0090] According to the above configuration, points in the background area that do not correspond to an object can be removed, so that object detection can be performed more efficiently and with higher accuracy.

[0091] (Supplementary Note 7) An object detection method comprising: selecting a plurality of seed points from a three-dimensional point cloud representing at least one object, and performing a first calculation process to calculate a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed point; performing a second calculation process to calculate, for each seed point, a second feature amount that is a feature amount of the seed point by referring to the first feature amount of the seed point; and performing a third calculation process to calculate, for each seed point, an object likelihood that indicates a likelihood of the seed point being an object by referring to the second feature amount of the seed point; selecting at least one temporary representative point from the plurality of seed points, and for each temporary representative point, re-selecting as a representative point a seed point with the highest object likelihood from among the seed points included in a neighborhood of the temporary representative point, generating, for each representative point, a cluster made up of the seed points included in a neighborhood of the representative point, and linking two clusters where a distance between cluster centers to which the representative points belong or a distance between the representative points is equal to or less than a threshold.

[0092] According to the above configuration, it is possible to perform object detection from a three-dimensional point cloud with high accuracy.

[0093] (Supplementary Note 8) An object detection program for causing a computer to function as an object detection device, comprising at least one processor included in the computer, a first calculation means for executing a first calculation process of selecting a plurality of seed points from a three-dimensional point cloud representing at least one object, and calculating a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed points, a second calculation means for executing a second calculation process of calculating, for each seed point, a second feature amount that is a feature amount of the seed point by referring to the first feature amount of the seed point, and an object probability that indicates the likelihood that the seed point is an object. and cluster generation means for selecting at least one temporary representative point from the plurality of seed points, and for each temporary representative point, reselecting as a representative point the seed point with the highest object likelihood from among the seed points included in a neighborhood of the temporary representative point, generating, for each representative point, a cluster made up of the seed points included in the neighborhood of the representative point, and linking two clusters where the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold.

[0094] According to the above configuration, it is possible to perform object detection from a three-dimensional point cloud with high accuracy.

[0095] [Additional Note 3] Part or all of the above-described embodiment can also be expressed as follows.

[0096] a first calculation process for calculating a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed point; a second calculation process for calculating a second feature amount that is a feature amount of each seed point by referring to the first feature amount of the seed point; a third calculation process for calculating an object likelihood that indicates a likelihood of the seed point being an object by referring to the second feature amount of the seed point; and a cluster generation process for selecting at least one temporary representative point from the plurality of seed points, re-selecting, for each temporary representative point, a seed point with the highest object likelihood from among the seed points included in a neighborhood of the temporary representative point, generating, for each representative point, a cluster made up of the seed points included in a neighborhood of the representative point, and linking two clusters where the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold.

[0097] The object detection device may further include a memory that stores a program for causing the processor to execute the first calculation process, the second calculation process, the third calculation process, and the cluster generation process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0098] REFERENCE SIGNS LIST 1, 1A, 1B ... object detection device 11 ... first calculation unit 12, 12A ... second calculation unit 13, 13A ... third calculation unit 14 ... cluster generation unit 15 ... classification unit 16 ... update unit

Claims

1. a first calculation means for executing a first calculation process that selects a plurality of seed points from a three-dimensional point cloud representing at least one object, and calculates a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed points; a second calculation means for performing a second calculation process to calculate, for each seed point, a second feature amount that is a feature amount of the seed point by referring to the first feature amount of the seed point; a third calculation means for executing a third calculation process for calculating, for each seed point, an object likelihood representing the likelihood of the seed point being an object, by referring to the second feature amount of the seed point; a cluster generating means for selecting at least one temporary representative point from the plurality of seed points, for each temporary representative point, reselecting as a representative point a seed point with the highest object likelihood from among the seed points included in a neighborhood area of ​​the temporary representative point, for each representative point, generating a cluster made up of the seed points included in the neighborhood area of ​​the representative point, and linking two clusters in which the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold value. Object detection device.

2. the third calculation process further calculates, for each seed point, a third feature amount that is a feature amount of the seed point; the object detection device further includes a classification means for calculating, for each cluster, a feature of the cluster by referring to the third feature of a seed point included in the cluster, and classifying each cluster into a class to which the cluster belongs by referring to the feature of the cluster. The object detection device according to claim 1 .

3. The first calculation means performing the first calculation process, which includes a selection process of selecting a plurality of points from the three-dimensional point cloud; a local cluster generation process of determining a local region including each of the plurality of points and generating a local cluster having points included in the local region among the points constituting the three-dimensional point cloud as elements; and a first convolution process of inputting, for each local cluster, coordinate values ​​of the points constituting the local cluster and outputting a feature amount of the local cluster as the first feature amount; a three-dimensional point cloud configured from the plurality of points, the three-dimensional point cloud having features of a local cluster including the plurality of points as features of the plurality of points, is used as a new input, and a cycle of processing included in the first calculation process is repeatedly executed, and the plurality of points selected in the final cycle are used as seed points, and the first calculation process is executed to calculate the features of the seed points as the first features. The object detection device according to claim 1 .

4. The second calculation process further calculates, for each seed point, a center position of an object to which the seed point belongs; the second calculation process is a second convolution process that receives the first feature amount of each seed point as an input, and outputs the center position of the object to which the seed point belongs and the second feature amount of the seed point, the third calculation process receives the second feature amount of each seed point as an input, a third convolution process that outputs the object likelihood of the seed point and the third feature amount of the seed point; The object detection device according to claim 2 .

5. a first loss calculation process for calculating a first loss, which is a sum of first deviation degrees, from first deviation degrees representing deviation degrees between the center positions of the objects to which each seed point belongs and a correct answer identified from teacher data; a second loss calculation process for calculating a second loss, which is a sum of second deviation degrees representing deviation degrees between the object accuracy of each seed point and a correct answer identified from training data; a third loss calculation process for calculating a third loss representing a sum of third deviation degrees from third deviation degrees representing deviation degrees between the classification result of each seed point in the classification means and a correct answer identified from the teacher data; further comprising an update means for executing an update process for updating a parameter that defines the first calculation process, a parameter that defines the second calculation process, and a parameter that defines the third calculation process so as to minimize a sum of the first loss, the second loss, and the third loss. The object detection device according to claim 4 .

6. The method further includes a removal means for removing points assigned to a background region from the three-dimensional point cloud. The object detection device according to claim 1 .

7. executing a first calculation process of selecting a plurality of seed points from a three-dimensional point cloud representing at least one object, and calculating a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed points; performing a second calculation process for calculating, for each seed point, a second feature amount that is a feature amount of the seed point by referring to the first feature amount of the seed point; performing a third calculation process for calculating, for each seed point, an object likelihood representing the likelihood of the seed point being an object, by referring to the second feature amount of the seed point; selecting at least one temporary representative point from the plurality of seed points, reselecting, for each temporary representative point, the seed point with the highest object likelihood from among the seed points included in a neighborhood of the temporary representative point as a representative point, generating, for each representative point, a cluster made up of the seed points included in the neighborhood of the representative point, and linking two clusters in which the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold value. Object detection methods.

8. An object detection program for causing a computer to function as an object detection device, the program comprising: a first calculation means for executing a first calculation process that selects a plurality of seed points from a three-dimensional point cloud representing at least one object, and calculates a first feature amount that is a feature amount of each seed point by referring to coordinate values ​​of the seed points; a second calculation means for performing a second calculation process to calculate, for each seed point, a second feature amount that is a feature amount of the seed point by referring to the first feature amount of the seed point; a third calculation means for executing a third calculation process for calculating, for each seed point, an object likelihood representing the likelihood of the seed point being an object, by referring to the second feature amount of the seed point; a cluster generating means for selecting at least one temporary representative point from the plurality of seed points, for reselecting, as a representative point, a seed point with the highest object likelihood among the seed points included in a neighborhood of each temporary representative point, for generating, for each representative point, a cluster made up of the seed points included in the neighborhood of the representative point, and linking two clusters in which the distance between the centers of the clusters to which the representative points belong or the distance between the representative points is equal to or less than a threshold value; Object detection program.