Object detection device, object detection method, and object detection program

Abstract

In order to detect an object with high precision, an object detection device (1) comprises: a first calculation means that selects a seed point; a second calculation means that calculates a center position of an object to which the seed point belongs; a third calculation means that calculates an object accuracy of the seed point; and a generation means that selects temporary representative points and representative points and generates a cluster on the basis of each representative point.

Description

[Technical Field]

[0001] This invention relates to an object detection device, an object detection method, and an object detection program. [Background technology]

[0002] Deep learning technologies, such as deep learning, have yielded remarkable results in various fields, including image and audio. Compared to images, three-dimensional point clouds have accurate spatial positioning and are robust to lighting fluctuations, leading to increased activity in the development of object detection technologies using three-dimensional point clouds.

[0003] Under these circumstances, Non-Patent Document 1 describes an example of a technique for estimating the type of object, the three-dimensional bounding rectangle, and the object-likeness based on a seed point selected from an input three-dimensional point cloud. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Charles R. Qi et. al., "Deep Hough Voting for 3D Object Detection in Point Clouds", IEEE / CVF International Conference on Computer Vision(ICCV), 2019, pp. 9276-9285 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] In Non-Patent Document 1, representative points are selected from the seed points in a way that avoids bias in the distribution of representative points. This leads to problems such as selected seed points not belonging to an object, or seed points belonging to objects with few points not being selected as representative points. Therefore, there is room for improvement in the object detection accuracy of the technology described in Non-Patent Document 1.

[0006] One aspect of the present invention has been made in view of the above-mentioned problems, and one example of its objective is to provide a technology for detecting objects with high accuracy. [Means for solving the problem]

[0007] An object detection device according to one aspect of the present invention includes: a first calculation means that selects a plurality of seed points from a three-dimensional point cloud representing at least one object and performs a first calculation process to calculate a first feature quantity, which is a feature quantity of each seed point, by referring to the coordinate values ​​of the seed point; a second calculation means that performs a second calculation process for each seed point to calculate a second feature quantity, which is a feature quantity of the seed point, by referring to the first feature quantity of the seed point; a third calculation means that performs a third calculation process for each seed point to calculate an object accuracy representing the object-likeness of the seed point by referring to the second feature quantity of the seed point; and a cluster generation means that selects at least one provisional representative point from the plurality of seed points, re-selects the seed point with the highest object accuracy among the seed points included in the neighborhood region of the provisional representative point as the representative point, generates a cluster consisting of seed points included in the neighborhood region of the representative point for each representative point, and connects two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold.

[0008] An object detection method according to one aspect of the present invention includes: selecting a plurality of seed points from a three-dimensional point cloud representing at least one object; performing a first calculation process to calculate a first feature quantity, which is a feature quantity of each seed point, by referring to the coordinate values ​​of the seed point; performing a second calculation process for each seed point to calculate a second feature quantity, which is a feature quantity of the seed point, by referring to the first feature quantity of the seed point; performing a third calculation process for each seed point to calculate an object accuracy, which represents the object-likeness of the seed point, by referring to the second feature quantity of the seed point; selecting at least one provisional representative point from the plurality of seed points; for each provisional representative point, re-selecting the seed point with the highest object accuracy among the seed points included in the neighborhood region of the provisional representative point as the representative point; generating a cluster for each representative point consisting of seed points included in the neighborhood region of the representative point; and connecting two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold.

[0009] An object detection program according to one aspect of the present invention is an object detection program for causing a computer to function as an object detection device, comprising: a first calculation means that performs a first calculation process in which at least one processor of the computer selects a plurality of seed points from a three-dimensional point cloud representing at least one object, and calculates a first feature quantity which is a feature quantity of each seed point by referring to the coordinate values ​​of the seed point; a second calculation means that performs a second calculation process for each seed point, calculating a second feature quantity which is a feature quantity of the seed point by referring to the first feature quantity of the seed point; and for each seed point, the The computer is configured to function as a third calculation means that performs a third calculation process to calculate an object accuracy representing the object-likeness of a point by referring to the second feature quantity of the seed point, and a cluster generation means that selects at least one provisional representative point from the plurality of seed points, and for each provisional representative point, re-selects the seed point with the highest object accuracy from among the seed points included in the neighborhood region of that provisional representative point as the representative point, generates a cluster consisting of seed points included in the neighborhood region of that representative point for each representative point, and connects two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold.

Advantages of the Invention

[0010] According to one aspect of the present invention, a technique for improving the detection accuracy of an object can be provided.

Brief Description of the Drawings

[0011] [Figure 1] It is a block diagram showing the configuration of an object detection device 1 according to Exemplary Embodiment 1 of the present invention. [Figure 2] It is a flowchart showing the flow of an object detection method S1 according to Exemplary Embodiment 1 of the present invention. [Figure 3] It is a diagram showing an example of the processing of a cluster generation unit 14 according to Exemplary Embodiment 1 of the present invention. [Figure 4] It is a block diagram showing the configuration of an object detection device 1A according to Exemplary Embodiment 2 of the present invention. [[ID=二十五]] [Figure 5] It is a flowchart showing the flow of an object detection method S1A according to Exemplary Embodiment 2 of the present invention. [Figure 6] It is a block diagram showing the configuration of an object detection device 1B according to Exemplary Embodiment 3 of the present invention. [Figure 7] It is a flowchart showing the flow of an object detection method S1B according to Exemplary Embodiment 3 of the present invention. [Figure 8] It is a block diagram showing the configuration of an object detection program according to Exemplary Embodiment 4 of the present invention.

Modes for Carrying Out the Invention

[0012] 〔Exemplary Embodiment 1〕 The first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for the exemplary embodiments described later.

[0013] (Overview of Object Detection Device 1) The object detection device 1 is a device for detecting an object from a three-dimensional point cloud indicating at least one object.

[0014] In order to detect an object, hierarchical feature amounts of a three-dimensional point cloud are extracted centering on a seed point selected from the input three-dimensional point cloud, the center of the object is estimated based on the extracted feature amounts, the seed points are selected and clustered based on the estimated center of the object, and for each cluster, a technique for estimating the type of the object, the three-dimensional circumscribed rectangle, and the objectness is disclosed. For the selection of the seed points, the FPS (Farthest Point Sampling) method is used so that the distribution of the selected seed points (hereinafter also referred to as representative points) does not deviate. However, in the selection of the seed points using the FPS method, there is a problem that false detection (selection of seed points unrelated to the object) and non-detection (failure to select seed points from an object with a small number of points) occur.

[0015] The object detection device 1 includes means for executing a process of selecting, as representative points, seed points having a high objectness (hereinafter also referred to as "object accuracy") among the seed points selected from the three-dimensional point cloud. By providing means for executing a process of selecting representative points based on the object accuracy, false detection and non-detection can be prevented.

[0016] (Configuration of the object detection device 1) The configuration of the object detection device 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the object detection device 1.

[0017] As shown in FIG. 1, the object detection device 1 includes a first calculation unit 11, a second calculation unit 12, a third calculation unit 13, and a cluster generation unit 14. The first calculation unit 11, the second calculation unit 12, the third calculation unit 13, and the cluster generation unit 14 are configurations that respectively realize the first calculation means, the second calculation means, the third calculation means, and the cluster generation means in this exemplary embodiment.

[0018] The first calculation unit 11 is configured to perform a first calculation process that selects multiple seed points from a three-dimensional point cloud representing at least one object and calculates a first feature quantity, which is a feature quantity of each seed point. The first feature quantity is calculated by referring to the coordinate values ​​of the seed points. The first calculation unit 11 outputs the coordinates of the seed points and the first feature quantity to the second calculation unit 12. It also outputs the coordinates of the seed points to the cluster generation unit 14.

[0019] The second calculation unit 12 is configured to perform a second calculation process for each seed point to calculate a second feature, which is the feature of that seed point. The second feature of each seed point can be calculated by referring to the first feature of that seed point. The second calculation unit 12 outputs the second feature to the third calculation unit 13.

[0020] The third calculation unit 13 is configured to perform a third calculation process for each seed point to calculate the object accuracy, which represents the object-likeness of that seed point. The object accuracy of each seed point can be calculated by referring to the second feature quantity of that seed point. The third calculation unit 13 outputs the object accuracy to the cluster generation unit 14.

[0021] The cluster generation unit 14 is configured to select at least one representative point from a plurality of seed points and generate clusters based on the representative point. The cluster generation unit 14 selects a provisional representative point, re-selects each representative point by referring to the neighborhood region of each provisional representative point, generates a cluster consisting of seed points for each representative point, and connects two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold. At least one provisional representative point is selected from the plurality of seed points. Each representative point is selected as the seed point with the highest object accuracy among the seed points included in the neighborhood region of each provisional representative point. The cluster generated for each representative point consists of seed points included in the neighborhood region of that representative point.

[0022] The first calculation unit 11, the second calculation unit 12, the third calculation unit 13, and the cluster generation unit 14 described above may be implemented, for example, by a neural network.

[0023] (Flow of object detection method S1) The flow of the object detection method S1 according to this exemplary embodiment will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the object detection method S1. As shown in Figure 2, the object detection method S1 includes a first calculation process S11, a second calculation process S12, a third calculation process S13, and a cluster generation process S14.

[0024] The first calculation process S11 selects multiple seed points from a three-dimensional point cloud representing at least one object and calculates a first feature quantity, which is a feature quantity of each seed point. The first feature quantity is calculated by referring to the coordinate values ​​of the seed point.

[0025] The second calculation process S12 calculates a second feature, which is the feature of that seed point, for each seed point. The second feature of each seed point can be calculated by referring to the first feature of that seed point.

[0026] The third calculation process S13 calculates the object probability for each seed point, which represents the object-likeness of that seed point. The object probability for each seed point can be calculated by referring to the second feature quantity of that seed point.

[0027] The cluster generation process S14 selects at least one representative point from a plurality of seed points and generates clusters based on the representative point. The cluster generation process S14 selects a provisional representative point, re-selects each representative point by referring to the neighborhood region of each provisional representative point, generates a cluster consisting of seed points for each representative point, and connects two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold. At least one provisional representative point is selected from the plurality of seed points. Each representative point is selected as the seed point with the highest object accuracy among the seed points included in the neighborhood region of each provisional representative point. The cluster generated for each representative point consists of seed points included in the neighborhood region of that representative point.

[0028] (Effects of object detection device 1 and object detection method S1) As described above, the object detection device 1 and object detection method S1 according to this exemplary embodiment employ a configuration in which the object accuracy of the seed point is referenced when selecting a representative point. With this configuration, by selecting a seed point with high object accuracy as the representative point, false detection or failure to detect in the selection of the seed point can be prevented. Therefore, the object detection device 1 and object detection method S1 according to this exemplary embodiment have the effect of improving object detection accuracy.

[0029] (Specific example of the first calculation unit 11) For calculating the first feature for each seed point, in addition to the coordinate values ​​of that seed point, color information (RGB values), normal vectors, curvature, etc., may also be referenced.

[0030] The first calculation process performed in the first calculation unit 11 may be a process that selects seed points from a three-dimensional point cloud, generates local clusters whose elements are points included in local regions among the points constituting the three-dimensional point cloud, and performs a first convolution to calculate a first feature as the feature of each local cluster. Multiple points are selected as seed points from the three-dimensional point cloud input to the first calculation unit 11. Local clusters are generated by determining a local region centered on each of the multiple seed points and using points from the three-dimensional point cloud included in that local region as elements. The first convolution process takes the coordinate values ​​of the points constituting the local cluster as input for each local cluster and calculates the feature of the local cluster as the first feature of the seed points belonging to that local cluster. This configuration is preferable because it can calculate a feature as the first feature that aggregates the feature of points other than the seed points that are included in the local region.

[0031] (Specific example of the second calculation unit 12) The second calculation unit 12 may further calculate the center position of the object to which each seed point belongs. The second calculation process performed by the second calculation unit 12 may be a second convolution process that takes a first feature quantity of each seed point as input and outputs the center position of the object indicated by that seed point and a second feature quantity of that seed point.

[0032] (Specific example of the third calculation unit 13) The third calculation unit 13 may further calculate a third feature for each seed point, which is a feature of that seed point. The third calculation process performed in the third calculation unit 13 may be a third convolution process that takes the second feature of each seed point as input and outputs the object accuracy of that seed point and the third feature of that seed point. The object accuracy may be a numerical value between 0 and 1.

[0033] (Specific example of cluster generation unit 14) Temporary representative points can be selected using FPS (Farthest Point Sampling). For example, select an arbitrary point p1 from multiple seed points, then select the point p2 that is furthest from the selected point {p1}. Next, select the point p3 that is furthest from the selected points {p1,p2}. Continue selecting the point furthest from the already selected points, and repeat this process until a predetermined number of points have been selected. Here, the distance metric can be, for example, the feature distance between the first, second, or third feature vectors.

[0034] The neighborhood region of a provisional representative point may be any region that includes the provisional representative point. The size of the neighborhood region of a provisional representative point may be a predetermined fixed size. If the third calculation unit 13 calculates a third feature and the cluster generation unit 14 is implemented by a neural network, the size of the neighborhood region of a provisional representative point may be a size determined based on the third feature of the provisional representative point.

[0035] The neighborhood region of a representative point may be any region that includes the representative point. The size of the neighborhood region of a representative point may be a predetermined fixed size. If the third calculation unit 13 calculates a third feature and the cluster generation unit 14 is implemented by a neural network, the size of the neighborhood region of a representative point may be a size determined based on the third feature of the representative point.

[0036] An example of the processing performed by the cluster generation unit 14 will be explained with reference to Figure 3. Figure 3 is a schematic diagram showing an example of the processing from a point cloud consisting of seed points to determining representative point A and its neighboring region.

[0037] The point cloud in the upper left of Figure 3 consists of seed points input to the cluster generation unit 14. In this point cloud, the points indicated by black circles are seed points, and the region X indicated by the rectangle corresponds to the region of object X. In the point cloud in the lower left, the points indicated by white circles are provisional representative points selected from the seed points by the cluster generation unit 14. The provisional representative point A' will be explained below. The region indicated by the dashed circle is the neighborhood region of provisional representative point A'. In the cluster generation unit 14, in the upper right of Figure 3, the representative point A with the highest object accuracy is re-selected from the neighborhood region of provisional representative point A'. Then, the seed points included in the neighborhood region of representative point A, indicated by the dashed line, are generated as a cluster of representative point A.

[0038] In the neighborhood of provisional representative point A', there was only one seed point included in the region X corresponding to the object, but in the neighborhood of representative point A, there are three seed points included in the region X corresponding to the object. In this way, by re-selecting representative point A, which has a high object accuracy, it is possible to cluster seed points with a high object accuracy in the neighborhood of representative point A.

[0039] (Modified version of the first calculation unit 11) The object detection device 1 may include a first calculation unit 11' instead of the first calculation unit 11.

[0040] The first calculation unit 11' is configured to perform a first calculation process that includes a selection process for selecting multiple points from a three-dimensional point cloud, a local cluster generation process for generating local clusters for each of the multiple points, and a first convolution process for calculating the feature quantities of each local cluster. Furthermore, the first calculation unit 11' repeatedly executes the cycle of processes included in the first calculation process, using a three-dimensional point cloud composed of each of the multiple points, with the feature quantities of the local clusters containing each of the multiple points as new input.

[0041] A local cluster is a cluster generated based on local regions that encompass each of several points. The size of the local region is determined by a fixed value or by a convolution of the coordinate values ​​of the points. For each local region created from multiple seed points, a cluster is created whose elements are the points included in that local region from among the points that make up the three-dimensional point cloud, and this is called a local cluster. The features of the local cluster are calculated by a first convolution operation using the coordinate values ​​that make up the local cluster as input.

[0042] (Modified version of object detection device 1) The object detection device 1 may include a removal means for removing points assigned to the background region from the three-dimensional point cloud. The background region refers to an asymmetric region not used for object detection, such as the ground and walls. For example, points corresponding to the background region may be estimated from the three-dimensional point cloud by segmentation. Furthermore, points corresponding to the background region may be removed from the three-dimensional point cloud by referring to the segmentation results of different modalities, such as images.

[0043] [Exemplary Embodiment 2] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in Exemplary Embodiment 1 will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0044] (Configuration of object detection device 1A) The configuration of the object detection device 1A according to this embodiment will be described with reference to Figure 4. Figure 4 is a block diagram showing the configuration of the object detection device 1A.

[0045] As shown in Figure 4, the object detection device 1A comprises a first calculation unit 11, a second calculation unit 12, a third calculation unit 13A, a cluster generation unit 14, and a classification unit 15. The first calculation unit 11, the second calculation unit 12, and the cluster generation unit 14 of the object detection device 1A have the same configuration as the first calculation unit 11, the second calculation unit 12, and the cluster generation unit 14 of the object detection device 1. In this embodiment, the third calculation unit 13A and the classification unit 15 are configured to realize the third calculation means and the classification means, respectively.

[0046] The third calculation unit 13A is configured to perform a third calculation process for each seed point, which further calculates a third feature quantity that is a feature quantity of that point. The third feature quantity for each seed point is calculated by referring to the second feature quantity of that seed point along with the object accuracy. The third feature quantity calculated by the third calculation unit 13A is output to the classification unit 15.

[0047] The classification unit 15 is configured to classify each cluster generated by the cluster generation unit 14 into a class corresponding to an object. The classification unit 15 performs the classification of each cluster by referring to the features of that cluster. The features of each cluster are calculated by referring to the third features of the seed points included in that cluster. The calculation process for the features of each cluster may be a convolution process that takes the third features of the seed points included in that cluster as input and outputs the features of that cluster.

[0048] In this embodiment, the classification unit 15 calculates the corresponding object score y for each object (each class) i. i Calculate the object score y. i It is defined, for example, as a probability value normalized by the softmax function such that its sum is 1, as in equation (1).

[0049]

number

[0050] Such a classification unit 15 may be implemented by a neural network, and in particular, by an MLP (Multi-layer perceptron) network.

[0051] (Flowchart of object detection method S1A) The flow of the object detection method S1A according to this exemplary embodiment will be explained with reference to Figure 5. Figure 5 is a flowchart showing the flow of the object detection method S1A. As shown in Figure 5, the object detection method S1A includes a first calculation process S11, a second calculation process S12, a third calculation process S13A, a cluster generation process S14, and a classification process S15. The first calculation process S11, the second calculation process S12, and the cluster generation process S14 of the object detection method S1A have the same configuration as the first calculation process S11, the second calculation process S12, and the cluster generation process S14 of the object detection method S1.

[0052] The third calculation process S13A further calculates a third feature for each seed point, which is a feature of that point. The third feature for each seed point is calculated by referring to the second feature of that seed point along with the object accuracy of that seed point.

[0053] The classification process S15 classifies the class to which each cluster generated by the cluster generation process S14 belongs. The classification of each cluster's class is performed by referring to the cluster's features. The features of each cluster are calculated by referring to the third feature of the seed point included in that cluster.

[0054] (Effect of object detection device 1A) As described above, the object detection device 1A according to this exemplary embodiment employs a configuration that classifies objects from cluster features. Therefore, in addition to the effects of the object detection device 1 according to exemplary embodiment 1, the object detection device 1A according to this exemplary embodiment can also be made to classify objects with high accuracy.

[0055] [Exemplary Embodiment 3] A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in Exemplary Embodiment 1 will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0056] (Configuration of object detection device 1B) The configuration of the object detection device 1B according to this embodiment will be described with reference to Figure 6. Figure 6 is a block diagram showing the configuration of the object detection device 1B.

[0057] As shown in Figure 6, the object detection device 1B includes a first calculation unit 11, a second calculation unit 12A, a third calculation unit 13A, a cluster generation unit 14, a classification unit 15, and an update unit 16. The first calculation unit 11 and cluster generation unit 14 of the object detection device 1B have the same configuration as the first calculation unit 11 and cluster generation unit 14 of the object detection device 1, and the third calculation unit 13A and classification unit 15 have the same configuration as the third calculation unit 13A and classification unit 15 of the object detection device 1A. In this embodiment, the third calculation unit 13A and the update unit 16 are configured to realize the third calculation means and the update means, respectively.

[0058] The second calculation unit 12A is configured to perform a second calculation process for each seed point, which further calculates the center position of the object to which that seed point belongs. The center position of the object to which each seed point belongs is calculated by referring to the first feature of that seed point along with the second feature, which is the feature of that seed point. The center position of the object calculated by the second calculation unit 12A is output to the update unit 16.

[0059] The update unit 16 is configured to perform an update process that updates the parameters defining the first calculation process, the parameters defining the second calculation process, and the parameters defining the third calculation process to update the first loss, the second loss, and the third loss. To calculate each loss, the update unit 16 refers to the object's center position calculated by the second calculation unit 12A, the object's accuracy calculated by the third calculation unit 13A, and the classification result calculated by the classification unit 15. The update is performed to minimize the sum of the first loss, the second loss, and the third loss.

[0060] The first loss is the sum of the first deviations, which represent the degree of discrepancy between the center position of the object to which each seed point belongs and the correct answer identified from the training data. The first loss calculation process calculates the first deviations and then calculates the first loss, which is the sum of the first deviations. The training data for the center position of the object to which each seed point belongs can be pre-calculated using any method.

[0061] The second loss is the sum of the second deviation values, which represent the difference between the object accuracy of each seed point and the correct answer identified from the training data. The second loss calculation process calculates the second deviation values ​​and then calculates the second loss, which is the sum of the second deviation values. The training data for the object accuracy of each seed point (training data that shows whether each seed point belongs to an object or not) can be pre-calculated.

[0062] The third loss is the sum of the third deviation values, which represent the degree of deviation between the classification result of each seed point in the classification unit 15 and the correct answer identified from the training data. The third loss calculation process calculates the third deviation values ​​and calculates the third loss, which is the sum of the third deviation values. The correct answer identified from the training data may be a one-hot vector where the probability value of each seed point being in the correct answer class to which it belongs is 1, and the probability value of being any other object is 0.

[0063] The first loss, the second loss, and the third loss may be calculated using the following formulas.

number

[0064]

Number

[0065] When calculating the first loss, in equations (2) and (3), L is the first loss, and y i is the center position of the object to which each seed point belongs, and t i is the correct answer specified from the teacher data. When calculating the second loss, in equations (2) and (3), L is the second loss, and y i is the object accuracy of each seed point, and t i is the correct answer specified from the teacher data. When calculating the third loss, in equations (2) and (3), L is the third loss, and y i is the result of the classification of each seed point, and t i is the correct answer specified from the teacher data.

[0066] (Flow of object detection method S1B) The flow of the object detection method S1B according to this exemplary embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the flow of the object detection method S1B. As shown in FIG. 7, the object detection method S1B includes a first calculation process S11, a second calculation process S12A, a third calculation process S13A, a cluster generation process S14, a classification process S15, and an update process S16. The first calculation process S11 and the cluster generation process S14 of the object detection method S1B are the same processes as the first calculation process S11 and the cluster generation process S14 of the object detection method S1, and the third calculation process S13A and the classification process S15 are the same processes as the third calculation process S13A and the classification process S15 of the object detection method S1A.

[0067] The second calculation process S12A executes a second calculation process for further calculating the center position of the object to which each seed point belongs for each seed point. The center position of the object to which each seed point belongs is calculated by referring to the first feature amount of the seed point together with the second feature amount which is the feature amount of the seed point.

[0068] The update process S16 updates the parameters defining the first calculation process, the parameters defining the second calculation process, and the parameters defining the third calculation process based on the first loss, the second loss, and the third loss. To calculate each loss, the update unit 16 refers to the object's center position calculated by the second calculation unit 12, the object's accuracy calculated by the third calculation unit 13, and the classification result calculated by the classification unit 15. The update is performed to minimize the sum of the first loss, the second loss, and the third loss.

[0069] The first loss is the sum of the first deviations, which represent the degree of deviation between the center position of the object to which each seed point belongs and the correct answer identified from the training data. The first loss calculation process calculates the first deviations and calculates the first loss, which is the sum of the first deviations.

[0070] The second loss is the sum of the second deviation values, which represent the difference between the object accuracy of each seed point and the correct answer identified from the training data. The second loss calculation process calculates the second deviation values ​​and then calculates the second loss, which is the sum of the second deviation values.

[0071] The third loss is the sum of the third deviations, which represent the degree of deviation between the classification result of each seed point and the correct answer identified from the training data. The third loss calculation process calculates the third deviation and then calculates the third loss, which is the sum of the third deviations.

[0072] (Effect of object detection device 1B) As described above, the object detection device 1B according to this exemplary embodiment employs a configuration in which the parameters defining the first calculation process, the parameters defining the second calculation process, and the parameters defining the third calculation process are updated based on the center position of the object to which each calculated seed point belongs, the second calculation process, and the classification result. Therefore, the object detection device 1B according to this exemplary embodiment provides the effect of being able to perform object detection with higher accuracy, in addition to the effects of the object detection device 1 according to exemplary embodiment 1.

[0073] [Examples of implementation using software] Some or all of the functions of the object detection devices 1, 1A, and 1B may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0074] In the latter case, the object detection devices 1, 1A, and 1B are implemented, for example, by a computer that executes instructions for a program, which is software that implements each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 8. Computer C comprises at least one processor C1 and at least one memory C2. The memory C2 stores a program P that causes computer C to operate as object detection devices 1, 1A, and 1B. In computer C, the processor C1 reads program P from memory C2 and executes it, thereby implementing each function of object detection devices 1, 1A, and 1B.

[0075] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0076] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0077] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0078] [Additional Note 1] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the embodiments described above are also included in the technical scope of the present invention.

[0079] [Additional Note 2] Some or all of the embodiments described above may also be described as follows. However, the present invention is not limited to the embodiments described below. (Note 1) An object detection device comprising: a first calculation means that selects a plurality of seed points from a three-dimensional point cloud representing at least one object and performs a first calculation process to calculate a first feature quantity, which is a feature quantity of each seed point, by referring to the coordinate values ​​of the seed point; a second calculation means that performs a second calculation process for each seed point to calculate a second feature quantity, which is a feature quantity of the seed point, by referring to the first feature quantity of the seed point; a third calculation means that performs a third calculation process for each seed point to calculate an object accuracy, which represents the object-likeness of the seed point, by referring to the second feature quantity of the seed point; and a cluster generation means that selects at least one provisional representative point from the plurality of seed points, re-selects the seed point with the highest object accuracy among the seed points included in the neighborhood region of the provisional representative point as the representative point, generates a cluster consisting of seed points included in the neighborhood region of the representative point for each representative point, and connects two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold.

[0080] With the above configuration, object detection can be performed with high accuracy from a three-dimensional point cloud.

[0081] (Note 2) The object detection device according to Appendix 1, further comprising: the third calculation process further calculates a third feature quantity for each seed point, which is the feature quantity of that seed point; the object detection device further comprises a classification means for calculating the feature quantity of each cluster for each cluster by referring to the third feature quantity of the seed points included in that cluster, and classifying the class to which each cluster belongs by referring to the feature quantity of that cluster.

[0082] With the above configuration, objects can be classified with high accuracy.

[0083] (Note 3) The object detection device according to Appendix 1 or 2, wherein, instead of the first calculation means, a first calculation means is provided that performs a first calculation process which includes: a selection process for selecting the plurality of seed points from the three-dimensional point cloud; a local cluster generation process for determining a local region encompassing each of the plurality of seed points and generating a local cluster whose elements are the points included in that local region among the points constituting the three-dimensional point cloud; and a first convolution process for each local cluster which takes the coordinate values ​​of the points constituting the local cluster as input and outputs the feature quantities of the local cluster as the first feature quantities, wherein the first calculation means is provided that takes a three-dimensional point cloud composed of each plurality of points, in which the feature quantities of the local clusters containing each plurality of points are the feature quantities of the plurality of points, repeatedly executes the cycle of the processes included in the first calculation process, and in the final cycle uses the plurality of points selected as seed points and calculates the feature quantities of the seed points as the first feature quantities.

[0084] According to the above configuration, since points obtained by hierarchically aggregating the features of local regions of a three-dimensional point cloud are used as seed points for object detection, object detection can be performed with higher accuracy.

[0085] (Note 4) The object detection device according to Appendix 2 or 3, wherein the second calculation process further calculates the center position of the object to which each seed point belongs for each seed point, the second calculation process is a second convolution process that takes the first feature quantity of each seed point as input and outputs the center position of the object to which the seed point belongs and the second feature quantity of the seed point, and the third calculation process is a third convolution process that takes the second feature quantity of each seed point as input and outputs the object accuracy of the seed point and the third feature quantity of the seed point.

[0086] According to the above configuration, the second and third calculation processes can be performed efficiently and with high accuracy.

[0087] (Note 5) A first loss calculation process calculates a first loss, which is the sum of the first deviations, from a first deviation representing the deviation between the center position of the object to which each seed point belongs and the correct answer identified from the training data; a second loss calculation process calculates a second loss, which is the sum of the second deviations, from a second deviation representing the deviation between the object accuracy of each seed point and the correct answer identified from the training data; and the results of the classification of each seed point in the classification means and the identified correct answer identified from the training data. An object detection device according to any of Appendix 4, further comprising: a third loss calculation process that calculates a third loss representing the sum of the third deviations from a third deviation representing the degree of deviation from the correct answer; and an update means that performs an update process that updates the parameters defining the first calculation process, the parameters defining the second calculation process, and the parameters defining the third calculation process so as to minimize the sum of the first loss, the second loss, and the third loss.

[0088] With the above configuration, parameter updates can be performed, enabling more accurate object detection.

[0089] (Note 6) The object detection device according to any one of the appendices 1 to 5, further comprising a removal means for removing points assigned to a background region from the three-dimensional point cloud.

[0090] With the above configuration, points in the background area that do not correspond to an object can be removed, enabling more efficient and accurate object detection.

[0091] (Note 7) An object detection method comprising: selecting multiple seed points from a three-dimensional point cloud representing at least one object; performing a first calculation process to calculate a first feature quantity, which is a feature quantity of each seed point, by referring to the coordinate values ​​of the seed point; performing a second calculation process for each seed point to calculate a second feature quantity, which is a feature quantity of the seed point, by referring to the first feature quantity of the seed point; performing a third calculation process for each seed point to calculate an object accuracy, which represents the object-likeness of the seed point, by referring to the second feature quantity of the seed point; selecting at least one provisional representative point from the multiple seed points; for each provisional representative point, re-selecting the seed point with the highest object accuracy among the seed points included in the neighborhood region of the provisional representative point as the representative point; generating a cluster for each representative point consisting of seed points included in the neighborhood region of the representative point; and concatenating two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold.

[0092] With the above configuration, object detection can be performed with high accuracy from a three-dimensional point cloud.

[0093] (Note 8) An object detection program for causing a computer to function as an object detection device, comprising: a first calculation means that performs a first calculation process in which at least one processor of the computer selects a plurality of seed points from a three-dimensional point cloud representing at least one object, and calculates a first feature quantity which is a feature quantity of each seed point by referring to the coordinate value of the seed point; a second calculation means that performs a second calculation process for each seed point, which is a feature quantity of the seed point, by referring to the first feature quantity of the seed point; and for each seed point, an object accuracy that represents the object-likeness of the seed point. An object detection program for causing a computer to function as: a third calculation means that performs a third calculation process calculated by referring to the second feature quantity of the seed point; a cluster generation means that selects at least one provisional representative point from the plurality of seed points, re-selects as the representative point the seed point with the highest object accuracy among the seed points included in the neighborhood of the provisional representative point for each provisional representative point, generates a cluster consisting of seed points included in the neighborhood of the representative point for each representative point, and connects two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold.

[0094] With the above configuration, object detection can be performed with high accuracy from a three-dimensional point cloud.

[0095] [Additional Note 3] Some or all of the embodiments described above can also be expressed as follows:

[0096] An object detection device comprising at least one processor, the processor performing: a first calculation process of selecting a plurality of seed points from a three-dimensional point cloud representing at least one object and calculating a first feature quantity, which is a feature quantity of each seed point, by referring to the coordinate values ​​of the seed point; a second calculation process of calculating a second feature quantity, which is a feature quantity of each seed point, by referring to the first feature quantity of the seed point; a third calculation process of calculating an object accuracy, which represents the object-likeness of each seed point, by referring to the second feature quantity of the seed point; and a cluster generation process of selecting at least one provisional representative point from the plurality of seed points, re-selecting the seed point with the highest object accuracy among the seed points included in the neighborhood region of the provisional representative point as the representative point, generating a cluster consisting of seed points included in the neighborhood region of the representative point for each representative point, and linking two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold.

[0097] Furthermore, this object detection device may also be equipped with memory, which may store a program that causes the processor to execute the first calculation process, the second calculation process, the third calculation process, and the cluster generation process. This program may also be recorded on a computer-readable, non-temporary, tangible recording medium. [Explanation of Symbols]

[0098] 1, 1A, 1B ···Object detection device 11 ···First Calculation Department 12,12A...Second calculation section 13,13A...Third calculation section 14. Cluster Generation Unit 15...Classification section 16...Update section

Claims

1. A first calculation means performs a first calculation process that selects multiple seed points from a three-dimensional point cloud representing at least one object, and calculates a first feature quantity, which is a feature quantity of each seed point, by referring to the coordinate values ​​of the seed point. A second calculation means performs a second calculation process for each seed point, in which a second feature quantity, which is a feature quantity of that seed point, is calculated by referring to the first feature quantity of that seed point. A third calculation means performs a third calculation process to calculate an object accuracy representing the object-likeness of each seed point by referring to the second feature quantity of that seed point, The cluster generation means includes: selecting at least one provisional representative point from the plurality of seed points; re-selecting the seed point with the highest object accuracy among the seed points included in the neighboring region of each provisional representative point as the representative point; generating a cluster consisting of seed points included in the neighboring region of each representative point; and connecting two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold. Object detection device.

2. The third calculation process further calculates a third feature, which is a feature of the seed point, for each seed point. The object detection device further includes a classification means that calculates the feature quantity of each cluster by referring to the third feature quantity of the seed point included in the cluster, and classifies the class to which each cluster belongs by referring to the feature quantity of the cluster. The object detection device according to claim 1.

3. The first calculation means is, The first calculation process includes a selection process to select multiple points from the three-dimensional point cloud, a local cluster generation process to determine a local region encompassing each of the multiple points and generate local clusters whose elements are the points included in that local region among the points constituting the three-dimensional point cloud, and a first convolution process for each local cluster that takes the coordinate values ​​of the points constituting the local cluster as input and outputs the feature quantities of the local cluster as the first feature quantities. A three-dimensional point cloud composed of the aforementioned plurality of points, wherein the feature quantities of local clusters including the plurality of points are used as the feature quantities of the plurality of points, is used as a new input, and the cycle of processing included in the first calculation process is repeatedly executed, the plurality of points selected in the final cycle are used as seed points, and the first calculation process is executed in which the feature quantities of the seed points are calculated as the first feature quantities. The object detection device according to claim 1.

4. The second calculation process further calculates the center position of the object to which each seed point belongs, The second calculation process is a second convolution process that takes the first feature quantity of each seed point as input and outputs the center position of the object to which the seed point belongs, and the second feature quantity of the seed point. The third calculation process takes the second feature quantity of each seed point as input, This is a third convolution process that outputs the object accuracy of the seed point and the third feature quantity of the seed point. The object detection device according to claim 2.

5. A first loss calculation process calculates a first loss, which is the sum of the first deviations, from a first deviation that represents the degree of deviation between the center position of the object to which each seed point belongs and the correct answer identified from the training data. A second loss calculation process calculates a second loss, which is the sum of the second deviation degrees, from a second deviation degree representing the degree of deviation between the object accuracy of each seed point and the correct answer identified from the training data. A third loss calculation process calculates a third loss, which is the sum of the third degree of deviation, from a third degree of deviation representing the degree of deviation between the classification result of each seed point in the classification means and the correct answer identified from the training data. The system further includes update means for performing an update process that updates the parameters defining the first calculation process, the parameters defining the second calculation process, and the parameters defining the third calculation process in order to minimize the sum of the first loss, the second loss, and the third loss. The object detection device according to claim 4.

6. The system further includes a removal means for removing points assigned to the background region from the aforementioned three-dimensional point cloud. The object detection device according to claim 1.

7. The first calculation process involves selecting multiple seed points from a three-dimensional point cloud representing at least one object, and calculating a first feature quantity, which is a feature quantity of each seed point, by referring to the coordinate values ​​of that seed point. For each seed point, a second calculation process is performed to calculate a second feature, which is a feature of that seed point, by referring to the first feature of that seed point. For each seed point, a third calculation process is performed to calculate the object accuracy representing the object-likeness of that seed point by referring to the second feature quantity of that seed point. The process includes selecting at least one provisional representative point from the aforementioned plurality of seed points, re-selecting the seed point with the highest object accuracy among the seed points included in the neighboring region of that provisional representative point as the representative point, generating a cluster consisting of seed points included in the neighboring region of that representative point for each representative point, and connecting two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is less than or equal to a threshold, Object detection method.

8. An object detection program for causing a computer to function as an object detection device, wherein at least one processor provided by the computer, A first calculation means performs a first calculation process that selects multiple seed points from a three-dimensional point cloud representing at least one object, and calculates a first feature quantity, which is a feature quantity of each seed point, by referring to the coordinate values ​​of the seed point. A second calculation means performs a second calculation process for each seed point, in which a second feature quantity, which is a feature quantity of that seed point, is calculated by referring to the first feature quantity of that seed point. A third calculation means performs a third calculation process to calculate an object accuracy representing the object-likeness of each seed point by referring to the second feature quantity of that seed point, A cluster generation means for a computer to function as follows: Selecting at least one provisional representative point from the aforementioned plurality of seed points; for each provisional representative point, re-selecting the seed point with the highest object accuracy from among the seed points included in the neighboring region of that provisional representative point as the representative point; for each representative point, generating a cluster consisting of seed points included in the neighboring region of that representative point; and connecting two clusters where the distance between the cluster centers to which the representative point belongs, or the distance between the representative points, is below a threshold. Object detection program.

Images