Information processing method, information processing device, and program

Abstract

The present invention makes it possible to effectively increase the density of a three-dimensional point cloud.　A plurality of clusters are generated by clustering three-dimensional point clouds acquired on the basis of the phase difference, or the time difference between the transmission time and the reception time, of a transmission signal and a reflection signal corresponding to each of a plurality of discretely arranged basic transmission points among a plurality of transmission points arranged in the form of a two-dimensional array. For each of the plurality of clusters, geometric information in a local space formed by the three-dimensional point clouds included in the cluster is calculated. For each of the plurality of clusters, a determination is made, on the basis of the geometric information, regarding whether to perform a density increase involving increasing the density of the three-dimensional point clouds included in the cluster by using another transmission point in addition to the basic transmission points corresponding to each point of the three-dimensional point clouds.

Description

Information processing method, information processing device, and program

[0001] The present technology relates to an information processing method, an information processing device, and a program, and more particularly to an information processing method that can be applied to a distance measuring system.

[0002] In recent years, so-called mobile robots, which can perform simple transportation tasks, have been increasingly introduced in logistics warehouses, factories, and other locations. Mobile robots are equipped with ranging systems and acquire environmental information based on three-dimensional point clouds obtained by the ranging systems.

[0003] When a mobile robot moves, for example, when running, acquiring information about the moving surface as environmental information is important for optimal path planning. In this case, by acquiring high-density information about not only nearby moving surfaces but also distant moving surfaces, the mobile robot can move smoothly with little acceleration or deceleration. In particular, for mobile robots that move at high speeds, this allows for smooth selection of the shortest route without slowing down. Furthermore, selecting the shortest route can reduce the operating time of the sensing system.

[0004] A known ranging system is a direct Time of Flight (dToF) sensor-based ranging system with multiple light-emitting points arranged in a two-dimensional array on a light-emitting unit. In this case, the higher the density of the light-emitting points, the higher the density of the resulting three-dimensional point cloud, resulting in higher resolution of the acquired environmental information. However, the higher the density of the light-emitting points, the greater the power consumption required for lighting and driving the sensor circuitry, making it difficult for a mobile robot equipped with a battery to operate for long periods of time.

[0005] For example, Patent Document 1 discloses a technique for detecting the edges (boundaries in the depth direction) of an object from a sparse depth image using a DNN or the like, and densifying the area near the edge.

[0006] International Publication No. 2022 / 168500

[0007] The purpose of this technology is to enable the density of a three-dimensional point cloud to be effectively increased.

[0008] The concept of this technology lies in an information processing method having: a cluster generation procedure for generating multiple clusters by clustering a three-dimensional point cloud acquired based on the time difference between the transmission time and the reception time or the phase difference between the transmission signal and the reflected signal corresponding to each of a plurality of basic transmission points that are discretely arranged among a plurality of transmission points arranged in a two-dimensional array; a geometric information calculation procedure for calculating, for each of the plurality of clusters, geometric information in a local space formed by the three-dimensional point cloud included in the cluster; and a high-density image determination procedure for determining, for each of the plurality of clusters, based on the geometric information, whether to increase the density of the three-dimensional point cloud by using other transmission points as well as the basic transmission points corresponding to each point of the three-dimensional point cloud included in the cluster.

[0009] This technology includes a cluster generation procedure, a geometric information calculation procedure, and a densification determination procedure. In the cluster generation procedure, a three-dimensional point cloud acquired based on the time difference or phase difference between the transmission time and the reception time of a transmission signal and a reflection signal corresponding to each of a plurality of basic transmission points that are discretely arranged among a plurality of transmission points arranged in a two-dimensional array is clustered to generate a plurality of clusters.

[0010] In the geometric information calculation step, for each of a plurality of clusters, geometric information in a local space formed by a group of three-dimensional points included in the cluster is calculated. For example, the geometric information may include at least one of cluster size information indicating the length of the main axis direction of the local space, cluster variance information corresponding to the thickness of the local space, and cluster normal direction information indicating the thickness direction of the local space.

[0011] In the densification determination procedure, for each of a plurality of clusters, it is determined based on geometric information whether or not to perform densification, which involves increasing the density of the 3D point cloud by using basic transmission points corresponding to each point of the 3D point cloud included in the cluster as well as other transmission points. In this technology, it is determined which locations in the 3D point cloud should use high-density information, and the process for this is called densification. Note that this process may also be called not only densification but also high-resolution, super-resolution, high-definition, etc. However, this process does not process low-resolution sensor information to high resolution, and is different from the technology generally referred to as high-resolution.

[0012] For example, the densification determination procedure may determine whether to perform densification based on information about the density of the 3D point cloud included in the local space and information about the thickness of the local space. In this case, for example, the densification determination procedure may determine to perform densification when a cluster size indicating the length of the main axis direction of the local space is equal to or greater than a first threshold and a cluster variance corresponding to the thickness of the local space is equal to or greater than a second threshold. This makes it possible to increase the density of the 3D point cloud near the boundary between different objects in the distance where the density of the 3D point cloud becomes sparse, thereby making it possible to correctly grasp the shape of the object near the boundary.

[0013] Furthermore, for example, the densification determination procedure may determine whether to perform densification based on information about the density of the 3D point cloud included in the local space and information about the thickness direction of the local space. In this case, for example, the densification determination procedure may determine to perform densification if the cluster size, which indicates the length of the local space in the principal axis direction, is equal to or greater than a first threshold and the deviation of the cluster normal direction, which indicates the thickness direction of the local space, from the vertical direction is within a second threshold. This makes it possible to increase the density of the 3D point cloud in a distant area where the 3D point cloud is sparse, for example, on a moving surface where a mobile robot moves, and thereby make it possible to correctly grasp the situation on the moving surface.

[0014] Furthermore, for example, the transmission point may be an emitting point that outputs an optical pulse signal as a transmission signal, and each point in the three-dimensional point cloud may be acquired based on the time difference of a first peak in histogram data of the time difference between the transmission times of multiple transmitted optical pulse signals and the reception times of reflected optical pulse signals, which is obtained corresponding to the output of multiple optical pulse signals.

[0015] In this case, for example, the densification determination procedure may determine whether to perform densification based on information on the density of the 3D point cloud included in the local space and histogram data corresponding to the points constituting the 3D point cloud included in the local space. In this case, for example, the densification determination procedure may determine to perform densification when a cluster size indicating the length of the main axis direction of the local space is equal to or greater than a first threshold, and the difference between the score value of the first peak and the score value of at least the second peak in the histogram data is equal to or less than a second threshold. This makes it possible to increase the density of the 3D point cloud in a distant area where the density of the 3D point cloud is sparse and is uneven at a fine level, thereby enabling the shape of this uneven area to be accurately grasped.

[0016] In this way, in this technology, a three-dimensional point cloud acquired based on transmission signals from a plurality of basic transmission points that are discretely arranged among a plurality of transmission points arranged in a two-dimensional array is clustered to generate a plurality of clusters, and for each of the plurality of clusters, geometric information in the local space formed by the three-dimensional point clouds included in the cluster is calculated. For each of the plurality of clusters, it is determined based on the geometric information whether or not to increase the density of the three-dimensional point cloud by using other transmission points in addition to the basic transmission points that correspond to each point of the three-dimensional point cloud included in the cluster. This technology only requires using the necessary other transmission points along with the basic transmission points, thereby reducing power consumption and making it possible to effectively increase the density of the three-dimensional point cloud.

[0017] Note that the present technology may further include a transmission point selection procedure for selecting other transmission points when performing densification, based on the positions of multiple basic transmission points corresponding to a three-dimensional point group included in a cluster. This makes it possible to appropriately select other transmission points to be used together with the basic transmission points.

[0018] In this case, for example, in the transmission point selection procedure, a transmission point inside a convex hull that includes multiple basic transmission points that correspond to the three-dimensional point cloud included in the cluster may be selected as the other transmission point. Also, in this case, for example, in the transmission point selection procedure, a transmission point inside an ellipse that includes multiple basic transmission points that correspond to the three-dimensional point cloud included in the cluster may be selected as the other transmission point. Also, in this case, for example, in the transmission point selection procedure, transmission points located around each of the multiple basic transmission points that correspond to the three-dimensional point cloud included in the cluster may be selected as the other transmission point.

[0019] Furthermore, the present technology may further include a transmission strength selection procedure for selecting the transmission strength of other transmission points when performing densification, based on information on a cluster normal direction that indicates the thickness direction of a local space formed by a three-dimensional point cloud included in the cluster. This makes it possible to appropriately select the transmission strength of other transmission points used together with the basic transmission point, thereby reducing power consumption. In this case, for example, the transmission strength selection procedure may select the transmission strength of the other transmission points so that the larger the deviation of the cluster normal direction from the direction of the multiple transmission points arranged in a two-dimensional array, the stronger the transmission strength of the other transmission points.

[0020] Furthermore, the present technology may further include a transmission point proportion selection procedure for selecting a proportion of other transmission points when performing densification based on information on the density of a three-dimensional point cloud included in a cluster. This makes it possible to appropriately select the proportion of other transmission points to be used together with the basic transmission points, thereby reducing power consumption. In this case, for example, the transmission point proportion selection procedure may select a larger proportion of other transmission points as the density of the three-dimensional point cloud included in the cluster decreases.

[0021] Furthermore, the present technology may further include a basic transmission point position change procedure for dynamically changing the positions of multiple basic transmission points among the multiple transmission points at a predetermined period, thereby reducing the load on specific transmitting elements and extending the life of the transmitter, and therefore the ranging system.

[0022] Another concept of the present technology lies in an information processing device including: a cluster generation unit that generates a plurality of clusters by clustering a three-dimensional point cloud acquired based on the time difference between the transmission time and the reception time of a transmission signal and a reflection signal corresponding to each of a plurality of basic transmission points that are discretely arranged among a plurality of transmission points arranged in a two-dimensional array; a geometric information calculation unit that calculates, for each of the plurality of clusters, geometric information in a local space formed by the three-dimensional point cloud included in the cluster; and a densification determination unit that determines, for each of the plurality of clusters, based on the geometric information, whether to perform densification to increase the density of the three-dimensional point cloud by using other transmission points as well as basic transmission points corresponding to each point of the three-dimensional point cloud included in the cluster.

[0023] Another concept of the present technology resides in a program for causing a computer to execute an information processing method, the program comprising: a cluster generation procedure for generating a plurality of clusters by clustering a three-dimensional point cloud acquired based on the time difference between the transmission time and the reception time of a transmission signal and a reflection signal corresponding to each of a plurality of basic transmission points that are discretely arranged among a plurality of transmission points arranged in a two-dimensional array; a geometric information calculation procedure for calculating, for each of the plurality of clusters, geometric information in a local space formed by the three-dimensional point cloud included in the cluster; and a densification determination procedure for determining, for each of the plurality of clusters, based on the geometric information, whether to perform densification to increase the density of the three-dimensional point cloud by using other transmission points as well as basic transmission points corresponding to each point of the three-dimensional point cloud included in the cluster.

[0024] 1 is a block diagram showing an example of the configuration of a robot control system according to an embodiment; FIG. 2 is a block diagram showing an example of the configuration of a ranging system; FIG. 3 is a diagram showing an example of the distribution of each point of a three-dimensional point cloud corresponding to basic laser light-emitting elements (basic light-emitting points) before clustering, and a state in which a plurality of clusters have been generated by clustering (grouping) neighboring point clouds in three-dimensional space; FIG. 4 is a diagram showing a schematic representation of the correspondence between a local space formed by three-dimensional point clouds included in a cluster and geometric information; FIG. 5 is a diagram showing an example of the distribution of each point of a three-dimensional point cloud existing in three-dimensional space corresponding to basic laser light-emitting elements (basic light-emitting points), along with a schematic representation of the change in density of the three-dimensional point cloud in the depth direction; FIG. 6 is a diagram showing an example of the distribution of each point of a three-dimensional point cloud corresponding to basic laser light-emitting elements (basic light-emitting points) existing in three-dimensional space; FIG. 7 is a diagram for explaining densification of a three-dimensional point cloud in an area where the geometric shape of an object (subject) is uneven at a finer level relative to the granularity of each point of the three-dimensional point cloud; FIG. 8 is a diagram showing a schematic representation of a densification processing sequence in a ranging system; FIG. 9 is a diagram for explaining an example of a method for selecting high-density light-emitting points; and FIG. 10 is a diagram for explaining another example of a method for selecting high-density light-emitting points. 1 is a diagram for explaining the selection of a proportion of densified light-emitting points; FIG. 1 is a diagram for explaining the selection of a proportion of densified light-emitting points; FIG. 2 is a diagram for explaining the deviation angle of the cluster normal direction from the distance sensor direction (light-emitting unit direction); FIG. 3 is a diagram for explaining the selection of the light-emitting intensity of densified light-emitting points; FIG. 4 is a diagram for explaining determining whether densification is necessary and deciding to emit a densification request signal based on the moving speed of a robot (mobile robot); FIG. 5 is a diagram for explaining control of dynamically changing the position of a basic light-emitting point (basic laser light-emitting element) at a predetermined period; FIG. 6 is a flowchart showing an example of the processing procedure of a ranging system; FIG. 7 is a block diagram showing an example of the hardware configuration of a computer; and FIG. 8 is a diagram showing an example of the distribution of each point of a three-dimensional point cloud existing in a three-dimensional space in a case where only basic laser light-emitting elements (basic light-emitting points) are subjected to light emission control, and in a case where, in addition to light emission control of basic laser light-emitting elements (basic light-emitting points), light emission control is also performed on other laser light-emitting elements (densification light-emitting points) for clusters determined to be subjected to densification.This figure shows an example of a case where only the basic laser light-emitting element (basic light-emitting point) is subjected to light emission control, and an example of a case where light emission control is performed on the basic laser light-emitting element (basic light-emitting point) as well as other laser light-emitting elements (high-density light-emitting points) for a cluster determined to be subjected to high density.

[0025] Hereinafter, modes for carrying out the invention (hereinafter referred to as "embodiments") will be described. The description will be given in the following order: 1. Embodiment 1-1. Example of configuration of robot control system 1-2. Example of configuration of distance measurement system 1-3. Example of processing procedure of distance measurement system 1-4. Processing by software 2. Modified examples

[0026] 1 shows an example of the configuration of a robot control system 10 according to an embodiment. The robot control system 10 includes a robot control unit 11, a ranging system 12, a SLAM (Simultaneous Localization and Mapping) execution unit 13, a path planning unit 14, and a movement control unit 15.

[0027] The robot control unit 11 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The robot control unit 11 controls all or part of the functions of the robot control system 10.

[0028] The ranging system 12 is a direct time-of-flight (dToF) sensor-based ranging system that outputs a three-dimensional point cloud at a frame cycle. Each point in the three-dimensional point cloud has relative three-dimensional position information from the ranging system 12. The SLAM execution unit 13 estimates the robot's self-position and creates an environmental map based on the three-dimensional point cloud output from the ranging system 12.

[0029] The path planning unit 14 plans the robot's behavior based on the information on its own position and the environmental map from the SLAM execution unit 13. The movement control unit 15 controls the movement of the robot by controlling the robot's movement mechanism based on the behavior plan by the path planning unit 14.

[0030] 2 shows an example of the configuration of the ranging system 12. The ranging system 12 has an overall control unit 101, a light emitting unit 102, a light emitting control unit 103, a light receiving unit 104, a light receiving control unit 105, a histogram point group processing unit 106, a basal light emitting point data extraction unit 107, a basal light emitting point data processing unit 108, and a densification determination processing unit 109.

[0031] The overall control unit 101 controls the operation of each unit of the ranging system 12. The light emitting unit 102 has a plurality of laser light emitting elements, such as vertical cavity surface emitting lasers (VCSELs), arranged in a two-dimensional array (two-dimensional lattice pattern). In this embodiment, the plurality of laser light emitting elements are arranged in a square lattice layout, but they may also be arranged in other layouts, such as a triangular lattice. The light emitting unit 102 uses the plurality of laser light emitting elements arranged in a two-dimensional array to diffusely irradiate a plurality of light emission pulse signals onto an object (subject) to be measured for ranging. Here, each of the plurality of laser light emitting elements arranged in a two-dimensional array constitutes a light emitting point.

[0032] The light emission control unit 103 controls the light emission from the plurality of laser light-emitting elements of the light-emitting unit 102, i.e., the output of light emission pulse signals, based on the control from the overall control unit 101. This control includes, for example, control to emit light in each frame not only from the plurality of basic laser light-emitting elements that are discretely arranged among the plurality of laser light-emitting elements arranged in a two-dimensional array, but also from other laser light-emitting elements that are used for increasing density.

[0033] The light receiving unit 104 has a plurality of light receiving elements arranged in a two-dimensional array (two-dimensional lattice) that receive reflected light pulse signals that are reflected back from an object (subject) in response to light emission pulse signals output from a plurality of laser light emitting elements arranged in a two-dimensional array in the light emitting unit 102 described above.

[0034] The light-receiving control unit 105 controls the operation of the light-receiving unit 104 based on control from the overall control unit 101. This control includes control to place only the light-receiving elements corresponding to the laser light-emitting elements that are outputting light-emitting pulse signals, among the multiple laser light-emitting elements of the light-emitting unit 102, in a light-receiving state. The light-receiving control unit 105 also supplies the overall control unit 101 with reception output signals of the reflected light pulse signals at each light-receiving element of the light-receiving unit 104.

[0035] The overall control unit 101 generates a digital signal indicating the time difference between the transmission time of the light emission pulse signal and the reception time of the reflected light pulse signal, associated with each of the multiple laser light-emitting elements emitting light in the light-emitting unit 102, and supplies the generated signal to the histogram point group processing unit 106.

[0036] The histogram point group generation processing unit 106 generates a histogram of the time differences indicated by the digital signal supplied from the overall control unit 101 for each frame, with a bin width corresponding to the time resolution of the time differences, in association with each of the multiple laser light-emitting elements emitting light from the light-emitting unit 102. Here, the bin width is the width of each frequency unit constituting the histogram.

[0037] Furthermore, the histogram point group processing unit 106 calculates the distance to the object (subject) for each frame based on the time difference between the first peak values ​​of the histogram in association with each of the multiple laser light-emitting elements emitted by the light-emitting unit 102. This allows the distance to the reflection point in the direction of the light emission pulse signals emitted from each of the multiple laser light-emitting elements emitted by the light-emitting unit 102 to be obtained.

[0038] Then, for each frame, the histogram point group processing unit 106 generates a three-dimensional point group (point cloud) showing the distribution of each reflection point in the coordinate system of the ranging system 12 based on the distance to the reflection point in the direction of the light emission pulse signals emitted from each of the multiple laser light-emitting elements emitting light in the light-emitting unit 102.

[0039] The basic light-emitting point data extraction unit 107 extracts, from the three-dimensional point cloud generated by the histogram point cloud processing unit 106, a three-dimensional point cloud corresponding to a plurality of basic laser light-emitting elements (basic light-emitting points) that are discretely arranged among a plurality of laser light-emitting elements (light-emitting points) arranged in a two-dimensional array, as basic light-emitting point data.

[0040] The basal light-emitting point data processing unit 108 generates multiple clusters by clustering the three-dimensional point cloud extracted by the basal light-emitting point data extraction unit 107 with neighboring point clouds in three-dimensional space. In this case, the clustering method is not limited to a specific one, as long as the number of points in the point cloud included in each cluster in three-dimensional space is somewhat equal. Possible clustering methods include, for example, Octmap, K-d tree, and K-nearest neighbor method.

[0041] In this technology, as will be described later, the density of the 3D point cloud in a sparse point cloud region is increased. The reason for making the number of points of the point cloud contained in each cluster equal is that the denser the region, the smaller the cluster size, and the sparser the region, the larger the cluster size, making it possible to determine the density of the 3D point cloud within the cluster from the cluster size. Note that, if the density of the point cloud within a cluster is not determined from the cluster size, the number of points of the point cloud contained in the cluster does not necessarily have to be equal.

[0042] Furthermore, the basal light-emitting point data processing unit 108 calculates, for each of the multiple clusters, geometric information in the local space formed by the 3D point clouds included in the cluster. The geometric information includes cluster size information indicating the length in the principal axis direction of the local space. This cluster size information is information for determining the density of the 3D point cloud within the cluster, and if the density of the point cloud within the cluster is not determined from the cluster size, this cluster size information is not necessarily required.

[0043] The geometric information also includes cluster variance information corresponding to the thickness of the local space formed by the 3D point clouds included in the cluster. If the 3D point clouds included in the cluster form a flat surface such as a floor or a wall, the value of this cluster variance will be small. On the other hand, if the 3D point clouds included in the cluster form an uneven object or an area where objects of different shapes intersect, the positions of each point in the 3D point cloud included in the cluster will vary in the thickness direction, and the value of this cluster variance will be large. This cluster variance information is information for determining the thickness of the local space formed by the 3D point clouds included in the cluster, and if the thickness of this local space is not determined from the cluster variance, this cluster variance information is not necessarily required.

[0044] The geometric information also includes information on the cluster normal direction, which indicates the thickness direction of the local space formed by the three-dimensional point group included in the cluster.

[0045] 3A shows an example of the distribution of points in a three-dimensional point cloud corresponding to basic laser light-emitting elements (basic light-emitting points) before clustering. FIG. 3B shows a state in which multiple clusters, six in this example, have been generated by clustering (grouping) neighboring points in three-dimensional space. As described above, for each cluster generated in this way, geometric information (cluster size information, cluster dispersion information, and cluster normal direction information) in the local space (shown as an ellipse in the figure) formed by the three-dimensional point clouds included in the cluster is calculated.

[0046] 4(a) and (b) schematically show the correspondence between the local space (shown as a cylindrical space in the figure) formed by the 3D point clouds included in the cluster and the geometric information. FIG. 4(a) shows an example in which the 3D point clouds included in the cluster form a flat surface such as a floor or wall. In this case, the thickness of the local space is small, and the cluster variance is a small value. FIG. 4(b) shows an example in which the 3D point clouds included in the cluster form a region where, for example, uneven objects or objects of different shapes intersect. In this case, the thickness of the local space is large, and the cluster variance is a large value.

[0047] Returning to Figure 2, the densification determination processing unit 109 determines, for each of the multiple clusters, based on the geometric information calculated by the basic light-emitting point data processing unit 108, whether or not to perform densification, which increases the density of the three-dimensional point cloud, by using not only the basic laser light-emitting element (basic light-emitting point) corresponding to each point of the three-dimensional point cloud included in the cluster, but also other laser light-emitting elements (light-emitting points).

[0048] FIG. 5 shows an example of the distribution of each point of a three-dimensional point cloud in three-dimensional space corresponding to a basic laser light-emitting element (basic light-emitting point), along with a dashed line a showing an outline of the change in density of the three-dimensional point cloud in the depth direction. As shown in the figure, the density of the three-dimensional point cloud decreases the further back, i.e., the farther away. To observe the environment accurately, it is possible to determine that all distant clusters where the density of the three-dimensional point cloud is sparse should be densified. However, in this case, the number of laser light-emitting elements that are activated increases, which is disadvantageous in that it increases power consumption.

[0049] FIG. 6 shows an example of the distribution of points in a three-dimensional point cloud corresponding to basic laser light-emitting elements (basic light-emitting points) existing in three-dimensional space. For example, in FIG. 6 , the area near the boundary between different objects, shown surrounded by a solid circular frame Sa, requires a high density of the three-dimensional point cloud in order to grasp the shape of the object. Also, for example, in FIG. 6 , the area of ​​the moving surface on which the robot moves, shown surrounded by a solid circular frame Sb, requires a high density of the three-dimensional point cloud in order to accurately grasp information about the moving surface. On the other hand, for example, in FIG. 6 , the area of ​​the wall surface, shown surrounded by a dashed circular frame Sc, does not require a high density of the three-dimensional point cloud.

[0050] In this embodiment, the densification determination processing unit 109 determines to perform densification only on clusters with low 3D point cloud density that require increasing the density of the 3D point cloud. The densification determination processing unit 109 determines to perform densification on clusters with low 3D point cloud density that may form regions where the 3D point cloud is not coplanar. The densification determination processing unit 109 makes this determination based on information on the density of the 3D point cloud contained in the local space of the cluster and information on the thickness of that local space. In this case, the densification determination processing unit 109 determines to perform densification on a cluster if the cluster size, which indicates the length of the cluster's local space in the principal axis direction, is equal to or greater than a first threshold, and the cluster variance, which corresponds to the thickness of the local space, is equal to or greater than a second threshold.

[0051] The densification determination processing unit 109 also determines to perform densification on clusters that have a low density of three-dimensional point clouds and that may constitute a moving surface (traveling surface). The densification determination processing unit 109 makes this determination based on information on the density of the three-dimensional point cloud contained in the local space of the cluster and information on the thickness direction of the local space. In this case, the densification determination processing unit 109 determines to perform densification on a cluster if the cluster size, which indicates the length of the main axis direction of the local space of the cluster, is equal to or greater than a first threshold, and the deviation of the cluster normal direction, which indicates the thickness direction of the local space, from the vertical direction is within a second threshold.

[0052] 7A shows an example of the distribution of each point of a three-dimensional point cloud corresponding to a basic laser light-emitting element (basic light-emitting point) existing in three-dimensional space. For example, as shown in FIG. 7B, which is surrounded by a solid-line circular frame Sd in FIG. 7A, in an area where the geometric shape of the object (subject) is uneven at a finer level relative to the granularity of each point of the three-dimensional point cloud, it is necessary to densify the three-dimensional point cloud in order to grasp the shape of the object. However, as described above, when determining whether to densify a cluster using information on the thickness of the local space of the cluster, such as information on the cluster dispersion, depending on the position of each point of the three-dimensional point cloud included in the cluster, the cluster dispersion may be small, and it may be determined that densification is not to be performed on the cluster.

[0053] In this case, by using the histogram data generated by the histogram point group processing unit 106 as described above, it is possible to determine whether the cluster corresponds to an area where the geometric shape of the object (subject) is uneven at a finer level, and by using this determination result, it is possible to determine that densification should be performed on the cluster.

[0054] That is, in the histogram data corresponding to each point of the 3D point cloud contained in the cluster corresponding to an area where the geometric shape of the object (subject) is uneven at a finer level, the score values ​​of the second peak, the third peak, etc. are close to the score value of the first peak, as shown in Figure 7(c), due to the influence of multipath caused by the uneven shape.

[0055] As a result, if the score value of the second peak in the histogram data is close to the score value of the first peak, it is possible that the cluster corresponds to an area where the geometric shape of the object (subject) is uneven at a finer level, and it is possible to determine that the cluster should be densified even if the cluster variance is small.

[0056] In this case, the densification determination processing unit 109 determines whether to perform densification based on information on the density of the 3D point cloud included in the local space of the cluster and histogram data corresponding to the points that make up the 3D point cloud included in the local space. In this case, the densification determination processing unit 109 determines to perform densification for the cluster if the cluster size, which indicates the length of the local space of the cluster in the direction of the major axis, is equal to or greater than a first threshold, and the difference between the score value of the first peak and the score value of at least the second peak in the histogram data is equal to or less than a second threshold.

[0057] The densification determination processing unit 109 supplies the densification determination result for each of the plurality of clusters for each frame to the overall control unit 101. Based on the densification determination result for each of the plurality of clusters supplied from the densification determination processing unit 109 in a certain frame, the overall control unit 101 controls the light emission from the plurality of laser light-emitting elements of the light-emitting unit 102 in the next frame via the light-emission control unit 103, and also controls the light-receiving state of the plurality of light-receiving elements of the light-receiving unit 104 in the next frame to a state that matches the light-emitting state of the light-emitting unit 102 via the light-receiving control unit 105.

[0058] In this case, for a cluster determined to undergo densification, the overall control unit 101 controls so that other laser light-emitting elements (densification light-emitting points) as well as basic laser light-emitting elements (basic light-emitting points) corresponding to each point of the three-dimensional point cloud included in the cluster are emitted. Also, for a cluster determined not to undergo densification, the overall control unit 101 controls so that only basic laser light-emitting elements (basic light-emitting points) corresponding to each point of the three-dimensional point cloud included in the cluster are emitted.

[0059] 8 is a schematic diagram showing a densification processing sequence in the ranging system 12. In this case, densification light-emitting points (other laser light-emitting elements used for densification) in a next frame are determined based on a three-dimensional point cloud corresponding to basic light-emitting points (basic laser light-emitting elements) in a certain frame, and in the next frame, the basic light-emitting points and the densification light-emitting points determined in the previous frame are emitted to increase the density of the three-dimensional point cloud.

[0060] When the frame rate is high, the amount of change in the sensor position in the next frame is small. Therefore, it is possible to increase the density of the 3D point cloud in the target area by emitting light from the high-density light-emitting points determined in one frame in the next frame. Note that it is also possible to increase the density of the 3D point cloud in the target area more accurately by calculating the amount of correction from the amount of change in the sensor position in the past few frames and correcting the position of the high-density light-emitting points.

[0061] As described above, for a cluster determined to undergo densification, the overall control unit 101 controls so that densification light-emitting points (other laser light-emitting elements used for densification) as well as basic light-emitting points (basic laser light-emitting elements) corresponding to each point of the three-dimensional point cloud included in the cluster are illuminated. In this case, the overall control unit 101 selects the densification light-emitting points based on the positions of the basic light-emitting points. Various selection methods are possible in this case. First to third methods are described below, but the present invention is not limited to these methods.

[0062] The first method will be described. In this first method, the overall control unit 101 selects, as densification light-emitting points, light-emitting points (laser light-emitting elements) inside a convex hull formed by the positions of multiple basic light-emitting points corresponding to the three-dimensional point cloud included in the cluster. Here, the convex hull refers to the smallest convex polygon (convex polyhedron) that contains all of the given points. FIG. 9( a) shows an example of light-emitting points (basic laser light-emitting elements) arranged in a two-dimensional array on the light-emitting unit 102. Here, circles represent light-emitting points, and among them, white circles and double white circles represent basic light-emitting points, and the double white circles represent basic light-emitting points corresponding to each point of the three-dimensional point cloud included in the cluster determined to undergo densification.

[0063] 9(b) shows the densified light-emitting points selected by the first method as hatched circles when the basic light-emitting points (double white circles) corresponding to each point of the three-dimensional point cloud included in the cluster determined to be densified are located as shown in FIG. 9(a). In this case, the light-emitting points inside the convex hull (shown by a constant chain line) formed by the positions of the basic light-emitting points (double white circles) corresponding to each point of the three-dimensional point cloud included in the cluster are selected as the densified light-emitting points (hatched circles).

[0064] The second method will be described below. In this second method, the overall control unit 101 selects, as high-density light-emitting points, light-emitting points (laser light-emitting elements) within an ellipse (including a circle) that includes all of the basic light-emitting points corresponding to the three-dimensional point group included in the cluster.

[0065] 10(a) shows the densification light-emitting points selected by the second method as hatched circles when the basic light-emitting points (double white circles) corresponding to each point of the three-dimensional point cloud included in the cluster determined to be densified are located as shown in Fig. 9(a). In this case, the light-emitting points inside an ellipse (shown by a constant chain line) that includes all of the basic light-emitting points (double white circles) corresponding to each point of the three-dimensional point cloud included in the cluster are selected as the densification light-emitting points (hatched circles).

[0066] A third method will now be described. In this third method, the overall control unit 101 selects, as high-density light-emitting points, light-emitting points (laser light-emitting elements) located around each of a plurality of basic light-emitting points corresponding to the three-dimensional point group included in the cluster.

[0067] 10(b) shows the densification light-emitting points selected by the third method as hatched circles when the basic light-emitting points (double white circles) corresponding to each point of the three-dimensional point cloud included in the cluster determined to be densified are located as shown in FIG. 9(a). In this case, the light-emitting points located around each of the basic light-emitting points (double white circles) corresponding to each point of the three-dimensional point cloud included in the cluster are selected as the densification light-emitting points (hatched circles). In the illustrated example, eight light-emitting points located around the basic light-emitting point (double white circle) are selected as the densification light-emitting points (hatched circles), but the number is not limited to eight.

[0068] Furthermore, as described above, for a cluster determined to undergo densification, the overall control unit 101 selects densified light-emitting points based on the positions of basic light-emitting points corresponding to each point of the three-dimensional point cloud included in the cluster, but selects the proportion of densified light-emitting points based on information on the density of the three-dimensional point cloud included in the cluster to suppress unnecessary light emission and reduce power consumption. In this case, the overall control unit 101 selects a higher proportion of densified light-emitting points as the density of the three-dimensional point cloud included in the cluster decreases. For example, the overall control unit 101 acquires information on point spacing as information on the density of the three-dimensional point cloud included in the cluster, and sets the proportion of densified light-emitting points based on the point spacing information.

[0069] 11 shows that the distance between two basic light-emitting points and the corresponding two points in the three-dimensional point cloud changes depending on the distance of the object (subject) from the distance measuring sensor. That is, when the distance between the points at a short distance is X, the distance between the points at a long distance is X' (>X).

[0070] In this case, when the point spacing is X', the sparseness level of the three-dimensional point cloud is high, so it is possible to increase the proportion of densified light-emitting points to increase the density of the three-dimensional point cloud, but on the other hand, when the point spacing is X, the sparseness level of the three-dimensional point cloud is not that high, so it is possible to reduce the proportion of densified light-emitting points to suppress power consumption. In the illustrated example, when the point spacing is X' and the sparseness level of the three-dimensional point cloud is high, both of the two light-emitting points between the two basic light-emitting points are selected as densified light-emitting points, and when the point spacing is X and the sparseness level of the three-dimensional point cloud is not that high, only one of the two light-emitting points between the two basic light-emitting points is selected as a densified light-emitting point.

[0071] In the illustrated example, the ratio of high density light emitting points is changed in two stages, but it may be changed in three or more stages.

[0072] 12(a) shows, with hatched circles, densification light-emitting points that are selected when the sparseness level of the three-dimensional point cloud included in the cluster is high, in the case where the basic light-emitting points (double white circles) corresponding to each point of the three-dimensional point cloud included in the cluster determined to be densified are located as shown in FIG. 9(a). In the example shown, all of the light-emitting points (laser light-emitting elements) inside an ellipse (shown with a constant chain line) that includes all of the multiple basic light-emitting points corresponding to the three-dimensional point cloud included in the cluster are selected as densification light-emitting points.

[0073] 12(b) shows, with hatched circles, densification light-emitting points selected when the sparseness level of the three-dimensional point cloud included in the cluster is not very high, in the case where the basic light-emitting points (double white circles) corresponding to each point of the three-dimensional point cloud included in the cluster determined to be densified are located as shown in FIG. 9(a). In the example shown, only some of the light-emitting points (laser light-emitting elements) inside an ellipse (shown with a constant chain line) that includes all of the multiple basic light-emitting points corresponding to the three-dimensional point cloud included in the cluster are selected as densification light-emitting points.

[0074] Note that Figures 12(a) and (b) show an example in which high-density light-emitting points are selected by the second method (see Figure 10(a)). However, although not shown, in examples in which high-density light-emitting points are selected by the first method (see Figure 9(b)) or the third method (see Figure 10(b)), it is possible to similarly suppress unnecessary light emission and reduce power consumption by selecting the proportion of high-density light-emitting points based on information on the density of the three-dimensional point cloud included in the cluster.

[0075] Furthermore, as described above, for clusters determined to undergo densification, the overall control unit 101 selects densification light-emitting points based on the positions of the basic light-emitting points corresponding to each point of the three-dimensional point cloud included in the cluster, but selects the light-emitting intensity of the densification light-emitting points based on information on the cluster normal direction, which indicates the thickness direction of the local space formed by the three-dimensional point cloud included in the cluster, thereby suppressing light emission at a light intensity higher than necessary and reducing power consumption.

[0076] 13(a) shows a case where the deviation angle of the cluster normal direction from the distance sensor direction (light-emitting unit direction) is θ, and Fig. 13(b) shows a case where the deviation angle of the cluster normal direction from the distance sensor direction (light-emitting unit direction) is θ' (< θ). The smaller the deviation angle of the cluster normal direction from the distance sensor direction (light-emitting unit direction), the higher the reflectivity of the light-emitting pulse signal from the light-emitting point (laser light-emitting element) of the light-emitting unit 102 in the distance sensor direction tends to be.

[0077] That is, when the angle of deviation of the cluster normal direction from the distance sensor direction (light-emitting unit direction) is small, even if the light emission intensity is reduced, there is a possibility that strong light will return as a reflected light pulse signal. Therefore, it is conceivable that the smaller the angle of deviation of the cluster normal direction from the distance sensor direction (light-emitting unit direction), the lower the light emission intensity of the high-density light-emitting points will be selected to suppress power consumption.

[0078] Fig. 14(a) shows a case where the deviation angle of the cluster normal direction from the distance sensor direction (light-emitting unit direction) is θ (see Fig. 13(a)), and the light-emission intensity of the high-density light-emitting points is set to a large P1. Fig. 14(b) shows a case where the deviation angle of the cluster normal direction from the distance sensor direction (light-emitting unit direction) is θ' (see Fig. 13(b)), and the light-emission intensity of the high-density light-emitting points is set to a small P2 (<P1).

[0079] Note that Figures 14(a) and (b) show an example in which high-density light-emitting points are selected by the second method (see Figure 10(a)). However, although not shown, in examples in which high-density light-emitting points are selected by the first method (see Figure 9(b)) or the third method (see Figure 10(b)), similarly, by selecting the light emission intensity of the high-density light-emitting points based on information on the cluster normal direction that indicates the thickness direction of the local space formed by the three-dimensional point group included in the cluster, it is possible to suppress light emission at a light emission intensity higher than necessary and thereby suppress power consumption.

[0080] Moreover, the overall control unit 101 may select either the proportion of high density light emitting points or the light emission intensity of the high density light emitting points, or may perform both of them.

[0081] 2, the overall control unit 101 controls the operation of acquiring a three-dimensional point cloud in the ranging system 12 based on a three-dimensional point cloud acquisition request signal S1 from the robot control system 10 (see FIG. 1). That is, when the robot control system 10 supplies the three-dimensional point cloud acquisition request signal S1 and requests acquisition of a three-dimensional point cloud, the overall control unit 101 controls the operation of each unit to acquire a three-dimensional point cloud.

[0082] In addition, the overall control unit 101 controls the operation of the basal light-emitting point data extraction unit 107, the basal light-emitting point data processing unit 108, and the densification determination processing unit 109 based on a densification request signal S2 from the robot control system 10 (see Figure 1).

[0083] In other words, when the robot control system 10 supplies a densification request signal S2 to the overall control unit 101 requesting densification, the overall control unit 101 operates the basic light-emitting point data extraction unit 107, the basic light-emitting point data processing unit 108, and the densification judgment processing unit 109 to obtain the judgment result of the densification cluster for each frame, and controls for densification in the following frame based on the judgment result.

[0084] In this case, for a cluster determined to undergo densification, control is performed so that not only the basic light-emitting points (basic laser light-emitting elements) corresponding to each point of the three-dimensional point cloud included in the cluster but also the densification light-emitting points (other laser light-emitting elements used for densification) emit light. Also, for a cluster determined not to undergo densification, control is performed so that only the basic light-emitting points (basic laser light-emitting elements) corresponding to each point of the three-dimensional point cloud included in the cluster emit light.

[0085] Furthermore, when the robot control system 10 does not supply a densification request signal S2 and densification is not requested, the overall control unit 101 does not operate the basal light-emitting point data extraction unit 107, the basal light-emitting point data processing unit 108, and the densification determination processing unit 109. In this case, control is performed so that only the basal light-emitting point (basic laser light-emitting element) emits light in each frame.

[0086] For example, the robot control system 10 determines whether densification is necessary based on the movement speed of the robot (mobile robot) and decides to issue a densification request signal S2. Note that the decision to issue the densification request signal S2 in the robot control system 10 is not limited to being based on the movement speed of the robot (mobile robot). For example, the robot control system 10 may decide to issue a densification request signal S2 based on a setting operation by the user.

[0087] For example, as shown in Figure 15(a), when the robot's movement speed is low, the range over which the robot can move in a certain period of time is narrow, and it is determined that it is sufficient to obtain a 3D point cloud with a high density only in the vicinity using only the light emitted from the basic light-emitting points (basic laser light-emitting elements), so no densification request signal S2 is issued from the robot control system 10, and densification is not performed by the ranging system 12. Also, for example, when the robot's movement speed is high, as shown in Figure 15(b), the range over which the robot can move in a certain period of time is wide, and it is determined that it is insufficient to obtain a 3D point cloud with a high density only in the vicinity using only the light emitted from the basic light-emitting points (basic laser light-emitting elements), so a densification request signal S2 is issued from the robot control system 10, and densification is performed by the ranging system 12.

[0088] 2, the overall control unit 101 may also perform control so that the position of the basic light-emitting point (basic laser light-emitting element) is dynamically changed at a predetermined period, such as once per frame or at a fixed number of light-emitting periods, as shown in Fig. 16. This reduces the load on a specific light-emitting point (basic laser light-emitting element), thereby enabling the light-emitting unit 102, and therefore the ranging system 12, to have a longer life.

[0089] 1-3. Example of Processing Procedure of Distance Measuring System The flowchart of FIG. 17 shows an example of processing procedure of the distance measuring system 12 shown in FIG.

[0090] First, the ranging system 12 starts processing in step ST1. In this case, the processing is started based on the start of supply of a 3D point cloud acquisition request signal S1 from the robot control unit 11 (see FIG. 1) of the robot control system 10 to the overall control unit 101, for example.

[0091] Next, in step ST2, the distance measuring system 12, under the control of the overall control unit 101, controls the light emission control unit 103 to emit light from the plurality of laser light emitting elements of the light emitting unit 102, that is, controls the output of light emission pulse signals.

[0092] This light emission control includes control of emitting light from a plurality of discretely arranged basic laser light emitting elements (basic light emitting points) among a plurality of laser light emitting elements arranged in a two-dimensional array, as well as other laser light emitting elements (high density light emitting points) used for densification. Note that in the first frame, light emission control is performed for only the basic light emitting points, and in each subsequent frame, light emission control is performed for the basic light emitting points and, if necessary, the high density light emitting points.

[0093] Next, when the 3D point cloud of the current frame has been acquired, the ranging system 12 determines in step ST4 whether or not there is a request to acquire a 3D point cloud of the next frame in the overall control unit 101. In this case, when a 3D point cloud acquisition request signal S1 is supplied from the robot control unit 11 (see FIG. 1) of the robot control system 10, the overall control unit 101 determines that there is a request to acquire a 3D point cloud of the next frame.

[0094] If the robot control unit 11 (see Figure 1) of the robot control system 10 stops supplying the 3D point cloud acquisition request signal S1 to the overall control unit 101 and it is determined that there is no request to acquire 3D points for the next frame, the ranging system 12 terminates processing in step ST5.

[0095] On the other hand, if the robot control unit 11 (see FIG. 1) of the robot control system 10 continues to supply the 3D point cloud acquisition request signal S1 to the overall control unit 101 and it is determined that there is a request to acquire 3D points for the next frame, the ranging system 12 determines in step ST6 whether there is a densification request in the overall control unit 101. In this case, if there is a densification request signal S2 supplied from the robot control unit 11 (see FIG. 1) of the robot control system 10, the overall control unit 101 determines that there is a densification request.

[0096] If the robot control unit 11 (see FIG. 1) of the robot control system 10 does not supply a densification request signal S2 to the overall control unit 101 and it is determined that there is no densification request, the ranging system 12 decides to perform light emission control of the basal light-emitting points in step ST7, and then returns to the processing of step ST2. In this case, in the light emission control for acquiring the 3D point cloud of the next frame in this step ST2, light emission control of only the basal light-emitting points is performed, and densification is not performed.

[0097] Furthermore, in step ST6, if a high density request signal S2 is supplied from the robot control unit 11 (see Figure 1) of the robot control system 10 to the overall control unit 101 and it is determined that a high density request exists, in step ST8, the ranging system 12 causes the basic light-emitting point data extraction unit 107 to extract, as basic light-emitting point data, a three-dimensional point cloud corresponding to the basic light-emitting point from the three-dimensional point cloud acquired by the histogram point cloud processing unit 106.

[0098] Next, in step ST9, the ranging system 12, in the basic light-emitting point data processing unit 108, clusters the basic light-emitting point data extracted by the basic light-emitting point data extraction unit 107, i.e., the three-dimensional point group corresponding to the basic light-emitting point, with nearby point groups in three-dimensional space to generate multiple clusters.

[0099] Next, in step ST10, the ranging system 12 calculates, in the basic light-emitting point data processing unit 108, geometric information (cluster size information, cluster variance information, cluster normal direction information, etc.) in the local space formed by the three-dimensional point cloud contained in each of the multiple clusters.

[0100] Next, in step ST11, the ranging system 12 determines, for each of the multiple clusters, in the densification determination processing unit 109, based on the geometric information calculated by the basic light-emitting point data processing unit 108, whether or not to perform densification to increase the density of the three-dimensional point cloud in the cluster, using not only the basic laser light-emitting element (basic light-emitting point) corresponding to each point of the three-dimensional point cloud included in the cluster, but also other laser light-emitting elements (densification light-emitting points).

[0101] Next, in step ST12, the overall control unit 101 of the ranging system 12 decides that for clusters that have been determined by the densification determination processing unit 109 to undergo densification, it will perform light emission control of other laser light emitting elements (densification light emitting points) along with the basic laser light emitting elements (basic light emitting points) corresponding to each point of the three-dimensional point cloud included in the cluster, and for clusters that have been determined not to undergo densification, it will determine that it will perform light emission control of the basic laser light emitting elements (basic light emitting points) corresponding to each point of the three-dimensional point cloud included in the cluster.

[0102] After the processing of step ST12, the ranging system 12 returns to the processing of step ST2. In this case, in the light emission control for acquiring the three-dimensional point cloud of the next frame in step ST2, for clusters determined by the densification determination processing unit 109 to be densified, light emission control is performed for both the basic light-emitting points and the densification light-emitting points to perform densification, and for clusters determined by the densification determination processing unit 109 not to be densified, light emission control is performed for only the basic light-emitting points, and densification is not performed.

[0103] 1-4. Processing by Computer Software The processing shown in the flowchart of Fig. 17 can be executed by hardware, but can also be executed by software. When a series of processes is executed by software, the programs that make up the software are installed from a recording medium into a computer that is built into dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0104] 18 is a block diagram showing an example of the hardware configuration of a computer 400. The computer 400 has a CPU 401, a ROM 402, a RAM 403, a bus 404, an input / output interface 405, an input unit 406, an output unit 407, a storage unit 408, a drive 409, a connection port 410, and a communication unit 411. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the computer 400 may further include components other than those shown here.

[0105] The CPU 401 functions as, for example, an arithmetic processing device or a control device, and controls the overall operation or part of the operation of each component based on various programs recorded in the ROM 402 , RAM 403 , storage unit 408 , or removable recording medium 501 .

[0106] The ROM 402 is a means for storing programs to be read into the CPU 401, data to be used for calculations, etc. The RAM 403 temporarily or permanently stores, for example, the programs to be read into the CPU 401 and various parameters that change as appropriate when the programs are executed.

[0107] The CPU 401, ROM 402, and RAM 403 are connected to one another via a bus 404. On the other hand, various components are connected to the bus 404 via an interface 405.

[0108] The input unit 406 may include, for example, a mouse, a keyboard, a touch panel, a button, a switch, a lever, etc. Furthermore, the input unit 406 may also include a remote controller (hereinafter referred to as a remote control) that is capable of transmitting control signals using infrared rays or other radio waves.

[0109] The output unit 407 is a device capable of visually or audibly notifying the user of acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile.

[0110] The storage unit 408 is a device for storing various types of data. For example, the storage unit 408 may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.

[0111] The drive 409 is a device that reads information recorded on a removable recording medium 501 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable recording medium 501 .

[0112] The removable recording medium 501 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable recording medium 501 may also be, for example, an IC card equipped with a contactless IC chip, an electronic device, etc.

[0113] The connection port 410 is a port for connecting an external device 502, such as a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or an optical audio terminal. The external device 502 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0114] The communication unit 411 is a communication device for connecting to the network 503, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0115] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0116] As described above, in the ranging system 12 shown in Figure 2, a three-dimensional point cloud acquired based on light emission pulse signals from a plurality of basic laser light-emitting elements (basic light-emitting points) that are discretely arranged among a plurality of laser light-emitting elements (light-emitting points) arranged in a two-dimensional array is clustered to generate a plurality of clusters, and for each of the plurality of clusters, geometric information in the local space formed by the three-dimensional point clouds included in the cluster is calculated. For each of the plurality of clusters, it is determined based on the geometric information whether or not to perform densification, which increases the density of the three-dimensional point cloud, by using other laser light-emitting elements (light-emitting points) in addition to the basic laser light-emitting element (basic light-emitting point) that corresponds to each point of the three-dimensional point cloud included in the cluster. This means that it is only necessary to use the other necessary laser light-emitting elements (densification light-emitting points) together with the basic laser light-emitting element (basic light-emitting point), which reduces power consumption and makes it possible to effectively increase the density of the three-dimensional point cloud.

[0117] 19(a) shows an example of the distribution of each point of a three-dimensional point cloud existing in a three-dimensional space when only the basic laser light-emitting element (basic light-emitting point) is subjected to light emission control. FIG. 19(b) shows an example of the distribution of each point of a three-dimensional point cloud existing in a three-dimensional space when, in addition to light emission control of the basic laser light-emitting element (basic light-emitting point), other laser light-emitting elements (densification light-emitting points) are also subjected to light emission control for clusters determined to be densified. In this case, for clusters CLT determined to be densified, the density of the point cloud included in the cluster is increased by light emission control of the other laser light-emitting elements (densification light-emitting points).

[0118] 20(a) shows an example of a case where only the basic laser light-emitting element (basic light-emitting point) is subjected to light emission control. FIG. 20(c) shows an example of a case where light emission control is performed on the basic laser light-emitting element (basic light-emitting point) and other laser light-emitting elements (densification light-emitting points) for a cluster CLT determined to be densified (see FIG. 19(b)). By performing light emission control on the basic laser light-emitting element (basic light-emitting point) and other laser light-emitting elements (densification light-emitting points) only for the cluster CLT determined to be densified in this way, power consumption can be reduced compared to when all laser light-emitting elements are subjected to light emission control, as shown in FIG. 20(b).

[0119] 2. Modifications In the above-described embodiment, a dToF sensor-based ranging system is shown as an example of the ranging system 12. The present technology can be similarly applied not only to ranging systems that use optical signals, but also to other ranging systems that use ultrasonic signals or radio signals. Some radio wave-based ranging systems receive the difference between the transmission time of a transmitted signal and the reception time of a reflected signal as a phase difference between the signal waveforms.

[0120] Furthermore, in the above-described embodiment, an example was shown in which the ranging system 12 constitutes the robot control system 10, but the ranging system 12 of the present technology can also be applied to drone control systems, automatic driving control systems, etc.

[0121] In addition, in the above-described embodiment, an example was shown in which the ranging system 12 integrally includes the light emitting unit 102, the light receiving unit 104, and other processing units, but some or all of the other processing units may be provided in an external device, such as a server on the cloud.

[0122] Furthermore, while the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0123] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0124] The present technology can also be configured as follows: (1) An information processing method including: a cluster generation step of generating multiple clusters by clustering a three-dimensional point cloud acquired based on the time difference between the transmission time and the reception time of a transmission signal and a reflection signal corresponding to each of a plurality of basic transmission points discretely arranged among a plurality of transmission points arranged in a two-dimensional array; a geometric information calculation step of calculating, for each of the plurality of clusters, geometric information in a local space formed by the three-dimensional point cloud included in the cluster; and a densification determination step of determining, for each of the plurality of clusters, based on the geometric information, whether to perform densification of the three-dimensional point cloud by using other transmission points as well as basic transmission points corresponding to each point of the three-dimensional point cloud included in the cluster. (2) The information processing method described in (1), wherein the geometric information includes at least one of cluster size information indicating the length in the principal axis direction of the local space, cluster dispersion information corresponding to the thickness of the local space, and cluster normal direction information indicating the thickness direction of the local space. (3) The information processing method according to (1), wherein the densification determination step determines whether to perform the densification based on information on the density of a three-dimensional point cloud included in the local space and information on a thickness of the local space. (4) The information processing method according to (3), wherein the densification determination step determines whether to perform the densification if a cluster size indicating a length in a direction of a major axis of the local space is equal to or greater than a first threshold and a cluster variance corresponding to the thickness of the local space is equal to or greater than a second threshold. (5) The information processing method according to (1), wherein the densification determination step determines whether to perform the densification based on information on the density of a three-dimensional point cloud included in the local space and information on a thickness direction of the local space. (6) The information processing method according to (5), wherein the densification determination step determines whether to perform the densification if a cluster size indicating a length in a direction of a major axis of the local space is equal to or greater than a first threshold and a deviation of a cluster normal direction indicating a thickness direction of the local space from a vertical direction is within a second threshold.(7) The information processing method according to (1), wherein the transmission point is a light-emitting point that outputs an optical pulse signal as the transmission signal, and each point of the three-dimensional point cloud is acquired based on a time difference of a first peak in histogram data of the time differences between the transmission times of a plurality of transmitted optical pulse signals and the reception times of a reflected optical pulse signal, obtained corresponding to the output of a plurality of optical pulse signals. (8) The information processing method according to (7), wherein the densification determination step determines whether to perform the densification based on information on the density of the three-dimensional point cloud included in the local space and the histogram data corresponding to points constituting the three-dimensional point cloud included in the local space. (9) The information processing method according to (8), wherein the densification determination step determines to perform the densification if a cluster size indicating a length in a major axis direction of the local space is equal to or greater than a first threshold and a difference between the score value of a first peak and at least a second peak in the histogram data is equal to or less than a second threshold. (10) The information processing method according to any one of (1) to (9), further comprising a transmission point selection step of selecting the other transmission points when performing the densification based on positions of a plurality of basic transmission points corresponding to a three-dimensional point cloud included in the cluster. (11) The information processing method according to (10), wherein, in the transmission point selection step, transmission points inside a convex hull that includes a plurality of basic transmission points corresponding to the three-dimensional point cloud included in the cluster are selected as the other transmission points. (12) The information processing method according to (10), wherein, in the transmission point selection step, transmission points inside an ellipse that includes a plurality of basic transmission points corresponding to the three-dimensional point cloud included in the cluster are selected as the other transmission points. (13) The information processing method according to (10), wherein, in the transmission point selection step, transmission points located around each of a plurality of basic transmission points corresponding to the three-dimensional point cloud included in the cluster are selected as the other transmission points. (14) The information processing method described in any one of (1) to (13), further comprising a transmission strength selection step of selecting the transmission strength of the other transmission points when performing the densification based on information on the cluster normal direction indicating the thickness direction of the local space formed by the three-dimensional point cloud included in the cluster.(15) The information processing method according to (14), wherein the transmission strength selection step selects the other transmission points so that the transmission strength increases as the deviation of the cluster normal direction from the direction of the multiple transmission points arranged in the two-dimensional array increases. (16) The information processing method according to any of (1) to (15), further comprising a transmission point proportion selection step of selecting a proportion of the other transmission points when performing the densification based on information on the density of a three-dimensional point cloud included in the cluster. (17) The information processing method according to (16), wherein the transmission point proportion selection step selects the other transmission points so that the proportion increases as the density of the three-dimensional point cloud included in the cluster decreases. (18) The information processing method according to any of (1) to (17), further comprising a basic transmission point position change step of dynamically changing positions of the multiple basic transmission points among the multiple transmission points at a predetermined period. (19) An information processing device comprising: a cluster generation unit that generates a plurality of clusters by clustering a three-dimensional point cloud acquired based on the time difference between the transmission time and the reception time or the phase difference of a transmission signal and a reflection signal corresponding to each of a plurality of basic transmission points that are discretely arranged among a plurality of transmission points arranged in a two-dimensional array; a geometric information calculation unit that calculates, for each of the plurality of clusters, geometric information in a local space formed by the three-dimensional point cloud included in the cluster; and a densification determination unit that determines, for each of the plurality of clusters, based on the geometric information, whether to perform densification to increase the density of the three-dimensional point cloud by using other transmission points as well as basic transmission points corresponding to each point of the three-dimensional point cloud included in the cluster.(20) A program for causing a computer to execute an information processing method, the program comprising: a cluster generation procedure for generating a plurality of clusters by clustering a three-dimensional point cloud acquired based on the time difference between the transmission time and the reception time or the phase difference of a transmission signal and a reflection signal corresponding to each of a plurality of basic transmission points discretely arranged among a plurality of transmission points arranged in a two-dimensional array; a geometric information calculation procedure for calculating, for each of the plurality of clusters, geometric information in a local space formed by the three-dimensional point cloud included in the cluster; and a densification determination procedure for determining, for each of the plurality of clusters, based on the geometric information, whether to perform densification to increase the density of the three-dimensional point cloud by using other transmission points as well as basic transmission points corresponding to each point of the three-dimensional point cloud included in the cluster.

[0125] DESCRIPTION OF SYMBOLS 10: Robot control system 11: Robot control unit 12: Ranging system 13: SLAM execution unit 14: Path planning unit 15: Movement control unit 101: Overall control unit 102: Light emitting unit 103: Light emitting control unit 104: Light receiving unit 105: Light receiving control unit 106: Histogram point group processing unit 107: Basic light emitting point data extraction unit 108: Basic light emitting point data processing unit 109: Densification determination processing unit

Claims

1. An information processing method comprising: a cluster generation step of generating multiple clusters by clustering a 3D point cloud acquired based on the time difference between the transmission time and the reception time or the phase difference of the transmission signal and the reflected signal corresponding to each of a plurality of basic transmission points discretely arranged among a plurality of transmission points arranged in a two-dimensional array; a geometric information calculation step of calculating, for each of the plurality of clusters, geometric information in a local space formed by the 3D point cloud included in the cluster; and a densification determination step of determining, for each of the plurality of clusters, based on the geometric information, whether to increase the density of the 3D point cloud by using other transmission points as well as the basic transmission points corresponding to each point of the 3D point cloud included in the cluster.

2. The information processing method according to claim 1, wherein the geometric information includes at least one of cluster size information indicating the length of the local space in the direction of the main axis, cluster distribution information corresponding to the thickness of the local space, and cluster normal direction information indicating the thickness direction of the local space.

3. The information processing method according to claim 1, wherein the densification determination step determines whether or not to perform the densification based on information on the density of the three-dimensional point cloud contained in the local space and information on the thickness of the local space.

4. The information processing method according to claim 3, wherein the densification determination step determines that the densification should be performed if a cluster size indicating the length of the local space in the direction of the principal axis is equal to or greater than a first threshold value and a cluster variance corresponding to the thickness of the local space is equal to or greater than a second threshold value.

5. The information processing method according to claim 1, wherein the densification determination step determines whether or not to perform the densification based on information on the density of the three-dimensional point cloud included in the local space and information on the thickness direction of the local space.

6. The information processing method according to claim 5, wherein the densification determination step determines that the densification should be performed if the cluster size indicating the length of the local space in the direction of the principal axis is equal to or greater than a first threshold value and the deviation of the cluster normal direction indicating the thickness direction of the local space from the vertical direction is within a second threshold value.

7. The information processing method according to claim 1, wherein the transmitting point is a light-emitting point that outputs an optical pulse signal as the transmitting signal, and each point in the three-dimensional point cloud is acquired based on the time difference of a first peak in histogram data of the time difference between the transmission times of multiple transmitted optical pulse signals and the reception times of reflected optical pulse signals, obtained in response to the output of multiple optical pulse signals.

8. The information processing method according to claim 7, wherein the densification determination step determines whether or not to perform the densification based on information on the density of the three-dimensional point cloud contained in the local space and the histogram data corresponding to the points that make up the three-dimensional point cloud contained in the local space.

9. The information processing method according to claim 8, wherein the densification determination step determines that the densification should be performed if a cluster size indicating the length of the local space in the direction of the major axis is equal to or greater than a first threshold value and if the difference between the score value of a first peak and the score value of at least a second peak in the histogram data is equal to or less than a second threshold value.

10. The information processing method according to claim 1, further comprising a transmission point selection step of selecting the other transmission points when performing the densification based on the positions of a plurality of basic transmission points corresponding to the three-dimensional point group included in the cluster.

11. The information processing method according to claim 10, wherein in the transmission point selection step, a transmission point inside a convex hull that includes a plurality of basic transmission points corresponding to the three-dimensional point group included in the cluster is selected as the other transmission point.

12. The information processing method according to claim 10, wherein in the transmission point selection step, a transmission point inside an ellipse containing a plurality of basic transmission points corresponding to the three-dimensional point group included in the cluster is selected as the other transmission point.

13. The information processing method according to claim 10, wherein in the transmission point selection step, transmission points located around each of a plurality of basic transmission points corresponding to the three-dimensional point group included in the cluster are selected as the other transmission points.

14. The information processing method according to claim 1, further comprising a transmission strength selection step of selecting the transmission strength of the other transmission points when performing the densification based on information on the cluster normal direction indicating the thickness direction of the local space formed by the three-dimensional point group included in the cluster.

15. An information processing method as described in claim 14, wherein in the transmission strength selection procedure, the transmission strength of the other transmission points is selected to be stronger the greater the deviation of the cluster normal direction from the direction of the multiple transmission points arranged in the two-dimensional array.

16. The information processing method according to claim 1, further comprising a transmission point proportion selection step of selecting the proportion of the other transmission points when performing the densification based on information on the density of the three-dimensional point cloud included in the cluster.

17. The information processing method according to claim 16, wherein in the transmission point proportion selection step, the proportion of the other transmission points is selected to be larger as the density of the three-dimensional point group included in the cluster decreases.

18. The information processing method according to claim 1, further comprising a basic transmission point position changing step of dynamically changing the positions of the plurality of basic transmission points among the plurality of transmission points at a predetermined cycle.

19. An information processing device comprising: a cluster generation unit that generates multiple clusters by clustering a three-dimensional point cloud acquired based on the time difference between the transmission time and the reception time or phase difference of the transmission signal and the reflected signal corresponding to each of multiple basic transmission points that are discretely arranged among multiple transmission points arranged in a two-dimensional array; a geometric information calculation unit that calculates, for each of the multiple clusters, geometric information in a local space formed by the three-dimensional point cloud included in the cluster; and a densification determination unit that determines, for each of the multiple clusters, based on the geometric information, whether to densify the three-dimensional point cloud by using other transmission points as well as basic transmission points corresponding to each point of the three-dimensional point cloud included in the cluster to increase the density of the three-dimensional point cloud.

20. A program for causing a computer to execute an information processing method, comprising: a cluster generation procedure for generating multiple clusters by clustering a 3D point cloud acquired based on the time difference between the transmission time and the reception time or phase difference of the transmission signal and the reflected signal corresponding to each of multiple basic transmission points discretely arranged among multiple transmission points arranged in a two-dimensional array; a geometric information calculation procedure for calculating, for each of the multiple clusters, geometric information in a local space formed by the 3D point cloud included in the cluster; and a densification determination procedure for determining, for each of the multiple clusters, based on the geometric information, whether to increase the density of the 3D point cloud by using other transmission points as well as basic transmission points corresponding to each point of the 3D point cloud included in the cluster.

Images