Signal processing device, signal processing method, and data production method

Abstract

To achieve, when control based on three-dimensional point cloud data of a target space is performed as control based on sensing data of the target space, both improved control accuracy and reduced processing burden by efficiently reducing the three-dimensional point cloud data.SOLUTION: A signal processing device according to the present technology comprises: a frequency data generation unit for, on the basis of three-dimensional point cloud data obtained by three-dimensional measurement of a target space, generating frequency data, which is data indicating the frequency of points at least in a depth direction; and a data reduction unit for performing a data reduction process on the three-dimensional point cloud data on the basis of the frequency data generated by the frequency data generation unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present technology relates to a signal processing apparatus, its method, and a data manufacturing method. In particular, as control based on sensing data of a target space, it relates to a technology for reducing the processing load of control processing when control based on three-dimensional point cloud data of the target space is performed.

Background Art

[0002] For example, devices capable of autonomous movement such as AGV (Automatic Guided Vehicle) and AMR (Autonomous Mobile Robot) are known. In controlling autonomous movement, it is desirable to be able to acquire 3D data (three-dimensional point cloud data) as sensing data of the target space. By acquiring 3D data, it is possible to improve the recognition accuracy of obstacles and the object recognition ability such as being able to recognize the height of an object. It is possible to improve the reliability of control and the versatility (for example, by knowing the height of an object, it becomes possible not only to avoid obstacles but also to perform operations such as loading goods onto the object).

[0003] Regarding related prior arts, Patent Document 1 below can be cited. Patent Document 1 below discloses a technique for reducing the data amount of point cloud data by filtering using a grid (30) preset on a three-dimensional space.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Here, since the amount of three-dimensional point cloud data is larger compared to two-dimensional data and the like, it is desirable to reduce the amount of data. However, according to the technology of Patent Document 1 above, since the grid is set at a predetermined fixed position in the three-dimensional space or a position predetermined by a user operation, there is a risk that the filtering using the grid may reduce even the point cloud data that is originally required.

[0006] This technology has been made in view of the above circumstances, and as control based on the sensing data of the target space, when control based on the three-dimensional point cloud data of the target space is performed, the purpose is to achieve both improvement in control accuracy and reduction in processing burden by efficiently reducing the three-dimensional point cloud data.

Means for Solving the Problems

[0007] The signal processing apparatus according to this technology includes a frequency data generation unit that generates frequency data, which is data indicating the frequency of points at least in the depth direction, based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space, and a data reduction unit that performs data reduction processing on the three-dimensional point cloud data based on the frequency data generated by the frequency data generation unit. According to the above configuration, it is possible to efficiently extract only the point cloud data in the region where the point cloud is densely present in the depth direction for the three-dimensional point cloud data, in other words, the point cloud data at the depth position where an object is estimated to exist.

[0008] Further, the signal processing method according to this technology is a signal processing method in which a signal processing apparatus generates frequency data, which is data indicating the frequency of points at least in the depth direction, based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space, and performs data reduction processing on the three-dimensional point cloud data based on the frequency data. Also by such a signal processing method, the same operation as the signal processing apparatus according to the above-described technology of this technology can be obtained.

[0009] Further, the data manufacturing method according to the present technology is a data manufacturing method for manufacturing control data used in a control system that performs control based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space, including the three-dimensional point cloud data and frequency data that is data indicating the frequency of points, at least in the depth direction, generated based on the three-dimensional point cloud data, and is a data manufacturing method for manufacturing the control data used when the control system performs data reduction processing on the three-dimensional point cloud data based on the frequency data. By using the control data manufactured by such a data manufacturing method, when control based on the three-dimensional point cloud data of the target space is performed, it becomes possible to efficiently reduce the three-dimensional point cloud data excluding the point cloud data at the depth position where an object is estimated to exist.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0011] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order. <1. First Embodiment> (1-1. Configuration of Signal Processing Device) (1-2. Regarding Frequency Data Generation) (1-3. Regarding Data Reduction Processing) (1-4. Processing Procedure) (1-5. Regarding Other Configuration Examples) <2. Second Embodiment> <3. Third Embodiment> <4. Modification Example> <5. Summary of Embodiments> <6. The Present Technology>

[0012] <1. First Embodiment> (1-1. Configuration of Signal Processing Device) FIG. 1 is a block diagram showing a configuration example of a signal processing apparatus 1 as a first embodiment according to the present technology. The signal processing apparatus 1 is configured as a three-dimensional surveying apparatus having a function of three-dimensionally measuring a target space to generate three-dimensional point cloud data. Here, as illustrated in FIG. 2, the signal processing apparatus 1 is assumed to be mounted on a moving body M and used. In this example, the moving body M is assumed to be a grounded moving body that can travel on the ground or a floor surface by driving a driven body in contact with the ground or the floor surface, such as wheels. Further, the moving body M is configured as an apparatus capable of autonomous movement such as an AGV (Automatic Guided Vehicle) or an AMR (Autonomous Mobile Robot), and is configured to be capable of performing control for autonomous movement based on the three-dimensional point cloud data obtained as the three-dimensional measurement result of the signal processing apparatus 1. The control for this autonomous movement includes control for avoiding collision with an object Ob in the target space.

[0013] Also, in this example, the three-dimensional measurement of the target space by the signal processing apparatus 1 is assumed to be performed by measuring the distance to an object with a distance measuring sensor. Specifically, in this example, a case where the ToF (Time of Flight) method is adopted as the distance measuring method is illustrated. As is well known, the ToF method is a method of measuring the distance based on the time information (flight time information of light) from when the light is irradiated to the object until the reflected light of the light irradiated to the object is received by the distance measuring sensor. As the ToF method, a dToF (direct ToF) method and an iToF (indirect ToF) method are known.

[0014] As shown in FIG. 1, the signal processing apparatus 1 includes a light emitting unit 2, a distance measuring sensor 3, a control unit 4, a signal processing unit 5, and an IMU (Inertial Measurement Unit) 6. The light emitting unit 2 emits light for distance measurement by the ToF method. For example, light in a predetermined wavelength band such as infrared light is emitted. Although illustration is omitted, the light emitted from the light emitting unit 2 is irradiated onto a space (target space) to be three-dimensionally measured through an irradiation optical system including a lens or the like.

[0015] In the case of the ToF method, the light emitting unit 2 emits light in pulses at a predetermined period. The light emission control of the light emitting unit 2 is performed by the control unit 4.

[0016] The control unit 4 is configured to include at least a light emission control circuit of the light emitting unit 2, and performs light emission control (in this example adopting the ToF method, light emission control for pulsed light emission) for causing the light emitting unit 2 to emit light in a predetermined light emission mode, and outputs a synchronization signal synchronized with the light emission period of the light emitting unit 2 to the distance measurement sensor 3.

[0017] The distance measurement sensor 3 includes a pixel array unit 31, a distance image generation unit 32, and a point cloud data generation unit 33. The pixel array unit 31 is formed by two-dimensionally arranging pixels each having a light receiving element. Although illustration is omitted, for the pixel array unit 31, the reflected light obtained by reflecting the light emitted from the light emitting unit 2 by an object is incident through a light receiving optical system including a lens or the like. For the distance measurement sensor 3, the above-described synchronization signal is input from the control unit 4, and the pixel array unit 31 performs a light receiving operation in a period synchronized with this synchronization signal.

[0018] The distance image generation unit 32 performs a predetermined calculation for distance calculation by the ToF method based on the light receiving signal for each pixel obtained by the pixel array unit 31 performing a light receiving operation corresponding to the ToF method, and generates a distance image. Here, the distance image means an image indicating information on the distance to the subject for each pixel.

[0019] The point cloud data generation unit 33 generates three-dimensional point cloud data indicating the three-dimensional structure of the target space based on the distance image generated by the distance image generation unit 32.

[0020] FIG. 3 is an explanatory diagram of three-dimensional point cloud data, and schematically shows the three-dimensional point cloud data obtained when three-dimensional measurement is performed in an environment where the object Ob exists in the front as shown in the previous FIG. 2. The three-dimensional point cloud data is data indicating the coordinates of each measured point in a world coordinate system defined for the target space. Specifically, the world coordinate system here is a coordinate system with the Z-axis in the depth direction, the X-axis in the lateral direction (a direction parallel to the horizontal plane and orthogonal to the Z-axis), and the Y-axis in the vertical direction (a direction orthogonal to the Z-axis and the X-axis), and the three-dimensional point cloud data is data indicating the (X, Y, Z) coordinates of each point.

[0021] As shown in the drawing, as the three-dimensional point cloud data obtained for the environment of the previous FIG. 2, the point cloud is concentrated in the Y direction on the front surface (the surface facing the signal processing device 1) of the object Ob, and the point cloud is sparse on the floor surface on which the object Ob is placed. Also, in this example, since the distance measurement for obtaining the three-dimensional point cloud data is performed by receiving the reflected light of the light irradiated to the object Ob side by the light emitting unit 2 as described above, the range in which the point cloud data can be obtained is limited to at least the range where the irradiation light by the light emitting unit 2 reaches. Further, since light does not reach or the amount of light reaching is extremely small on the rear side of the front surface of the object Ob, almost no point cloud can be obtained on the rear side of the front surface of the object Ob.

[0022] Here, for the distance image obtained by the distance image generation unit 32, when the horizontal line direction in the pixel array unit 31 is the u direction and the direction orthogonal to the horizontal line direction is the v direction, the position of each pixel is represented in the (u, v) coordinate system. Therefore, the distance z calculated for each pixel can be expressed in the coordinates of (u, v, z).

[0023] The point cloud data generation unit 33 shown in FIG. 1 generates three-dimensional point cloud data by converting the (u, v, z) coordinates for each pixel in the distance image into the (X, Y, Z) coordinates in the world coordinate system. The coordinate conversion at this time can be performed based on optical parameters (camera parameters) such as the focal length of the above-described light receiving optical system.

[0024] The three-dimensional point cloud data generated by the point cloud data generation unit 33 is output to the signal processing unit 5.

[0025] The IMU 6 is configured to include motion sensors such as an acceleration sensor and a gyro sensor (angular velocity sensor), and detects information indicating the inclination of the signal processing device 1 in each direction of yaw, pitch, and roll as the attitude information of the signal processing device 1. As shown in the figure, a synchronization signal, specifically a synchronization signal synchronized with the frame period of the pixel array unit 31, is input to the IMU 6 from the distance measurement sensor 3, and the IMU 6 detects the attitude information at a period synchronized with the synchronization signal. The attitude information detected by the IMU 6 is output to the signal processing unit 5.

[0026] The signal processing unit 5 is configured to include a processor such as a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), an FPGA (Field-Programmable Gate Array), or a DSP (Digital Signal Processor), and performs various signal processes on the three-dimensional point cloud data. As shown in the figure, the signal processing unit 5 has functions as a coordinate conversion unit 51, a frequency data generation unit 52, and a data reduction unit 53.

[0027] The coordinate conversion unit 51 performs coordinate conversion of the three-dimensional point cloud data input from the distance measurement sensor 3 based on the attitude information input from the IMU 6. The coordinate conversion here is performed to compensate for the deviation from the ideal attitude regarding the mounting attitude of the signal processing device 1. For example, when the mounting attitude of the signal processing device 1 deviates from the ideal attitude, such as when the signal processing device 1 is mounted obliquely, data indicating the correct coordinates in the assumed world coordinate system cannot be obtained as the three-dimensional point cloud data. Therefore, coordinate conversion processing of the three-dimensional point cloud data is performed so that the attitude of the signal processing device 1 detected by the IMU 6 coincides with the ideal attitude, so that the three-dimensional point cloud data indicates the correct coordinates in the world coordinate system.

[0028] Note that when performing coordinate transformation by the coordinate transformation unit 51, it is not essential to provide the IMU 6. For example, a distance measurement operation by the distance measurement sensor 3 is executed on a predetermined calibration board or the like, and the mounting posture of the signal processing device 1 is detected based on the distance image obtained thereby. Then, based on the information on the mounting posture thus detected, a method can also be adopted in which the coordinate transformation unit 51 performs coordinate transformation of the three-dimensional point cloud data.

[0029] The frequency data generation unit 52 generates frequency data (histogram), which is data indicating the frequency of points at least in the depth direction (that is, the Z direction) based on the three-dimensional point cloud data after coordinate transformation by the coordinate transformation unit 51.

[0030] The data reduction unit 53 performs data reduction processing on the three-dimensional point cloud data based on the frequency data generated by the frequency data generation unit 52.

[0031] Hereinafter, the generation method of the frequency data by the frequency data generation unit 52 and the details of the data reduction processing by the data reduction unit 53 will be described.

[0032] (1-2. Regarding frequency data generation) An example of the generation method of the frequency data by the frequency data generation unit 52 will be described with reference to FIGS. 4 to 8. By generating frequency data, which is data indicating the frequency of points in the Z direction for the three-dimensional point cloud data as described above, the distribution of the point cloud density in the Z direction can be estimated. In the case of the three-dimensional point cloud data illustrated in FIG. 3 above, for the floor surface where the object Ob does not exist, the density of the point cloud is sparse, in other words, the frequency of the measured points is low. On the other hand, for the part where the object Ob exists, the density of the point cloud is high, and the frequency of the measured points is high. Therefore, by generating the frequency data as described above and identifying the Z-direction region in the frequency data where the frequency is equal to or higher than the threshold value, it is possible to identify the existence region of the object Ob in the Z direction (which becomes the existence region of the front portion of the object Ob in this example).

[0033] However, merely identifying the existence region of the object Ob in the Z direction leaves the existence regions of the object Ob in the X and Y directions unknown. For example, it becomes difficult to perform driving control such as passing the moving body M while avoiding the region where the object Ob exists. Also, even when performing control such as stopping the moving body M in front of the object Ob so as to avoid a collision with the object Ob, if the existence regions of the object Ob in the X and Y directions are unknown, there is a risk that the moving body M will be stopped unnecessarily, such as when it is stopped at an X-direction position where the object Ob does not exist.

[0034] For this reason, in this example, as the frequency data, data is generated that shows not only the frequency of points in the Z direction but also the frequency of points in the orthogonal direction (X direction or Y direction) to the Z direction. Specifically, in this example, the frequency data generates data that shows the frequency of points in two directions, the Z direction and the X direction.

[0035] FIG. 4 is a top view of the three-dimensional point cloud data shown in FIG. 3 above, viewed from above the X-Z plane. By generating the frequency data that shows the frequency of points in two directions, the Z direction and the X direction, as described above, it becomes possible to identify the region indicated as "P" in the figure, that is, the existence region of the object Ob on the X-Z plane.

[0036] Specific methods for generating the frequency data will be described with reference to the first example and the second example. FIG. 5 is an explanatory diagram of a method for generating frequency data as the first example. In the first example, as shown by the dotted frame in the figure, a plurality of cells S that divide the X-Z plane into a grid pattern are defined, and the frequency of points is obtained for each cell S, thereby generating frequency data that shows the frequency of points in two directions, the Z direction and the X direction. Here, regarding the size of the cell S, it may be the size of one coordinate in the world coordinate system, or it can be the size of multiple coordinates.

[0037] Figure 6 is a diagram showing the visualized frequency data. From Figure 6, it can be seen that in this case, the frequency data shows a high frequency in the region where the front part of the object Ob exists on the X-Z plane.

[0038] In the case of the first example, the frequency data generation unit 52 generates frequency data indicating the frequency of points in the X-Z plane as shown in Figure 6 by obtaining the frequency of points for each cell S by the method described in Figure 5 based on the three-dimensional point cloud data.

[0039] Figure 7 is an explanatory diagram of a frequency data generation method as a second example. In the second example, as shown by the dotted line in the figure, a plurality of lines L that divide the X-Z plane into strip shapes in the X direction are defined, and for each line L, frequency data indicating the frequency of points in the Z direction and the X direction is generated by obtaining the frequency of points in the Z direction. Regarding the width of the line L, it may be the size of one coordinate in the world coordinate system, or it can be the size of multiple coordinates.

[0040] Figure 8 is a diagram schematically showing an example of the frequency of points in the Z direction obtained for each line L. For example, when assuming the three-dimensional point cloud data in Figure 3, the change in the frequency of points in the Z direction is small for the line L near the end in the X direction (because it is almost the floor surface). On the other hand, for the line L in the central part where the object Ob exists, the frequency of points in the Z direction shows a characteristic that the frequency increases in a certain region in the Z direction as shown in the figure. By synthesizing the data of the frequency of points in the Z direction for each such line L, frequency data similar to that shown in Figure 6 can be obtained.

[0041] (1-3. Regarding the data reduction process) The data reduction unit 53 performs a process of extracting point cloud data in which the frequency is equal to or higher than a predetermined threshold value from the frequency data generated by the frequency data generation unit 52. Specifically, in the frequency data, an X-Z region in which the frequency is equal to or higher than a predetermined threshold value is specified, and point cloud data in which the X and Z coordinates are coordinates within the X-Z region is extracted. As a result, it is possible to efficiently extract point cloud data in a region where it is estimated that the object Ob exists in the X-Z plane.

[0042] Here, in the data reduction unit 53, as the reduced data for the three-dimensional point cloud data output to the subsequent stage, it is conceivable to directly output the point cloud data extracted as described above.

[0043] Alternatively, as the reduced data, it is also conceivable to generate and output data with a further reduced data amount for the point cloud data extracted as described above. For example, it is conceivable to generate two-dimensional data obtained by projecting the point cloud data extracted as described above onto a two-dimensional plane (X-Z plane) defined by the Z-axis and the X-axis, and output the two-dimensional data. Alternatively, it is also conceivable to extract and output only the Z-direction coordinate information of the point cloud data extracted as described above.

[0044] (1-4. Processing procedure) For confirmation, referring to the flowchart shown in FIG. 9, an example of the processing procedure executed by the signal processing unit 5 shown in FIG. 1 will be described. First, in step S101, the signal processing unit 5 inputs three-dimensional point cloud data. The process of this step S101 is a process of inputting the three-dimensional point cloud data obtained by the distance measurement sensor 3 for each frame period of the distance image.

[0045] In step S102 following step S101, the signal processing unit 5 acquires attitude information. That is, the attitude information of the signal processing device 1 detected by the IMU 6 is acquired.

[0046] In step S103 following step S102, the signal processing unit 5 performs coordinate conversion of the three-dimensional point cloud data based on the attitude information. This is the processing as the coordinate conversion unit 51 described above.

[0047] In step S104 following step S103, the signal processing unit 5 performs frequency data generation processing. That is, based on the three-dimensional point cloud data after the coordinate conversion in step S103, frequency data is generated by the method as the first example or the second example described above.

[0048] In step S105 following step S104, the signal processing unit 5 executes data reduction processing. That is, based on the frequency data generated in step S104, data reduction processing for the three-dimensional point cloud data is performed as described in the processing of the previous data reduction unit 53 to obtain reduced data.

[0049] In step S106 following step S105, the signal processing unit 5 executes output processing of the reduced data. The reduced data can be output, for example, to a control device that performs traveling control of the moving body M and is mounted on the moving body M.

[0050] In step S107 following step S106, the signal processing unit 5 determines whether the processing is completed. That is, it determines whether a predetermined condition that should be the end of the processing for data reduction of the three-dimensional point cloud data is satisfied.

[0051] In step S107, if it is determined that the processing is not completed, the signal processing unit 5 returns to step S101. That is, in this case, for the next frame, the processing from steps S101 to S106 is executed.

[0052] On the other hand, in step S107, if it is determined that the processing is completed, the signal processing unit 5 ends the series of processing shown in FIG. 9.

[0053] (1-5. Regarding other configuration examples) Note that, in the above description, as an example of frequency data, data indicating the frequency of points in the Z direction or data indicating the frequency of points in two directions of the Z direction and the X direction was given. However, as frequency data, it is also conceivable to generate data indicating the frequency of points in three directions of the Z direction, the X direction, and the Y direction.

[0054] Also, although not particularly mentioned above, it is also conceivable to make it possible to select whether to execute data reduction on the three-dimensional point cloud data by a user operation. In this case, when the setting to execute data reduction is made by an operation, the signal processing unit 5 executes the processes as the coordinate conversion unit 51, the frequency data generation unit 52, and the data reduction unit 53 described above, and outputs the reduced data. On the other hand, when the setting not to execute data reduction is made by an operation, at least the processes as the frequency data generation unit 52 and the data reduction unit 53 are not executed, and three-dimensional point cloud data instead of the reduced data is output.

[0055] Also, in the above description, an example in which the signal processing unit 5 is provided outside the distance measuring sensor 3 was given. However, as another example of the signal processing device 1A shown in FIG. 10, a configuration in which a distance measuring sensor 3A having the signal processing unit 5 inside is provided instead of the distance measuring sensor 3 can also be adopted.

[0056] <2. Second Embodiment> Subsequently, the second embodiment will be described. The second embodiment is an embodiment corresponding to the case where a plurality of measurement units for performing three-dimensional measurement are provided. For example, as illustrated in FIG. 11, assume a signal processing device 1B having three measurement units: a measurement unit 1a, a measurement unit 1b, and a measurement unit 1c. As will be described later, the "measurement unit" mentioned here means a unit part that has a light emitting unit 2 and a distance measuring sensor 3 and is capable of generating three-dimensional point cloud data. At this time, in the target space, it is assumed that, as shown in the figure, an object Ob1 is in front of the measurement unit 1a arranged at the center, an object Ob2 is in front of the measurement unit 1b arranged on the left side of the measurement unit 1a, and an object Ob3 is in front of the measurement unit 1c arranged on the right side of the measurement unit 1a.

[0057] FIG. 12 is a top view of the three-dimensional point cloud data obtained by each of the measurement units 1a, 1b, and 1c as viewed from above the X-Z plane. Comparing with the top view shown in FIG. 4 above, it can be seen that by providing a plurality of measurement units, the range in which three-dimensional measurement is possible can be expanded. Assuming the object arrangement in FIG. 11, in the three-dimensional point cloud data of the measurement unit 1a, as shown by "P1" in the figure, a region where the point cloud is dense is observed in the region where the front part of the object Ob1 exists. Also, in the three-dimensional point cloud data of the measurement unit 1b, a region where the point cloud is dense is observed in the region where the front part of the object Ob2 exists as shown by "P2" in the figure, and in the three-dimensional point cloud data of the measurement unit 1c, a region where the point cloud is dense is observed in the region where the front part of the object Ob3 exists as shown by "P3" in the figure.

[0058] In the second embodiment, after obtaining the combined three-dimensional point cloud data by combining the three-dimensional point cloud data obtained for each measurement unit, frequency data is generated for this combined three-dimensional point cloud data, and data reduction processing of the combined three-dimensional point cloud data based on the frequency data is performed.

[0059] FIG. 13 is a block diagram showing a configuration example of a signal processing apparatus 1B as a second embodiment for realizing the data reduction method as the second embodiment as described above. In the following description, parts that are the same as the parts already described will be denoted by the same reference numerals and the same step numbers, and the description thereof will be omitted. Also, in FIG. 13, a configuration example including two measurement units 1a and 1b as measurement units will be described, but the number of measurement units may be at least plural.

[0060] In the signal processing device 1B, the differences from the signal processing device 1 are that two measurement units, namely a measurement unit 1a and a measurement unit 1b, are provided as a measurement unit having a light emitting unit 2 and a distance measurement sensor 3, a control unit 4B is provided instead of the control unit 4, and a signal processing unit 5B is provided instead of the signal processing unit 5.

[0061] The control unit 4B performs light emission control for both the light emitting unit 2 in the measurement unit 1a and the light emitting unit 2 in the measurement unit 1b. In addition, it outputs a synchronization signal to both the distance measurement sensor 3 in the measurement unit 1a and the distance measurement sensor 3 in the measurement unit 1b.

[0062] In the figure, regarding the synchronization signal for the IMU6 (the frame synchronization signal of the distance measurement sensor 3), an example is shown in which the synchronization signal of the distance measurement sensor 3 in the measurement unit 1b is input. However, if the frame periods of the distance measurement sensors 3 of both the measurement unit 1a and the measurement unit 1b are synchronized, the synchronization signal may be input from any of the distance measurement sensors 3.

[0063] The signal processing unit 5B is different from the signal processing unit 5 in that a coordinate conversion unit 51B is provided instead of the coordinate conversion unit 51 and a synthesis unit 54 is added. The coordinate conversion unit 51B performs coordinate conversion on the three-dimensional point cloud data input from each of the measurement units 1a and 1b by the same method as in the case of the first embodiment based on the attitude information input from the IMU6.

[0064] The synthesis unit 54 synthesizes the three-dimensional point cloud data that has been coordinate-converted by the coordinate conversion unit 51B. Regarding the synthesis of the three-dimensional point cloud data, for the overlapping measurement parts between the measurement units, it is conceivable to adopt only the point cloud data of one of the measurement units. Also, here an example is shown in which the three-dimensional point cloud data for each measurement unit is coordinate-converted and then synthesized. However, it is also possible to perform coordinate conversion on the synthesized three-dimensional point cloud data.

[0065] In this case, the frequency data generation unit 52 generates frequency data by the same method as in the case of the first embodiment based on the synthesized three-dimensional point cloud data obtained by the synthesis unit 54.

[0066] FIG. 14 is a flowchart showing an example of the procedure of the process executed by the signal processing unit 5B. The differences from the process shown in FIG. 9 are that the input process of step S201 is performed instead of the input process of step S101, the coordinate conversion process of step S202 is performed instead of the coordinate conversion process of step S103, and the synthesis process of step S202 is added.

[0067] In step S201, the signal processing unit 5B inputs three-dimensional point cloud data from each measurement unit. That is, the three-dimensional point cloud data is input from the distance measurement sensors 3 of both the measurement units 1a and 1b.

[0068] Also, in step S202, the signal processing unit 5B performs coordinate conversion processing on each three-dimensional point cloud data based on the attitude information acquired in step S102. This is the process as the coordinate conversion unit 51B described above.

[0069] Then, in step S203 following step S202, the signal processing unit 5B performs a process of synthesizing the three-dimensional point cloud data. This is the process as the synthesis unit 54 described above.

[0070] The signal processing unit 5B advances the process to step S104 in response to having executed the synthesis process of step S203. Since the processes after step S104 are the same as those described in FIG. 9, duplicate explanations are avoided.

[0071] <3. Third Embodiment> Here, in the above-described first and second embodiments, an example is given in which the frequency data generation process and the data reduction process based on the frequency data are performed in the same processor. However, it is also conceivable to adopt a configuration in which these frequency data generation processes and frequency data are executed by separate processors, respectively.

[0072] FIG. 15 is a block diagram showing a configuration example of the signal processing apparatus 1C as a third embodiment. Here, as a configuration example in which the generation process of frequency data and the data reduction process based on the frequency data are executed by different processors, a case of adopting a configuration based on the signal processing apparatus 1A shown in FIG. 10 above is illustrated.

[0073] The differences from the signal processing apparatus 1A shown in FIG. 10 are that a distance measurement sensor 3C is provided instead of the distance measurement sensor 3A, and a subsequent stage signal processing unit 7 is added. The distance measurement sensor 3C is different from the distance measurement sensor 3A in that a signal processing unit 5C is provided instead of the signal processing unit 5. The signal processing unit 5C is obtained by omitting the data reduction unit 53 from the signal processing unit 5. The signal processing unit 5C outputs the three-dimensional point cloud data coordinate-converted by the coordinate conversion unit 51 to the subsequent stage signal processing unit 7 provided outside the distance measurement sensor 3C together with the frequency data generated by the frequency data generation unit 52.

[0074] The subsequent stage signal processing unit 7 is configured to include a processor such as a microcomputer having, for example, a CPU, a ROM, and a RAM, an FPGA, or a DSP, and has a function as the data reduction unit 53 as shown in the figure. In the subsequent stage signal processing unit 7, the data reduction unit 53 inputs the three-dimensional point cloud data and the frequency data output from the signal processing unit 5C, and based on the frequency data, executes a data reduction process on the three-dimensional point cloud data by the same method as the method described in the first embodiment.

[0075] Here, a data structure example of the three-dimensional point cloud data and the frequency data transferred from the signal processing unit 5C to the subsequent stage signal processing unit 7 will be described with reference to FIG. 16. Here, a data structure example assuming the MIPI (Mobile Industry Processor Interface) standard will be described as a transfer standard between the signal processing unit 5C (distance measurement sensor 3C) and the subsequent stage signal processing unit 7.

[0076] For example, in the example of FIG. 16A, frequency data is stored in the "Embedded data" area provided for each frame of the transfer data. In this case, the three-dimensional point cloud data is stored in the actual data storage area as the "Effective pixel area".

[0077] Also, the example of FIG. 16B is an example in which frequency data is stored for each line within the frame of the transfer data. For example, frequency data is stored in the SOL (Start of Line) area in the MIPI standard.

[0078] The example of FIG. 16C is an example in which the three-dimensional point cloud data and the frequency data are transferred separately. In this case, the association between the three-dimensional point cloud data and the frequency data can be considered to be performed by attaching the same frame number to each data.

[0079] FIG. 17 is a block diagram showing a configuration example of the signal processing device 1D as another example in the third embodiment. This signal processing device 1D is obtained by applying the configuration as the third embodiment to the signal processing device 1 shown in FIG. 1. In the signal processing device 1D, the differences from the signal processing device 1 are that a signal processing unit 5C is provided instead of the signal processing unit 5, and a subsequent-stage signal processing unit 7 is added.

[0080] Note that also in the third embodiment, it is possible to adopt a configuration corresponding to the case where a plurality of measurement units are provided as in the second embodiment.

[0081] <4. Variation Example> Although the embodiments according to the present technology have been described above, the present technology is not limited to the above-described specific examples and can adopt configurations as various variation examples. For example, in the above, a configuration in which the generation process of frequency data and the data reduction process of three-dimensional point cloud data based on the frequency data are performed in the signal processing device has been exemplified, but the data reduction process of three-dimensional point cloud data based on the frequency data can also be considered to be performed in an external device of the signal processing device.

[0082] Fig. 18 shows a specific configuration example. In the figure, the signal processing device 1E is different from the signal processing device 1 in that a signal processing unit 5C is provided instead of the signal processing unit 5. The signal processing device 1E outputs the three-dimensional point cloud data coordinate-transformed by the coordinate transformation unit 51 to the external device 10 together with the frequency data generated by the frequency data generation unit 52.

[0083] The external device 10 is configured to include, for example, a microcomputer having a CPU, a ROM, a RAM, etc., or a processor such as an FPGA or a DSP, and has a function as a data reduction unit 53 as shown in the figure. As this external device 10, a device that performs travel control of the moving body M based on the three-dimensional point cloud data is assumed. Specifically, the external device 10 has a function of performing travel control of the moving body M based on the reduced data obtained by the data reduction unit 53 performing data reduction processing based on the frequency data.

[0084] Here, when assuming a control system in which the signal processing device 1E performs the generation process of the frequency data as described above and the external device 10 performs the data reduction process of the three-dimensional point cloud data based on the frequency data, the signal processing device 1E transfers the three-dimensional point cloud data and the frequency data to the external device 10. At this time, for the transfer data as well, similar to the example described in Fig. 16 above, for the frequency data, a method of storing it in the additional data area provided for each predetermined data unit such as a frame or a line in the transfer data can be considered. Alternatively, a method of transferring the three-dimensional point cloud data and the frequency data separately can also be considered. Here, the data including the three-dimensional point cloud data and the frequency data transferred from the signal processing device 1E to the external device 10 can be paraphrased as "control data" in terms of the data used for control based on the three-dimensional point cloud data.

[0085] Here, in the description so far, a grounded moving body has been exemplified as the moving body M, but as the moving body M, for example, a flying body such as a drone or an airplane, or a moving body other than a grounded type can also be considered. Also, in relation to this, the control based on the three-dimensional point cloud data in the present technology is not limited to the travel control of the moving body M. In the present technology, regarding the control based on the three-dimensional point cloud data, the specific control content is not particularly limited.

[0086] Also, in the previous description, regarding three-dimensional measurement, the case of performing distance measurement by the ToF method was exemplified, but in the present technology, the method of three-dimensional measurement is not limited to the ToF method. For example, it is also conceivable to perform three-dimensional measurement by a LiDAR (Light Detection And Ranging) method other than the ToF method, a stereo camera, a monocular SLAM (Simultaneous Localization and Mapping), a compound eye SLAM, or the like. Alternatively, as the three-dimensional measurement, it is also conceivable to adopt a method using ultrasonic distance measurement, an electromagnetic wave radar, or the like.

[0087] <Summary of Embodiment> As described above, the signal processing apparatus (the same as 1, 1A, 1B, 1D) as an embodiment includes a frequency data generation unit (the same as 52) that generates frequency data, which is data indicating the frequency of points at least in the depth direction, based on the three-dimensional point cloud data obtained by three-dimensionally measuring the target space, and a data reduction unit (the same as 53) that performs data reduction processing on the three-dimensional point cloud data based on the frequency data generated by the frequency data generation unit. According to the above configuration, it is possible to efficiently extract only the point cloud data in the region where the point cloud is densely present in the depth direction, in other words, the point cloud data at the depth position where an object is estimated to exist, for the three-dimensional point cloud data. Therefore, as control based on the sensing data of the target space, when control based on the three-dimensional point cloud data of the target space is performed, it is possible to efficiently reduce the three-dimensional point cloud data excluding the point cloud data at the depth position where an object is estimated to exist, and it is possible to achieve both an improvement in control accuracy and a reduction in processing load. In addition, it is possible to reduce the amount of transfer data to the subsequent processor that performs control, reduce the data transfer time, reduce the bandwidth of the communication path, and improve the processing speed of the subsequent processing.

[0088] Further, in the signal processing apparatus according to the embodiment, the frequency data generation unit generates, as frequency data, data indicating the frequencies of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction. Thereby, as frequency data, data indicating in which region on the two-dimensional plane (Z-X plane or Z-Y plane) defined by the axis (Z axis) in the depth direction and the axis (X axis or Y axis) orthogonal thereto the points are concentrated can be obtained. Therefore, by performing data reduction processing on the three-dimensional point cloud data based on such frequency data, it is possible to efficiently extract the point cloud data only for the region where an object is presumed to exist in the direction orthogonal to the depth direction.

[0089] Furthermore, in the signal processing apparatus according to the embodiment, the frequency data generation unit generates, as frequency data, data indicating the frequencies of points in two directions, namely, the depth direction and the horizontal direction. Thereby, as frequency data, data indicating in which region on the two-dimensional plane (Z-X plane) defined by the axis (Z axis) in the depth direction and the axis (X axis) in the horizontal direction the points are concentrated can be obtained. Therefore, by performing data reduction processing on the three-dimensional point cloud data based on such frequency data, it is possible to efficiently extract the point cloud data only for the region where an object is presumed to exist in the horizontal direction.

[0090] Furthermore, the signal processing apparatus according to the embodiment includes a coordinate conversion unit (51, 51B) that performs coordinate conversion on the three-dimensional point cloud data, and the frequency data generation unit generates frequency data based on the three-dimensional point cloud data after coordinate conversion by the coordinate conversion unit. Thereby, even when the posture of the measurement unit that performs three-dimensional measurement is different from the ideal posture, it becomes possible to perform coordinate conversion so as to obtain the three-dimensional point cloud data when the measurement is performed in the ideal posture. Therefore, the installation freedom degree of the measurement unit can be improved.

[0091] In addition, in the signal processing apparatus (the same 1B) as an embodiment, a combining unit (the same 54) that combines a plurality of three-dimensional point group data obtained by three-dimensionally measuring a target space from different viewpoints is provided, and the frequency data generation unit generates frequency data based on the three-dimensional point group data after being combined by the combining unit. Thereby, when a plurality of measurement units are provided to expand the measurement range, the three-dimensional point group data can be efficiently reduced, and both improvement of control accuracy and reduction of processing load can be achieved. In addition, when a plurality of measurement units are provided to expand the measurement range, since the number of three-dimensional point groups to be handled increases, it is particularly suitable to efficiently reduce the three-dimensional point group data according to this technology.

[0092] Furthermore, in the signal processing apparatus (the same 1, 1A, 1B, 1D) as an embodiment, the data reduction unit performs a process of extracting and outputting point group data in which the frequency in the frequency data is equal to or higher than a threshold value from the three-dimensional point group data. Thereby, as the reduced data, only the point group data in the region where an object is estimated to exist in the depth direction can be efficiently extracted and output.

[0093] Furthermore, in the signal processing apparatus as an embodiment, the frequency data generation unit generates, as frequency data, data indicating the frequency of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction, and the data reduction unit performs a process of extracting and outputting point group data in which the frequency is equal to or higher than a threshold value from the frequency data indicating the frequency of points in the two directions generated by the frequency data generation unit. Thereby, as the reduced data, only the point group data in the region where an object is estimated to exist in the direction orthogonal to the depth direction can be efficiently extracted and output. The data amount of the reduced data can be reduced more than when only the point group data in the region where an object is estimated to exist in the depth direction is extracted and output.

[0094] Also, in the signal processing apparatus according to the embodiment, the frequency data generation unit generates, as frequency data, data indicating the frequencies of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction. The data reduction unit extracts, from the three-dimensional point cloud data, the point cloud data in which the frequency in the frequency data is equal to or greater than a threshold value, and outputs two-dimensional data obtained by projecting the extracted point cloud data onto a two-dimensional plane defined by the axis in the depth direction and the axis in the orthogonal direction. As a result, as the reduced data, it is possible to output data reduced to only two types of data, namely, data indicating the region where an object is estimated to exist in the direction orthogonal to the depth direction and data indicating the position of the object in the depth direction. It is possible to reduce the amount of the reduced data as compared with the case of extracting and outputting only the point cloud data of the region where an object is estimated to exist in the direction orthogonal to the depth direction.

[0095] Furthermore, in the signal processing apparatus according to the embodiment, the frequency data generation unit generates, as frequency data, data indicating the frequencies of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction. The data reduction unit extracts, from the three-dimensional point cloud data, the point cloud data in which the frequency in the frequency data is equal to or greater than a threshold value, and extracts and outputs only the coordinate information in the depth direction of the extracted point cloud data. As a result, as the reduced data, it is possible to output data reduced to only the data indicating the position of the object in the depth direction.

[0096] The signal processing method of the embodiment is a signal processing method in which a signal processing apparatus generates frequency data, which is data indicating at least the frequency of points in the depth direction, based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space, and performs data reduction processing on the three-dimensional point cloud data based on the frequency data. Also, by such a signal processing method, the same operations and effects as those of the signal processing apparatus according to the above-described embodiment can be obtained.

[0097] Also, a data manufacturing method as an embodiment is a data manufacturing method for manufacturing control data used in a control system (signal processing device 1E and external device 10) that performs control based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space. The data manufacturing method includes three-dimensional point cloud data and frequency data that is data indicating the frequency of points in at least the depth direction, generated based on the three-dimensional point cloud data. The control system manufactures control data used when performing data reduction processing on the three-dimensional point cloud data based on the frequency data. When control based on the three-dimensional point cloud data of the target space is performed using the control data manufactured by such a data manufacturing method, it becomes possible to efficiently reduce the three-dimensional point cloud data excluding the point cloud data at the depth position where an object is estimated to exist. Therefore, it is possible to achieve both an improvement in control accuracy and a reduction in processing load.

[0098] Note that the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0099] <6. The present technology> The present technology can also adopt the following configuration. (1) A frequency data generation unit that generates frequency data, which is data indicating the frequency of points in at least the depth direction, based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space, A data reduction unit that performs data reduction processing on the three-dimensional point cloud data based on the frequency data generated by the frequency data generation unit, A signal processing device. (2) The frequency data generation unit generates, as the frequency data, data indicating the frequency of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction. The signal processing device according to (1) above. (3) The depth data generation unit generates data indicating the frequency of points in two directions, i.e., the depth direction and the lateral direction, as the frequency data. The signal processing device according to (2) above. (4) It includes a coordinate conversion unit that performs coordinate conversion of the three-dimensional point cloud data. The frequency data generation unit generates the frequency data based on the three-dimensional point cloud data after coordinate conversion by the coordinate conversion unit. The signal processing device according to any one of (1) to (3) above. (5) It includes a synthesis unit that synthesizes a plurality of three-dimensional point cloud data obtained by three-dimensional measurement of the target space from different viewpoints. The frequency data generation unit generates the frequency data based on the three-dimensional point cloud data after synthesis by the synthesis unit. The signal processing device according to any one of (1) to (4) above. (6) The data reduction unit performs a process of extracting and outputting point cloud data in which the frequency in the frequency data is equal to or higher than a threshold value from the three-dimensional point cloud data. The signal processing device according to any one of (1) to (5) above. (7) The frequency data generation unit generates data indicating the frequency of points in two directions, i.e., the depth direction and the direction orthogonal to the depth direction, as the frequency data. The data reduction unit performs a process of extracting and outputting point cloud data in which the frequency is equal to or higher than a threshold value in the frequency data indicating the frequency of points in the two directions generated by the frequency data generation unit. The signal processing device according to (6) above. (8) The frequency data generation unit generates data indicating the frequency of points in two directions, i.e., the depth direction and the direction orthogonal to the depth direction, as the frequency data. The data reduction unit extracts point cloud data in which the frequency in the frequency data is equal to or higher than a threshold value from the three-dimensional point cloud data, and outputs two-dimensional data obtained by projecting the extracted point cloud data onto a two-dimensional plane determined by the axis in the depth direction and the axis in the orthogonal direction. The signal processing device according to any one of (1) to (5) above. (9) The frequency data generation unit generates, as the frequency data, data indicating the frequencies of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction. The data reduction unit extracts point cloud data in which the frequency in the frequency data is equal to or higher than a threshold value from the three-dimensional point cloud data, and extracts and outputs only the coordinate information in the depth direction of the extracted point cloud data. The signal processing device according to any one of (1) to (5) above. (10) A signal processing device generates frequency data, which is data indicating at least the frequency of points in the depth direction, based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space. performs data reduction processing on the three-dimensional point cloud data based on the frequency data. A signal processing method. (11) A data manufacturing method for manufacturing control data used in a control system that performs control based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space, including the three-dimensional point cloud data and frequency data generated based on the three-dimensional point cloud data, the frequency data being data indicating at least the frequency of points in the depth direction, the control data used when the control system performs data reduction processing on the three-dimensional point cloud data based on the frequency data is manufactured A data manufacturing method.

Description of Signs

[0100] 1, 1A, 1B, 1C, 1D, 1E Signal processing device M Moving body Ob, Ob1, Ob2, Ob3 Objects 1a, 1b, 1c Measuring units 2 Light emitting unit 3, 3A, 3C Distance measuring sensors 31 Pixel array unit 32 Distance Image Generation Unit 33 Point Cloud Data Generation Unit 4,4B Control Unit 5,5B,5C Signal Processing Unit 51,51B Coordinate Conversion Unit 52 Frequency Data Generation Unit 53 Data Reduction Unit 54 Synthesis Unit 6 IMU S Cell L Line 7 Post-stage Signal Processing Unit 10 External Device

Claims

1. A frequency data generation unit that generates frequency data, which is data indicating the frequency of points at least in the depth direction, based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space; A data reduction unit that performs data reduction processing on the three-dimensional point cloud data based on the frequency data generated by the frequency data generation unit, A signal processing device.

2. The frequency data generation unit generates, as the frequency data, data indicating the frequency of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction. The signal processing device according to claim 1.

3. The frequency data generation unit generates, as the frequency data, data indicating the frequency of points in two directions, namely, the depth direction and the horizontal direction. The signal processing device according to claim 2.

4. Comprising a coordinate conversion unit that performs coordinate conversion on the three-dimensional point cloud data, The frequency data generation unit generates the frequency data based on the three-dimensional point cloud data after coordinate conversion by the coordinate conversion unit. The signal processing device according to claim 1.

5. Comprising a synthesis unit that synthesizes a plurality of three-dimensional point cloud data obtained by three-dimensionally measuring the target space from different viewpoints, The frequency data generation unit generates the frequency data based on the three-dimensional point cloud data after synthesis by the synthesis unit. The signal processing device according to claim 1.

6. The data reduction unit performs a process of extracting and outputting point cloud data in which the frequency in the frequency data is equal to or higher than a threshold value from the three-dimensional point cloud data. The signal processing device according to claim 1.

7. The frequency data generation unit generates, as the frequency data, data indicating the frequency of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction, The data reduction unit performs a process of extracting and outputting point cloud data in which the frequency is equal to or higher than a threshold value in the frequency data indicating the frequency of points in the two directions generated by the frequency data generation unit. The signal processing device according to claim 6.

8. The frequency data generation unit generates, as the frequency data, data indicating the frequency of points in two directions, namely, the depth direction and the direction orthogonal to the depth direction, The data reduction unit extracts point cloud data in which the frequency in the frequency data is equal to or higher than a threshold value from the three-dimensional point cloud data, and outputs two-dimensional data obtained by projecting the extracted point cloud data onto a two-dimensional plane defined by the axis in the depth direction and the axis in the orthogonal direction. The signal processing device according to claim 1.

9. The depth data generation unit generates, as the frequency data, data indicating the frequencies of points in two directions, i.e., the depth direction and the direction orthogonal to the depth direction. The data reduction unit extracts, from the three-dimensional point cloud data, point cloud data in which the frequency in the frequency data is equal to or greater than a threshold value, and extracts and outputs only the coordinate information in the depth direction of the extracted point cloud data. The signal processing device according to claim 1.

10. A signal processing device generates frequency data, which is data indicating the frequencies of points at least in the depth direction, based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space. performs data reduction processing on the three-dimensional point cloud data based on the frequency data. A signal processing method.

11. A data manufacturing method for manufacturing control data used in a control system that performs control based on three-dimensional point cloud data obtained by three-dimensionally measuring a target space, including the three-dimensional point cloud data and frequency data generated based on the three-dimensional point cloud data, the frequency data being data indicating the frequencies of points at least in the depth direction. The control data used when the control system performs data reduction processing on the three-dimensional point cloud data based on the frequency data is manufactured. A data manufacturing method.

Images