Control device and control method
By estimating the ground orientation and adjusting the imaging direction, the problem of low three-dimensional measurement accuracy on the sloped ground is solved, the number and resolution of point cloud samples are improved, and high-precision three-dimensional measurement is achieved.
Patent Information
- Application Number
- CN202380028137.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-24
- Filing Date
- 2023-03-15
- Publication Date
- 2024-11-26
AI Technical Summary
In three-dimensional measurements, sloped ground causes a decrease in the area of ground captured image during vertical imaging and a decrease in the number of point cloud samples, reducing the accuracy of measurement.
By estimating the orientation information of the ground, adjusting the imaging direction to image in a direction corresponding to the orientation of the ground, a three-dimensional point cloud is generated, and appropriate sensing control is performed on the sloped ground to increase the number of point cloud samples and resolution.
The three-dimensional measurement accuracy on the slope ground is improved, the resolution of point cloud data is increased, and the high-precision three-dimensional measurement of the target area is ensured.
Smart Images

Figure CN119032253A_ABST
Abstract
Description
Technical Field
[0001] The present technology relates to a control device and a method for controlling the device, and more particularly, to a control technology for sensing in the case of three-dimensionally measuring a target area on the basis of sensing information obtained by sensing a ground side from the air. Background Art
[0002] For example, in a technology for achieving three-dimensional measurement of a target area, a three-dimensional point cloud representing the three-dimensional structure of the target area is generated based on sensing information obtained by sensing the ground side from the air using a flying body of a drone (for example, see Patent Document 1).
[0003] Patent Document 1 discloses generating a three-dimensional point cloud based on an image of a ground surface captured by a laser ranging sensor installed in a flying object (the captured image is hereinafter referred to as “imaging”).
[0004] In addition, patent document 1 discloses a technology in which, on the premise that each area in the target area is imaged at a certain flight altitude, a three-dimensional point cloud can be generated with the minimum number of captured images by controlling the imaging interval according to the height above the ground plane to keep the degree of overlap / side overlap between the captured images constant.
[0005] [Citation List]
[0006] [Patent Document]
[0007] [Patent Document 1]
[0008] Japanese Patent Publication No. 2019-214294 Summary of the invention
[0009] [Technical issues]
[0010] Here, as also shown in the above-mentioned Patent Document 1, in three-dimensional measurement, the imaging direction (sensing direction) from the air is set to a direction perpendicular to the horizontal plane to image the ground side.
[0011] However, the ground to be measured may include a ground that is a slope with a continuously changing ground height. When imaging in the vertical direction as described above is performed on a ground such as a slope, the ground in the captured image tends to have a reduced area, where the number of samples in the point cloud is reduced.
[0012] In view of the above, an object of the present technology is to improve the accuracy of three-dimensional measurement when the area to be measured includes a slope.
[0013] [Solution to the problem]
[0014] A control device according to the present technology includes: a point cloud generation unit that generates a three-dimensional point cloud representing a three-dimensional structure of a target area based on sensing information obtained by sensing the ground side from the air; an estimation unit that estimates the orientation of the ground based on a captured image obtained by imaging the ground side from the air; and a control unit that performs control related to sensing for generating a three-dimensional point cloud of the target area based on information about the orientation of the ground estimated by the estimation unit.
[0015] Therefore, when a three-dimensional point cloud of a target area is generated based on sensing information targeting the ground, appropriate sensing control corresponding to the orientation of the ground may be performed, for example, sensing may be performed at an orientation aligned with the orientation of the ground.
[0016] In addition, the control method according to the present technology is a control method performed by a signal processing device, the method including: generating a three-dimensional point cloud indicating a three-dimensional structure of a target area based on sensing information obtained by sensing the ground side from the air; estimating the orientation of the ground based on a captured image obtained by imaging the ground side from the air, and performing control related to sensing for generating a three-dimensional point cloud of the target area based on information about the estimated orientation of the ground.
[0017] This control method also produces effects similar to those exerted by the control device according to the present technology described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is a diagram showing an overview of the configuration of a measurement system according to an embodiment of the present technology.
[0019] Figure 2 It is an explanatory diagram showing the three-dimensional measurement technology according to the embodiment.
[0020] Figure 3 is an explanatory diagram showing a reduction in the number of samples in a point cloud of a slope.
[0021] Figure 4 It is an explanatory diagram showing the three-dimensional measurement technology of the first embodiment.
[0022] Figure 5 : is a block diagram showing a configuration example of a control device as a first embodiment.
[0023] Figure 6 is a diagram showing an embodiment of a technology for generating a three-dimensional point cloud based on a range image.
[0024] Figure 7 1 is a block diagram showing a configuration example of a ground direction estimation unit in the first embodiment.
[0025] Figure 8It is an explanatory diagram showing an embodiment of a normal vector calculation technique performed by a normal vector calculation unit.
[0026] Fig. 9 It is an explanatory diagram showing an embodiment of a technique for generating an angle image executed by an angle image generating unit.
[0027] Fig.10 1 is an explanatory diagram showing an example of a calculation technique for a representative angle in an embodiment.
[0028] Fig.11 1 is a flowchart showing an example of a processing procedure for realizing the three-dimensional measurement technology as the first embodiment.
[0029] Fig.12 : is a block diagram showing a configuration example of a control device as a second embodiment.
[0030] Fig.13 : is a flowchart showing an example of a processing procedure for realizing the three-dimensional measurement technology as the second embodiment.
[0031] Fig.14 It is an explanatory diagram showing a three-dimensional measurement technique as a third embodiment.
[0032] Fig.15 is a block diagram showing a configuration example of a control device as a previous device.
[0033] Fig.16 is a block diagram showing an example of a configuration of a control device as a subsequent device.
[0034] Fig.17 : is a flowchart showing an example of a processing procedure for realizing the processing as the previous means in the third embodiment.
[0035] Fig.18 : is a flowchart showing an example of a processing procedure for realizing processing as a subsequent device in the third embodiment.
[0036] Fig.19 : is a block diagram showing a configuration example of a control device as a modification.
[0037] Fig. 20 : is a block diagram showing a configuration example of a ground direction estimation section included in the control device as a modification. DETAILED DESCRIPTION
[0038] With reference to the drawings, embodiments according to the present technology will hereinafter be described in the following order.
[0039] <1. First Embodiment>
[0040] (1-1. System Overview)
[0041] (1-2. Configuration of control device)
[0042] (1-3. Processing procedures)
[0043] <2. Second Embodiment>
[0044] <3. Third Embodiment>
[0045] <4. Modifications>
[0046] <5. Conclusion of Implementation Method>
[0047] <6. This technology>
[0048] <1. First Embodiment>
[0049] (1-1. System Overview)
[0050] Figure 1 is a diagram showing an overview of the configuration of a measurement system according to an embodiment of the present technology.
[0051] As shown in the figure, the measurement system as an embodiment includes at least an imaging control device 1 and a moving body M.
[0052] The imaging control device 1 is configured as a computer device including an imaging section (imaging section 2 described below) that obtains a captured image, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.
[0053] Here, "imaging" as used herein refers to sensing performed by a sensor including pixels arranged two-dimensionally, each pixel including a light receiving element. In addition, "captured image" refers to an image obtained by sensing performed by such a sensor, that is, an image indicating information for each pixel based on a light receiving signal.
[0054] In the first embodiment, the imaging unit is provided with an RGB sensor (RGB sensor 21 described below) for obtaining an RGB image and a distance sensor ("distance sensor 22" described below) for obtaining a distance image. The RGB image represents an image (color image) indicating an R (red) illuminance value, a G (green) illuminance value, and a B (blue) illuminance value for each pixel. The distance image represents an image indicating information related to the distance to an object for each pixel.
[0055] The mobile body M broadly refers to an object that can move in a state where the imaging control device 1 is mounted (supported) thereon. In the present embodiment, the mobile body M is a flying object, for example, a drone, an airplane, a helicopter, and the like.
[0056] In the measurement system of the present embodiment, when the mobile body M equipped with the imaging control device 1 is moving in the air above the target area At, the imaging control device 1 images the target area At for measurement. Then, based on the captured image, point cloud data indicating the three-dimensional structure of the target area At is generated.
[0057] Will refer to Figure 2 Describe specific techniques used for three-dimensional measurement.
[0058] As shown in the figure, in the present embodiment, a plurality of imaging execution points Pi are set in the target area At. In the present embodiment, as a flight route Rf for the mobile body M, a flight route Rf passing through the plurality of imaging execution points Pi thus set in the target area At is designated, and the mobile body M is made to fly according to the flight route Rf, and the imaging control device 1 images the ground side at each imaging execution point Pi to generate a three-dimensional point cloud based on the captured image of each imaging execution point Pi. Then, in the measurement system, point cloud data representing the three-dimensional structure of the entire target area At is generated based on information about the three-dimensional point cloud obtained for each imaging execution point Pi. In other words, measurement data indicating the three-dimensional measurement result of the entire target area At is obtained.
[0059] Here, also as described above, the ground may be a slope, and in the case where the ground side is imaged in a direction perpendicular to the horizontal surface relative to the ground as a slope, the ground in the captured image may have a reduced area where the number of samples in the point cloud is reduced.
[0060] Figure 3 is an explanatory diagram indicating that the number of samples in the point cloud decreases with a slope as described above. Figure 3 In A, under the premise that the imaging direction is perpendicular to the horizontal surface, and in the case where the ground to be imaged is approximately horizontal, the sampling density of the point cloud is schematically represented by “D1”. Figure 3 In B, under the same premise, when the ground to be imaged is a sloped ground, the sampling density is schematically represented by “D2”.
[0061] Figure 3 A and Figure 3 The comparison between B shows that for slopes, the number of samples in the point cloud tends to decrease.
[0062] In view of the above, in the case where the target area At includes a slope, the slope may reduce the accuracy of three-dimensional measurement.
[0063] Therefore, in this embodiment, the following technology is adopted as a technology for three-dimensional measurement.
[0064] First, basically, Figure 4As shown in A of , imaging is performed with the imaging direction set to be perpendicular to the horizontal surface. At this time, the imaging control device 1 generates a three-dimensional point cloud based on the captured image of the ground surface. In the case where the ground is not a slope, information about the three-dimensional point cloud can be regarded as appropriate information.
[0065] Then, in the present embodiment, the orientation of the ground is estimated based on the captured image obtained by imaging in the imaging direction set to be perpendicular to the horizontal plane as described above, and based on the information about the estimated orientation of the ground, it is determined whether the ground to be imaged is a slope. In the case where the ground is determined not to be a slope, the imaging control device 1 waits for reaching the lower imaging execution point Pi. In other words, in the case where the ground is not a slope, the point cloud data of the entire target area At is generated using the information about the three-dimensional point cloud generated by imaging in the imaging direction set to be perpendicular to the horizontal plane as described above.
[0066] On the other hand, in the case where the ground is determined to be a slope, the imaging control device 1 controls the imaging direction to an orientation corresponding to the orientation of the ground, images the ground, and generates a three-dimensional point cloud based on the captured image. Then, after generating a three-dimensional point cloud based on the captured image obtained with the imaging direction set to an orientation corresponding to the orientation of the ground, the imaging control device 1 waits for reaching the next imaging execution point Pi.
[0067] (1-2. Configuration of control device)
[0068] Figure 5 : is a block diagram showing a configuration example of an imaging control device 1 as a first embodiment for realizing the three-dimensional measurement technology as the three-dimensional measurement technology of the first embodiment as described above.
[0069] As shown in the figure, the imaging control device 1 includes an imaging section 2, a signal processing section 3, a storage section 4, a control section 5, a position sensor 6, an actuator 7, and an IMU (Inertial Measurement Unit).
[0070] The imaging unit 2 is configured to obtain a captured image for generating at least a three-dimensional point cloud. In the present embodiment, assuming point cloud data as a result of three-dimensional measurement indicating the target area At, for each point, data (X, Y, Z, R, G, B) indicating three-dimensional coordinate information (X, Y, Z) and information (R, G, B) as a brightness value of each of R, G, and B are generated.
[0071] Therefore, the imaging section 2 in the present embodiment is provided not only with the distance measuring sensor 22 for obtaining a captured image as a distance image for generating a three-dimensional point cloud, but also with the RGB sensor 21 for obtaining the captured image as an RGB image.
[0072] The imaging section 2 includes a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor as an RGB sensor 21 in which R pixels, G pixels, and B pixels are two-dimensionally arranged in an array according to a predetermined rule, for example, a Bayer array.
[0073] In addition, the distance measuring sensor 22 is assumed to be a type in which pixels having light receiving elements are two-dimensionally arranged. For example, the sensor used is compatible with distance measuring based on a ToF scheme such as an iToF (indirect time of flight) scheme or a dToF (direct ToF) scheme.
[0074] It should be noted that when the ranging sensor 22 used is compatible with the ToF scheme, the ranging sensor 22 is provided with a projector part for obtaining reflected light from the object (for example, a projector part for projecting infrared light, etc.), and since the projector part is not very relevant to the essence of the present technology, it is not shown in the figure.
[0075] The imaging section 2 includes an RGB image generating section 23 and a distance image generating section 24 , as well as an RGB sensor 21 and a distance measuring sensor 22 .
[0076] The RGB image generation section 23 performs color interpolation processing such as demosaic processing on the captured image based on the RAW image output from the RGB sensor 21 to generate an RGB image indicating a luminance value of each of R, G, and B on a per-pixel basis.
[0077] The distance image generation section 24 performs a predetermined operation for distance calculation according to the ToF scheme based on a light reception signal of each pixel obtained by the distance measuring sensor 22 performing a light reception operation corresponding to the ToF scheme to generate a distance image.
[0078] The signal processing unit 3 includes a processor, for example, a DSP (digital signal processor), an FPGA (field programmable gate array), etc., and performs various processing operations for generating point cloud data representing the results of three-dimensional measurement of the target area At based on the images captured from the imaging unit 2 (i.e., the distance image and the RGB image in the present embodiment).
[0079] Specifically, the signal processing unit 3 includes functions as a three-dimensional point cloud generating unit 3a, a ground direction estimating unit 3b, and a point cloud data generating unit 3c.
[0080] The three-dimensional point cloud generating unit 3 a generates a three-dimensional point cloud based on the range image obtained by the imaging unit 2 .
[0081] refer to Figure 6 , an embodiment of a technology for generating a three-dimensional point cloud based on a range image will be described.
[0082] As described above, the distance image is an image indicating the distance from each pixel to the object (the symbol of the distance is hereinafter referred to as "z"). In the case where the coordinate system of the distance image is a (u, v) coordinate system, the value z of each pixel in the distance image can be expressed as z(u, v) as shown in the figure.
[0083] In addition, as shown in the figure, the position of the optical axis of the imaging optical system that forms a distance image using the distance measuring sensor 22 is expressed as (c x , c y ), and the focal length of the imaging optical system is represented by "f".
[0084] The generation of the 3D point cloud based on the range image is performed as using the camera parameters as the above optical axis position (c x , c y ) and focal length f to convert the value z (u, v) of each pixel into coordinate information (X, Y, Z) of a point in three-dimensional space, as shown below.
[0085] X=(c x -u)×(z(u,v)) / f
[0086] Y=(c y -u)×(z(u,v)) / f
[0087] Z=z(u,v)
[0088] exist Figure 5 In the embodiment, the ground position estimation unit 3b estimates the position of the ground based on the captured image obtained by imaging the ground side from the air. Specifically, as described above, the ground position estimation unit 3b in the present embodiment estimates the position of the ground based on the information about the three-dimensional point cloud generated by the three-dimensional point cloud generation unit 3a based on the range image.
[0089] In addition, the processing of the ground direction estimation unit 3b will be described below.
[0090] The point cloud data generating unit 3c generates point cloud data (X, Y, Z, R, G, B) as a result of three-dimensional measurement of the target area At based on the information (X, Y, Z) about the three-dimensional point cloud generated by the three-dimensional point cloud generating unit 3a and the RGB image (R, G, B) obtained by the imaging unit 2.
[0091] The storage section 4 includes a nonvolatile memory such as an HDD (Hard Disk Drive), an SDD (Solid State Drive), etc., and is mainly used to store data used for signal processing by the signal processing section 3. For example, the storage section 4 is used to store a range image and an RGB image.
[0092] The control section 5 includes a microcomputer including, for example, a CPU, a ROM, a RAM, and the like, and the CPU controls the entire imaging control apparatus 1 by executing processing based on a program stored in a predetermined storage device (for example, the above-mentioned ROM).
[0093] The control unit 5 is connected to the position sensor 6 and the actuator 7 .
[0094] The position sensor 6 includes a GNSS (Global Navigation Satellite System) sensor, for example, a GPS (Global Positioning System) sensor or the like, to detect the position of the imaging control device 1 .
[0095] Note that the configuration for detecting the position of the imaging control device 1 is not limited to the configuration using the GNSS sensor. For example, the position may be detected by a technique such as SLAM (Simultaneous Positioning and Mapping) or UWB (Ultra Wideband).
[0096] The actuator 7 is configured as an actuator such as a motor for driving a mechanism (imaging direction changing mechanism) for changing the imaging direction of the imaging unit 2 .
[0097] In the present embodiment, driving the actuator 7 enables the imaging direction of the imaging section 2 to be changed into the pan direction and the tilt direction.
[0098] Based on the position information about the imaging control device 1 detected by the position sensor 6, the control section 5 can determine that the reference position has been reached. Figure 2 The imaging described is performed at point Pi.
[0099] In the signal processing unit 3 , the control unit 5 can change the imaging direction of the imaging unit 2 to a direction corresponding to the ground direction by controlling the drive of the actuator 7 based on the information on the ground direction estimated by the ground direction estimation unit 3 b .
[0100] The control section 5 realizes the above-mentioned operation by controlling the operation of the imaging section 2 and the signal processing section 3 and driving the actuator 7 based on the position information about the imaging control device 1 detected by the position sensor 6. Figure 4 The three-dimensional measurement technique of the first embodiment is described. The specific processing executed by the control unit 5 to realize the three-dimensional measurement technique will be described below.
[0101] The IMU 8 includes a motion sensor such as an acceleration sensor or a gyro sensor (angular velocity sensor), and detects the inclination of the imaging section 2 in each of the yaw direction, the pitch direction, and the roll direction as attitude information on the imaging section 2 .
[0102] The attitude information detected by the IMU 8 is fed to the signal processing section 3 and used for a process of estimating the ground orientation, which is performed by the ground orientation estimating section 3b as described below.
[0103] Figure 7 2 is a block diagram showing a configuration example of the ground direction estimating section 3 b.
[0104] As shown in the figure, the ground orientation estimation unit 3 b includes a normal vector calculation unit 31 , an angle image generation unit 32 , a binarization processing unit 33 , a mask unit 34 , and a representative angle calculation unit 35 .
[0105] The three-dimensional point cloud generated by the three-dimensional point cloud generating unit 3a is input to the normal vector calculating unit 31. The normal vector calculating unit 31 determines the normal vector of each position on the ground based on the positional relationship between mutually adjacent points in the three-dimensional point cloud.
[0106] refer to Figure 8 , an embodiment of a calculation technique of a normal vector by the normal vector calculation section 31 will be described.
[0107] In this embodiment, a normal vector is calculated for each point in the three-dimensional point cloud.
[0108] In this case, based on the coordinate information (indicated as X in the figure) about the point on the three-dimensional space determined for each pixel in the range image cam , Y cam and Z cam ) determines the normal vector. Specifically, based on the three-dimensional coordinate information (X, Y, Z) about three pixels including the pixel to be processed in the distance image (expressed as (i, j) in the camera coordinate system), one of the two pixels adjacent to the pixel to be processed in the vertical direction (v direction) (the pixel is hereinafter referred to as the “vertically adjacent pixel”), and one of the two pixels adjacent to the pixel to be processed in the horizontal direction (u direction) (the pixel is hereinafter referred to as the “horizontally adjacent pixel”), the normal vector A(i, j) of the surface of the triangle formed by connecting the points of the three pixels is determined.
[0109] Based on the vector H(i, j) from the point of the pixel to be processed to the point of the vertically adjacent pixel and the vector W(i, j) from the point of the pixel to be processed to the point of the horizontally adjacent pixel, the normal vector A(i, j) can be calculated by the following equation:
[0110] Normal vector A(i, j)
[0111] =(W(i,j)×H(i,j)) / |W(i,j)×H(i,j)|
[0112] The normal vector A(i, j) as described above is calculated for each point in the range image (in other words, for each point in the generated three-dimensional point cloud).
[0113] Note that the normal vector A is not limited to being determined for each point in the three-dimensional point cloud. For example, a technique similar to the technique described above can be used to determine the normal vector A for each region containing multiple points. This does not change the fact that the normal vector is determined for each position on the ground.
[0114] exist Figure 7 In the embodiment, based on the normal vector A(i, j) calculated by the normal vector calculation section 31, the angle image generation section 32 generates an angle image which is an image indicating the angle of the ground orientation for each pixel.
[0115] Fig. 9 This is an explanatory diagram of an embodiment of a technique for generating an angle image executed by the angle image generating unit 32 .
[0116] Here, the normal vector A(i, j) calculated by the normal vector calculation unit 31 for each pixel in the distance image is represented as normal vector A cam .
[0117] As shown in the figure, by calculating the normal vector A of each pixel cam and the perpendicular vector h cam The inner product of the vertical vector h is used to generate the angle image. cam is a vector indicating the vertical direction in the camera coordinate system (u, v coordinate system). In other words, since the present embodiment assumes that the distance image used as the basis for generating the angle image is captured with the imaging direction of the imaging section 2 set to a direction perpendicular to the horizontal direction, the vertical vector h cam is a vector indicating the direction perpendicular to the horizontal plane.
[0118] According to the "θ=cons-1[A cam h cam ]", an angle image is generated by calculating the angle θ that indicates the ground direction for each pixel.
[0119] Here, the distance image used as the basis for generating the angle image is not necessarily captured in the imaging direction perpendicular to the horizontal plane. Figure 5 The posture information (yaw, pitch, and roll) about the imaging unit 2 detected by the IMU 8 shown in FIG. 8 is used to correct the vertical vector h for generating the angle image. cam Therefore, even if the imaging direction of the imaging unit 2 deviates from the direction perpendicular to the horizontal surface when capturing a distance image used as a basis for generating an angle image, the deviation can be corrected and an appropriate angle image representing the angle θ relative to the direction perpendicular to the horizontal surface can be obtained.
[0120] exist Figure 7In the present invention, a binarization processing section 33, a masking section 34, and a representative angle calculation section 35 are provided to determine a representative angle representing a representative angle θ of the ground surface to be imaged based on the angle image.
[0121] Fig.10 It is an explanatory diagram of an example of a calculation technique of a representative angle in the embodiment.
[0122] Fig.10 A schematically shows an angle image generated by the angle image generating unit 32 .
[0123] For Fig.10 The binarization processing unit 33 generates a binarized image in which pixels having an angle θ (absolute value) greater than a predetermined angle threshold value are marked as "1" and other pixels are marked as "0" (see Fig.10 B).
[0124] For the binarized image obtained by the binarization processing section 33, the mask section 34 generates a mask image in which, among image regions where the angle θ is greater than or equal to the angle threshold, image regions whose areas are smaller than a predetermined area threshold are masked (see Fig.10 In other words, the image region remaining after masking has an angle θ greater than or equal to the angle threshold and an area greater than or equal to the area threshold.
[0125] The representative angle calculation section 35 calculates a representative angle based on the mask image generated by the mask section 34 and the angle image generated by the angle image generation section 32. Specifically, the representative angle calculation section 35 specifies a representative point of an image area having an angle θ greater than or equal to an angle threshold value in the mask image (see Fig.10 The representative point can be defined as the center point of the image region in the mask image, which has an angle θ greater than or equal to the angle threshold.
[0126] Furthermore, the representative angle calculation section 35 specifies the angle θ in the pixel serving as the representative point as the representative angle (see Fig.10 the lower part of D).
[0127] Note that in the case where the mask image includes a plurality of image regions having an angle θ greater than or equal to the angle threshold, the representative point may be designated as the center point of one of the image regions having the largest area.
[0128] In addition, the representative angle is not limited to the angle θ of the representative point as described above. For example, the representative angle may be calculated by using the average value of the angle θ of each pixel having an angle θ greater than or equal to the angle threshold value belonging to a specific image area in the mask image as the representative angle.
[0129] Here, in the first embodiment, the fact that the representative angle calculation section 35 has calculated the representative angle indicates that the ground whose orientation is to be estimated (ie, the ground whose distance image has been captured to estimate the orientation of the ground) is a slope.
[0130] Therefore, in the first embodiment, the control unit 5 can determine whether the target ground surface is a slope according to whether the ground orientation estimation unit 3 b has calculated the representative angle.
[0131] (1-3. Processing procedures)
[0132] Fig.11 1 is a flowchart showing an example of a processing procedure executed by the control section 5 to realize the three-dimensional measurement technology as the first embodiment.
[0133] It should be noted that, in this embodiment, when executing Fig.11 In the process described in , the direction of imaging by the imaging section 2 is assumed to be a direction perpendicular to the horizontal plane.
[0134] First, in step S101, the control section 5 determines whether the imaging execution point Pi has been reached. That is, based on the position information on the imaging control device 1 detected by the position sensor 6, the imaging control device 1 determines whether the imaging control device 1 has reached any imaging execution point Pi.
[0135] In the case where it is determined in step S101 that the imaging execution point Pi has not been reached, the control unit 5 proceeds to step S102 to determine whether the processing is terminated, that is, whether a predetermined condition is satisfied under which the processing of the three-dimensional measurement of the entire target area At is terminated, for example, whether a predetermined operation is input through an external device or a predetermined notification is provided. In step S102, in the case where it is determined that the processing is not terminated, the control unit 5 returns to step S101.
[0136] As a result of the processing in steps S101 and S102 described above, the control section 5 waits for reaching the imaging execution point Pi or the end of the processing.
[0137] In step S101, in the case where it is determined that the imaging execution point Pi has been reached, the control section 5 proceeds to step S103 to give an instruction to perform imaging and processing by the signal processing section 3. That is, the control section 5 instructs the execution imaging section 2 to capture the image of the RGB image and the range image and the processing of the signal processing section 3 as the three-dimensional point cloud generation section 3a and the ground orientation estimation section 3b.
[0138] In step S104 following step S103, the control section 5 determines whether the representative angle has been calculated. That is, the processing as the ground orientation estimation section 3b as described above is performed to determine whether the representative angle has been calculated.
[0139] In the case where it is determined in step S104 that the representative angle has not been calculated (in other words, in the case where it is determined that the ground surface is not a slope), the control section 5 returns to step S101.
[0140] Thus, when the ground to be imaged is not a slope, wait until the next imaging execution point Pi is reached. That is, when the ground is not a slope, point cloud data about the entire target area At is generated using information about a three-dimensional point cloud generated by imaging in an imaging direction set to be perpendicular to the horizontal plane.
[0141] On the other hand, in the case where it is determined in step S104 that the representative angle has been calculated (in the case where it is determined that the ground is a slope), the control unit 5 proceeds to step S105 to execute processing for changing the imaging direction to a predetermined direction. In this example, it is assumed that the inclination direction of the slope includes only the left and right direction among the front, rear, left, and right directions. In addition, the first embodiment does not need to set the imaging direction to be directly opposite to the ground as in the second embodiment described below, and only requires at least the orientation of the imaging direction close to the ground. Specifically, in the case where the ground is a slope that rises obliquely to the right (see Figure 3 B and Figure 4 B), that is, when the representative angle is a positive angle, the actuator 7 is driven and controlled in such a way that the imaging direction is tilted to the right by a preset angle (in other words, an angle of 0 degrees) from the direction perpendicular to the horizontal plane. On the other hand, when the ground is a slope that rises obliquely to the left (that is, the representative angle is a negative angle), the actuator 7 is driven and controlled in such a way that the imaging direction is tilted to the left by a preset angle from the direction perpendicular to the horizontal plane.
[0142] In step S106 following step S105, the control section 5 gives an instruction for the signal processing section to perform imaging and processing. The processing in step S106 is the same as the processing in step S103 described above.
[0143] In the case where the ground is a slope, execution of the process in step S106 causes the imaging section 2 to capture an RGB image and a range image in an imaging direction corresponding to the orientation of the ground, and to generate a three-dimensional point cloud based on the captured range image.
[0144] It should be noted that in step S106 , the instruction to the signal processing section 3 to perform its processing may be only an instruction to the three-dimensional point cloud generating section 3 a to perform its processing.
[0145] In step S107 following step S106, the control unit 5 performs processing for changing the imaging direction back to the vertical direction, that is, processing for driving and controlling the actuator 7 to change the imaging direction back to the direction vertical to the horizontal plane. Then, the control unit 5 returns to step S101.
[0146] This allows, when the next imaging execution point Pi is reached, the processing in step S103 to be performed with the imaging direction set to the vertical direction.
[0147] Furthermore, according to the processing flow of the above-mentioned steps S104→S105→S106→S107→S101, in the case where the ground is a slope, imaging is performed while controlling the imaging direction to an orientation corresponding to the orientation of the ground, a three-dimensional point cloud is generated based on the captured distance image, and then waiting for reaching the next imaging execution point Pi. Thus, for the imaging execution point Pi which is a slope, information on the three-dimensional point cloud generated when generating point cloud data about the entire target area At can be used, wherein the camera direction is set to an orientation corresponding to the orientation of the ground.
[0148] If it is determined in step S102 that the processing is completed, the control unit 5 ends Fig.11 A series of processing operations shown in .
[0149] It should be noted that since the above-described embodiment assumes that the tilt direction of the ground includes only the right direction and the left direction among the front, rear, right, and left directions, control of the imaging direction is performed only in the right direction and the left direction, but control for tilting the imaging direction in the front-rear direction may also be performed in association with the tilt in the front-rear direction. Alternatively, control for tilting the imaging direction in each of the front, rear, right, and left directions may be performed in association with the tilt in each of the front, rear, right, and left directions.
[0150] <2. Second Embodiment>
[0151] Next, a second embodiment is described. In the second embodiment, when the ground is a slope, the imaging direction is controlled to be a direction indicated by a target angle determined adaptively with respect to the orientation of the ground.
[0152] Fig.12 1 is a block diagram showing a configuration example of an imaging control device 1A as the second embodiment.
[0153] In the following description, the same parts as those already described are denoted by the same symbols or the same step numbers, and the description thereof is omitted.
[0154] In contrast, the imaging control device 1A is different from the imaging control device 1 in that a control section 5A is provided instead of the control section 5 .
[0155] The control unit 5A is similar to the control unit 5 in terms of hardware configuration, but is different from the control unit 5 in that the control unit 5A performs the following Fig.13 In the processing, instead of the above Fig.11 in the processing.
[0156] Fig.13 1 is a flowchart showing an example of a processing procedure executed by the control section 5A to realize the three-dimensional measurement technology as the second embodiment.
[0157] This process is similar to Fig.11 The processing described in is different in that the processing in step S201 is performed instead of the processing in step S105.
[0158] Specifically, in step S201, the control unit 5A performs processing for controlling the imaging direction to be directly opposite to the ground based on the representative angle, that is, the actuator 7 is driven and controlled so that the angle of the imaging direction changes from an angle perpendicular to the horizontal plane (0 degrees) to an angle consistent with the representative angle.
[0159] Note that in the second embodiment, the control of the imaging direction when the ground is a slope is not limited to the control of setting the target angle to an angle in the direction directly opposite to the ground as described above, and the replaceable possible control includes, for example, a control of setting the target angle to an angle offset from a preset angle (i.e., a representative angle in the present embodiment) in the direction directly opposite to the ground.
[0160] Note that the above description assumes that in the case where the mask image includes a plurality of image areas having an angle θ greater than or equal to the angle threshold, only the image area having the largest area (i.e., the representative angle is set) is the same as in the case of the first embodiment described above, but in the second embodiment, in the case where the mask image includes a plurality of image areas having an angle θ greater than or equal to the angle threshold, a representative point and a representative angle may be specified for each image area. In this case, at the imaging execution point Pi determined as a slope, imaging may be performed for each representative angle, wherein the imaging direction is at an angle determined adaptively relative to the representative angle.
[0161] <3. Third Embodiment>
[0162] The third embodiment reduces the time required to three-dimensionally measure the target area At by using two imaging control devices (a former imaging control device and a latter imaging control device).
[0163] Specifically, Fig.14 As shown, the third embodiment assumes the use of two imaging control devices, namely, the imaging control device 1B as a preceding device in front on the flight route Rf and the imaging control device 1C as a following device following the imaging control device 1B on the flight route Rf.
[0164] In the third embodiment, the imaging control device 1B as the preceding device performs imaging in the imaging direction perpendicular to the horizontal plane at each imaging execution point Pi, generates a three-dimensional point cloud based on the captured image, and estimates the orientation of the ground. In addition, in the case where the imaging control device 1B determines that the ground is a slope due to the estimation of the orientation of the ground, it notifies the imaging control device 1C as the subsequent device that the imaging execution point is a slope.
[0165] In other words, in the third embodiment, the imaging control device 1B as the preceding device performs a function of generating a three-dimensional point cloud for a non-slope and a function of notifying a subsequent device of the detection if it detects a slope.
[0166] At the same time, at the imaging execution point Pi where the ground is a slope, when the imaging control device 1B as the previous device notifies the imaging control device 1C as the subsequent device, the imaging control device 1C performs imaging in the imaging direction set to the direction corresponding to the orientation of the ground and generates a three-dimensional point cloud based on the captured image.
[0167] In the above-mentioned first and second embodiments, when the imaging execution point Pi is a slope, after performing imaging and generation of a three-dimensional point cloud with the imaging direction set to a vertical direction, the imaging control device 1 or 1A needs to perform imaging and generation of a three-dimensional point cloud with the imaging direction changed to a direction corresponding to the ground orientation.
[0168] On the contrary, according to the three-dimensional measurement technology of the third embodiment as described above, even in the case where the imaging execution point Pi is a slope, the imaging control device 1B as the previous device only needs to perform imaging and generation of the three-dimensional point cloud once, and the imaging control device 1C as the subsequent device only needs to perform imaging and generation of the three-dimensional point cloud of the slope once.
[0169] Therefore, the third embodiment can reduce the length of time the imaging control device stays at the imaging execution point Pi as a ramp, compared with the cases of the first and second embodiments, enabling reduction of the time required to three-dimensionally measure the target area At.
[0170] Fig.15 is a block diagram showing a configuration example of an imaging control device 1B as the former device, and Fig.16 : is a block diagram showing an example of a configuration of an imaging control device 1C as the latter device.
[0171] exist Fig.15 In comparison, the imaging control device 1B is different from the imaging control device 1 described in the first embodiment in that a control section 5B is provided instead of the control section 5 , a communication section 9 is added, and the actuator 7 is omitted.
[0172] The communication section 9 is provided to perform data communication with the imaging control device 1C as a subsequent device, and, for example, can perform wireless data communication with the imaging control device 1C.
[0173] The control section 5B is similar to the control section 5 in hardware configuration, but is different from the control section 5 in that the control section 5B executes processing as a previous means for realizing the three-dimensional measurement technique of the third embodiment as described above.
[0174] exist Fig.16 In comparison, the imaging control device 1C is different from the imaging control device 1 described in the first embodiment in that a control unit 5C is provided instead of the control unit 5, a communication unit 9 is added, a signal processing unit 3C is provided instead of the signal processing unit 3, and the IMU 8 is omitted.
[0175] The communication section 9 is provided to perform data communication with the imaging control device 1B as the preceding device, and, for example, wireless data communication may be performed with the communication section 9 of the imaging control device 1B.
[0176] In contrast, the signal processing unit 3C is different from the signal processing unit 3 in that the function as the ground direction estimation unit 3b is omitted.
[0177] The control section 5C is similar to the control section 5 in hardware configuration, but is different from the control section 5 in that the control section 5C executes processing as a subsequent device for realizing the three-dimensional measurement technique of the third embodiment as described above.
[0178] Fig.17 : is a flowchart showing an example of a processing procedure executed by the control section 5B to realize the processing as the previous means in the third embodiment.
[0179] The processing process is the same as above Fig.11 The processing of the control section 5 shown in FIG. 1 is different in that the processing in step S301 is performed instead of the processing from step S105 to step S106 .
[0180] Specifically, in step S301, the control section 5B performs processing for notifying the subsequent device of the current imaging execution point Pi and the representative angle. That is, the control section 5B performs processing for notifying the imaging control device 1C (control section 5C) of information indicating the current imaging execution point Pi and information about the calculated representative angle determined in step S104 via the communication section 9.
[0181] Fig.18 : is a flowchart showing an example of a processing procedure executed by the control section 5C to realize a process as a subsequent device in the third embodiment.
[0182] The processing process is the same as above Fig.13 The processing of the control section 5A shown in is different in that the processing in step S103 is omitted, the processing in step S401 is performed instead of the processing in step S104, and the processing in step S107 is omitted.
[0183] In the case where it is determined in step S101 that the imaging execution point Pi has been reached, the control section 5C determines in step S401 whether the representative angle of the current imaging execution point Pi has been notified to the control section 5C. This corresponds to determining whether the current imaging execution point Pi is a slope based on whether the notification information has been provided from the imaging control device 1B as the previous device.
[0184] If it is determined in step S401 that the current imaging execution point Pi does not notify the representative angle, the control unit 5C returns to step S101. In other words, for the imaging execution point Pi which is not a slope, the following means prevents the imaging unit 2 from performing imaging and the signal processing unit 3C from performing processing based on the captured image.
[0185] On the other hand, in the case where it is determined in step S401 that the representative angle has been notified for the current imaging execution point Pi, the control section 5C proceeds to the process in step S201. Thus, for the imaging execution point Pi which is a slope, the subsequent device captures an image in the imaging direction at an angle adaptively determined relative to the representative angle, and performs processing (generating a three-dimensional point cloud in this embodiment) based on the captured image by the signal processing section 3C.
[0186] It should be noted that in the above-mentioned embodiments, as in the case of the second embodiment, the subsequent device performs imaging in the imaging direction at an angle adaptively determined relative to the ground orientation, but, alternatively, as in the case of the first embodiment, the subsequent device can perform imaging in the imaging direction at a preset angle.
[0187] <4. Modifications>
[0188] The embodiments according to the present technology have been described above. However, the present technology is not limited to the specific embodiments described above and various configurations can be taken as modifications.
[0189] For example, in the above-described embodiment, the orientation of the ground is estimated based on information about the normal vector calculated from the range image. However, the orientation of the ground may be estimated based on a polarization image obtained using the polarization sensor 25.
[0190] Fig.19 : is a block diagram showing an example of a configuration of an imaging control device 1D which estimates a change in the ground orientation based on a polarization image as described above.
[0191] In contrast, the imaging control device 1D is different from the imaging control device 1 of the first embodiment in that an imaging unit 2D is provided instead of the imaging unit 2 , a signal processing unit 3D is provided instead of the signal processing unit 3 , and a control unit 5D is provided instead of the control unit 5 .
[0192] The imaging section 2D is different from the imaging section 2 in that it is provided with a polarization sensor 25 and a polarization image generating section 26 in addition to the RGB sensor 21 , the distance measuring sensor 22 , the RGB image generating section 23 , and the distance image generating section 24 .
[0193] The signal processing unit 3D is different from the signal processing unit 3 in that a ground direction estimation unit 3bD is provided instead of the ground direction estimation unit 3b.
[0194] The control unit 5D is similar to the control unit 5 in terms of hardware configuration, but differs from the control unit 5 in that, in addition to the instructions for capturing RGB images and distance images, the control unit 5D also provides instructions for capturing polarization images to the polarization sensor 25 and the polarization image generating unit 26 as instructions for performing imaging.
[0195] As is well known, the polarization sensor 25 includes polarization pixel units arranged in two dimensions, each polarization pixel unit includes a plurality of pixels, each pixel is provided with a polarization filter to transmit light (linearly polarized light) of different polarization directions through each pixel. For example, the polarization pixel unit includes four pixels and is configured so that the pixels receive light with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, respectively.
[0196] The polarization image generation section 26 generates a polarization image based on the RAW image (image representing the received light signal value of each pixel) obtained by the polarization sensor 25. The polarization image represents an image representing information on the polarization state of incident light of each pixel.
[0197] For example, the polarization image generation unit 26 performs interpolation processing on the RAW image as synchronization processing to generate a polarization direction image representing the light signal value received in each polarization direction of each pixel. Specifically, in this example, for each pixel, a polarization direction image is generated, which represents the received light signal value for light in each polarization direction corresponding to 0 degrees, 45 degrees, 90 degrees, and 135 degrees as described above.
[0198] In addition, based on the polarization direction image, an image representing normal information of each pixel is generated as a polarization image. As is well known, when the received light signal value corresponding to each polarization direction of 0 degrees, 45 degrees, 90 degrees, 135 degrees, etc. is obtained for each pixel, a sine wave representing the polarization state can be determined by fitting based on the received light signal value of each polarization direction. In addition, such a sine wave representing the polarization state is determined for each pixel, and by performing predetermined calculations using the maximum value, minimum value, average value, etc. of the sine wave, various types of polarization information can be calculated for each pixel.
[0199] Based on the above polarization direction image, the polarization image generation section 26 determines the above waving of each pixel, and performs predetermined calculation based on the sine wave, thereby generating a polarization image representing normal line information (normal line direction) of each pixel.
[0200] In the signal processing section 3D, based on the polarization image generated by the polarization image generating section 26, the ground direction estimating section 3bD estimates the direction of the ground, and specifically in the present embodiment, calculates a representative angle.
[0201] Fig. 20 It is a block diagram showing a configuration example of the ground direction estimation unit 3bD.
[0202] In contrast, the ground direction estimation unit 3bD and Figure 7 The ground direction estimation unit 3 b shown is different in that the normal vector calculation unit 31 is omitted and an angle image generation unit 32D is provided instead of the angle image generation unit 32 .
[0203] The angle image generating section 32D generates the above-mentioned angle image based on the polarization image generated by the polarization image generating section 26. The polarization image is an image indicating normal information (i.e., information indicating the normal direction) for each pixel, and therefore the angle image generating section 32D performs processing for converting the normal direction into the angle θ for each pixel of the polarization image to generate an angle image indicating the angle θ for each pixel.
[0204] The configuration of the angle image generation unit 32D and subsequent units is similar to that of the ground direction estimation unit 3b, and the representative angle is calculated from the angle image using the binarization unit 33, the mask unit 34, and the representative angle calculation unit 35. Therefore, redundant description is omitted.
[0205] It should be noted that in the above-mentioned embodiments, the configuration as a change using the polarization sensor 25 is applied to the imaging control device 1 as the first embodiment, but the configuration as a change can be appropriately applied to the imaging control device 1A in the second embodiment and the imaging control device 1B as the previous device in the third embodiment.
[0206] Here, in the above-mentioned embodiment, a three-dimensional point cloud is generated based on a range image. However, a three-dimensional point cloud may be generated based on an RGB image. For example, a technique used for SfM (Structure from Motion) may be used to generate a three-dimensional point cloud based on an RGB image.
[0207] In addition, a three-dimensional point cloud can be generated using a radar. That is, in the present technology, it is not necessary to generate a three-dimensional point cloud based on a captured image, and it is only necessary to generate a three-dimensional point cloud based on sensing information obtained by sensing the ground side from the air.
[0208] Furthermore, in the above-described embodiment, a flying object is described as an embodiment of a mobile body M for moving the imaging control device according to the present technology. However, the mobile body M is not limited to a flying object. For example, by moving a mobile body such as a vehicle that supports a rod-shaped member (long object) on which at least the imaging unit 2 (or 2D) is mounted, it is possible to perform imaging from a high altitude while moving on the ground side, as in the case of using a flying object.
[0209] In addition, in the above-described embodiment, the orientation of the imaging section 2 (or 2D) is changed to change the imaging direction (sensing direction). However, the orientation of the entire imaging control device 1 (1A) may be changed. Alternatively, in a possible configuration, the imaging direction is changed by controlling the orientation of a moving body M supporting the imaging control device.
[0210] <5. Conclusion of Implementation Method>
[0211] As described above, the control device (imaging control device 1, 1A, 1B or 1D) as an embodiment includes: a point cloud generation unit (three-dimensional point cloud generation unit 3a) that generates a three-dimensional point cloud representing the three-dimensional structure of the target area based on sensing information obtained by sensing the ground side from the air; an estimation unit (ground orientation estimation unit 3b, 3bD) that estimates the ground orientation based on a camera image obtained by photographing the ground side from the air, and a control unit (control unit 5, 5A, or 5B) that performs control related to sensing for generating a three-dimensional point cloud of the target area based on information related to the ground orientation estimated by the estimation unit.
[0212] Therefore, when a three-dimensional point cloud of a target area is generated based on sensing information targeting the ground, appropriate sensing control corresponding to the orientation of the ground may be performed, for example, sensing may be performed at an orientation aligned with the orientation of the ground.
[0213] Since sensing can be performed in an appropriate orientation corresponding to the ground orientation, the number of samples used to sense information about the slope can be increased, making it possible to increase the number of samples of the point cloud. Therefore, the resolution of the point cloud data can be increased. This in turn makes it possible to improve the accuracy of three-dimensional measurement when the area to be measured includes a slope.
[0214] Furthermore, in the control device as the embodiment, the control unit executes control for causing sensing to be performed in a sensing direction corresponding to the ground orientation as control related to sensing based on information on the ground orientation.
[0215] Therefore, when a three-dimensional point cloud of a target area is generated based on sensing information targeting the ground, sensing may be performed at an orientation corresponding to the orientation of the ground.
[0216] Therefore, in the case where the area to be measured includes a slope, the accuracy of three-dimensional measurement can be improved.
[0217] Furthermore, in the control device according to the embodiment, the control unit executes control for changing the sensing direction to a direction corresponding to the ground orientation as control related to sensing.
[0218] For example, through control such as a change in the orientation of the imaging section or a change in the orientation of a moving body supporting the imaging section, the imaging direction becomes a direction corresponding to the orientation of the ground.
[0219] Therefore, in the case where the area to be measured includes a slope, the accuracy of three-dimensional measurement can be improved.
[0220] Furthermore, in the control device (imaging control device 1) as an embodiment, as control for changing the sensing direction, the control section (control section 5) performs control for changing the sensing direction to a direction of a preset angle (see Fig.11 ).
[0221] Thus, when sensing for generating a three-dimensional point cloud is performed in a sensing direction corresponding to the orientation of the ground, the above control eliminates the need to adaptively determine the target angle of the sensing direction (e.g., the angle in the direction directly opposite to the ground) according to the orientation of the ground.
[0222] Therefore, when attempting to improve the accuracy of three-dimensional measurement in a case where the area to be measured includes a slope, the processing load can be reduced.
[0223] Furthermore, in the control device (imaging control device 1A) as an embodiment, as control for changing the sensing direction, the control section (control section 5A) performs control for changing the sensing direction to a direction indicated by a target angle determined adaptively with respect to the ground orientation (see Fig.13 ).
[0224] Therefore, based on information obtained by sensing in a sensing direction at an angle adaptively calculated for an estimated ground orientation (eg, an angle in a direction directly opposite to the ground orientation), a three-dimensional point cloud for the slope is generated.
[0225] Therefore, the accuracy of three-dimensional measurement of the slope can be improved.
[0226] In addition, in the control device (imaging control device 1 or 1A) as an embodiment, for each of the sensing execution points for generating a three-dimensional point cloud, the control unit (control unit 5 or 5A) causes the estimation unit to estimate the orientation of the ground. Slope determination is performed based on information about the estimated orientation of the ground, and in a case where it is determined that the current sensing execution point among the sensing execution points is not a slope, wait for the next one of the sensing execution points to arrive, but in a case where it is determined that the current one of the sensing execution points is a slope, control is performed so that sensing for generating a three-dimensional point cloud is performed in a sensing direction corresponding to the orientation of the ground, and wait for the next one of the sensing execution points to arrive (see Fig.11 and Fig.13 ).
[0227] Therefore, as in the imaging control devices 1 and 1A described in the first and second embodiments, compared to a case in which a measurement form is adopted in which imaging for generating a three-dimensional point cloud is performed at each sensing execution point specified on the target area At, sensing can be performed on a slope at an appropriate orientation corresponding to the ground orientation.
[0228] Therefore, in the case where the area to be measured includes a slope, the accuracy of three-dimensional measurement can be improved.
[0229] Furthermore, in the control device (imaging control device 1B) as an embodiment, as control related to sensing, by notifying another device (imaging control device 1C) provided with a point cloud generating unit about information on the ground orientation estimated by the estimating unit, the control unit (control unit 5B) causes the other device to perform sensing in a sensing direction corresponding to the ground orientation (see Fig.17 ).
[0230] Like the imaging control device 1B as the preceding device described in the third embodiment, this can realize the following control device: notifying other devices as subsequent devices of information about the ground orientation so that the other devices perform sensing for generating a three-dimensional point cloud in a sensing direction corresponding to the ground orientation.
[0231] Therefore, for a three-dimensional survey performed in a case where the area to be surveyed includes a slope, the time required for the survey can be shortened while improving the measurement accuracy.
[0232] Furthermore, in the control device as an embodiment, the point cloud generation unit generates a three-dimensional point cloud as the distance image based on the sensing information.
[0233] The distance image means an image indicating information about the distance of each pixel to an object.
[0234] By generating a three-dimensional point cloud based on such a range image, the three-dimensional point cloud can be appropriately generated to allow three-dimensional measurement to be appropriately performed.
[0235] Furthermore, in the control device as the embodiment, the estimation unit (ground direction estimation unit 3 b ) estimates the ground direction based on the captured image as the range image.
[0236] Therefore, the generation of a three-dimensional point cloud and the estimation of the ground position can be performed based on the common range image.
[0237] Therefore, only the distance measuring sensor is required to generate a three-dimensional point cloud while estimating the ground position, making it possible to reduce the number of components for realizing three-dimensional measurement according to the present embodiment and allowing cost reduction.
[0238] Furthermore, in the control device according to the embodiment, the estimating unit estimates the orientation of the ground surface based on the three-dimensional point cloud generated by the point cloud generating unit.
[0239] This allows a common process for generating a three-dimensional point cloud for obtaining point cloud data as three-dimensional measurement data of a target area and for generating a three-dimensional point cloud for estimating the position of the ground.
[0240] This can improve the processing efficiency for achieving the generation of point cloud data and the estimation of ground position.
[0241] Furthermore, in the control device as an embodiment, the estimation section determines a normal vector for each position on the ground surface based on a positional relationship between points adjacent to each other in the three-dimensional point cloud, and estimates the orientation of the ground surface based on the normal vector.
[0242] Therefore, the orientation of the ground can be properly estimated from the three-dimensional point cloud.
[0243] In addition, in the control device as the embodiment, the estimation unit (ground direction estimation unit 3 bD) estimates the direction of the ground based on the captured image as the polarization image.
[0244] The polarization image refers to an image representing information on the polarization state of incident light for each pixel.
[0245] By estimating the orientation of the ground using such a polarization image, the orientation of the ground can be appropriately estimated.
[0246] In addition, in the control device as an embodiment, the estimation unit generates an angle image indicating the angle of the ground orientation of each pixel based on the captured image, and calculates a representative angle indicating the representative orientation of the ground based on the angle in the image area in which the angle is greater than or equal to a predetermined angle threshold in the angle image.
[0247] Thus, based on the information about the representative angle indicating the representative orientation with respect to the target ground, it can be determined whether the ground is a slope, and sensing can be appropriately performed in the sensing direction adaptively determined with respect to the orientation of the ground.
[0248] Therefore, in the case where the area to be measured includes a slope, the accuracy of three-dimensional measurement can be improved.
[0249] In addition, in the control device as an embodiment, the estimation unit calculates the representative angle based on the angle of an image area in the angle image, wherein the angle of the area is greater than or equal to an angle threshold and the area has an area greater than or equal to a predetermined area threshold.
[0250] Thus, by removing small areas caused by noise or the like from even-numbered image areas having angles greater than or equal to the angle threshold, the representative angle can be appropriately calculated.
[0251] Therefore, for imaging for generating a three-dimensional point cloud, sensing in an appropriate sensing direction corresponding to the ground position can be performed, thereby allowing the accuracy of three-dimensional measurement of the target area to be improved.
[0252] The control method of the embodiment is a control method performed by a signal processing device, and includes: processing of generating a three-dimensional point cloud indicating a three-dimensional structure of a target area based on sensing information obtained by sensing the ground side from the air; estimating the orientation of the ground based on a captured image obtained by imaging the ground side from the air, and performing control related to sensing for generating the three-dimensional point cloud of the target area based on information about the estimated orientation of the ground.
[0253] This control method can produce effects similar to those of the control device of the above-described embodiment.
[0254] Note that the effects described herein are merely illustrative and not limiting, and any other effects may be generated.
[0255] <6. This technology>
[0256] The present technology can also adopt the following configurations. (1)
[0258] A control device, comprising:
[0259] a point cloud generating unit that generates a three-dimensional point cloud representing a three-dimensional structure of a target area based on sensing information obtained by sensing the ground side from the air;
[0260] an estimating section that estimates the orientation of the ground based on a captured image obtained by imaging the ground side from the air; and
[0261] The control section performs control related to sensing for generating a three-dimensional point cloud of the target area based on the information on the orientation of the ground estimated by the estimation section. (2)
[0263] According to the control device of (1) above,
[0264] As control related to sensing based on information on the orientation of the ground, the control section performs control for causing sensing to be performed in a sensing direction corresponding to the orientation of the ground. (3)
[0266] According to the control device of (2) above,
[0267] As control related to sensing, the control unit performs control for changing the sensing direction to a direction corresponding to the orientation of the ground. (4)
[0269] According to the control device of (3) above,
[0270] The control section performs control for changing the sensing direction to a direction of a preset angle as control for changing the sensing direction. (5)
[0272] According to the control device of (3) above,
[0273] As control for changing the sensing direction, the control section performs control for changing the sensing direction to a direction indicated by a target angle adaptively determined with respect to the orientation of the ground. (6)
[0275] A control device according to any one of (1) to (5) above, wherein:
[0276] For each of the sensing execution points for generating a three-dimensional point cloud, the control unit causes the estimation unit to estimate the orientation of the ground, perform slope determination based on information about the estimated orientation of the ground, and, in a case where it is determined that the current sensing execution point among the sensing execution points is not a slope, wait for reaching the next sensing execution point among the sensing execution points, but in a case where it is determined that the current sensing execution point among the sensing execution points is a slope, perform control for causing sensing for generating a three-dimensional point cloud to be performed in a sensing direction corresponding to the orientation of the ground and wait for reaching the next sensing execution point among the sensing execution points. (7)
[0278] A control device according to any one of (1) to (5) above, wherein:
[0279] As control related to sensing, by notifying another device provided with the point cloud generating unit of information on the orientation of the ground estimated by the estimating unit, the control unit causes the other device to perform sensing in a sensing direction corresponding to the orientation of the ground. (8)
[0281] A control device according to any one of (1) to (7) above, wherein:
[0282] The point cloud generation unit generates a three-dimensional point cloud based on the sensing information as the range image. (9)
[0284] According to the control device of (8) above,
[0285] The estimation section estimates the orientation of the ground surface based on the captured image as the range image. (10)
[0287] According to the control device of (9) above,
[0288] The estimation unit estimates the orientation of the ground surface based on the three-dimensional point cloud generated by the point cloud generation unit. (11)
[0290] According to the control device of (10) above,
[0291] The estimation section determines a normal vector for each position on the ground surface based on a positional relationship between points adjacent to each other in the three-dimensional point cloud, and estimates the orientation of the ground surface based on the normal vector. (12)
[0293] A control device according to any one of (1) to (7) above, wherein:
[0294] The estimation section estimates the orientation of the ground based on the captured image as the polarization image. (13)
[0296] A control device according to any one of (1) to (12) above, wherein:
[0297] The estimation unit generates an angle image indicating the angle of the ground orientation of each pixel based on the captured image, and calculates a representative angle indicating the representative orientation of the ground based on the angle in the image area where the angle is greater than or equal to a predetermined angle threshold in the angle image. (14)
[0299] According to the control device of (13) above,
[0300] The estimation section calculates a representative angle based on an angle of an image region in the angle image, wherein the region in the angle image is greater than or equal to an angle threshold and the region has an area greater than or equal to a predetermined region threshold. (15)
[0302] A control method performed by a signal processing device comprises the following steps:
[0303] generating a three-dimensional point cloud indicating a three-dimensional structure of a target area based on sensing information obtained by sensing the ground side from the air;
[0304] estimating the orientation of the ground based on a captured image obtained by imaging the ground side from the air; and
[0305] Control related to sensing for generating a three-dimensional point cloud of the target area is performed based on information about the estimated orientation of the ground.
[0306] [Reference Numbers List]
[0307] 1. 1A, 1B, 1C, 1D imaging control device
[0308] M Mobile
[0309] At target area
[0310] Pi Imaging Execution Point
[0311] Rf flight path
[0312] 2. 2D imaging unit
[0313] 21 RGB sensor
[0314] 22 Distance sensor
[0315] 23 RGB image generation unit
[0316] 24Distance image generation unit
[0317] 3. 3D signal processing unit
[0318] 3a 3D point cloud generation department
[0319] 3b, 3bD ground position estimation unit
[0320] 3c Point Cloud Data Generation Department
[0321] 4 Storage
[0322] 5.5A, 5B, 5C control unit
[0323] 6 Position Sensor
[0324] 7 Actuator
[0325] 8IMU
[0326] 31 Normal vector calculation unit
[0327] 32. 32D angle image generation unit
[0328] 33 Binarization Processing Unit
[0329] 34 Mask Department
[0330] 35 Representative angle calculation unit
[0331] 9 Ministry of Communications
[0332] 25 Polarization Sensor
[0333] 26 Polarization image generation unit
Claims
1. A control device, comprising: a point cloud generating unit that generates a three-dimensional point cloud representing a three-dimensional structure of a target area based on sensing information obtained by sensing the ground side from the air; an estimating section that estimates the orientation of the ground based on a captured image obtained by imaging the ground side from the air; as well as A control section performs control related to sensing for generating the three-dimensional point cloud of the target area based on information on the orientation of the ground estimated by the estimation section.
2. The control device according to claim 1, wherein: The control unit performs control for causing sensing to be performed in a sensing direction corresponding to the orientation of the ground as control related to sensing based on information on the orientation of the ground.
3. The control device according to claim 2, wherein: The control section performs control for changing the sensing direction to a direction corresponding to the orientation of the ground as control related to sensing.
4. The control device according to claim 3, wherein: The control section performs, as control for changing the sensing direction, control for changing the sensing direction to a direction of a preset angle.
5. The control device according to claim 3, wherein: The control section performs, as control for changing the sensing direction, control for changing the sensing direction to a direction indicated by a target angle adaptively determined with respect to an orientation of the ground.
6. The control device according to claim 1, wherein: For each of the sensing execution points for generating the three-dimensional point cloud, the control unit causes the estimation unit to estimate the orientation of the ground, perform slope determination based on information about the estimated orientation of the ground, and, if it is determined that a current one of the sensing execution points is not a slope, wait for reaching the next one of the sensing execution points, but if it is determined that the current one of the sensing execution points is a slope, execute control for performing sensing for generating the three-dimensional point cloud in a sensing direction corresponding to the orientation of the ground and wait for reaching the next one of the sensing execution points.
7. The control device according to claim 1, wherein: As control related to sensing, by notifying another device provided with the point cloud generation unit of information about the orientation of the ground estimated by the estimation unit, the control unit causes the other device to perform sensing in a sensing direction corresponding to the orientation of the ground.
8. The control device according to claim 1, wherein: The point cloud generation unit generates the three-dimensional point cloud based on sensing information as a range image.
9. The control device according to claim 8, wherein: The estimation section estimates the orientation of the ground surface based on the captured image as a range image.
10. The control device according to claim 9, wherein: The estimating unit estimates the orientation of the ground surface based on the three-dimensional point cloud generated by the point cloud generating unit.
11. The control device according to claim 10, wherein: The estimation section determines a normal vector of each position on the ground surface based on a positional relationship between points adjacent to each other in the three-dimensional point cloud, and estimates an orientation of the ground surface based on the normal vector.
12. The control device according to claim 1, wherein: The estimation section estimates the orientation of the ground surface based on the captured image as a polarization image.
13. The control device according to claim 1, wherein: The estimation unit generates an angle image representing the angle of the ground located for each pixel based on the captured image, and calculates a representative angle representing the representative orientation of the ground based on the angles in the image area in the angle image where the angles are greater than or equal to a predetermined angle threshold.
14. The control device according to claim 13, wherein: The estimation unit calculates the representative angle based on the angle of an image region having an angle greater than or equal to an angle threshold and an area greater than or equal to a predetermined area threshold in the angle image.
15. A control method performed by a signal processing device, comprising the following processing: generating a three-dimensional point cloud representing a three-dimensional structure of a target area based on sensing information obtained by sensing the ground side from the air; estimating the orientation of the ground based on a captured image obtained by imaging the ground side from the air; as well as Control related to sensing for generating the three-dimensional point cloud of the target area is performed based on information about the estimated position of the ground.
