Control device, imaging system, and imaging method

Abstract

To provide a mobile object, a control device, and an imaging method, which can reduce image data.SOLUTION: A mobile object includes: a mobile object main body; an imaging device that is provided on the mobile object main body and captures an image of an object, the imaging device including a calibrated image data acquisition device and a three-dimensional data acquisition device; and a control device that acquires unit image data from the imaging device for each angle of view of the image data acquisition device, which associates image data with three-dimensional data from the three-dimensional data acquisition device for the object, performs plane estimation for the imaging target based on the three-dimensional data, determines whether the imaging target is a plane, and if it is determined to be a plane, determines a first distance until the next unit image data is acquired based on the plane information.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a control device, an imaging system, and an imaging method. [Background technology]

[0002] In recent years, a technology has been proposed in which a camera is mounted on a mobile object, such as a drone, to capture images of a structure and generate a three-dimensional model from the captured images.

[0003] For example, Patent Document 1 proposes a technology in which two-dimensional image data of an object is acquired by a moving object equipped with a camera, a three-dimensional point cloud is generated using SfM (Structure from Motion), and a three-dimensional model is generated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-114954 Summary of the Invention [Problem to be solved by the invention]

[0005] In SfM, imaging areas are overlapped with each other to acquire a large amount of two-dimensional image data, and a three-dimensional point cloud of the object is generated by estimating the coordinates of the object and the self-position. This requires processing a large amount of two-dimensional image data, which can increase processing time.

[0006] The present invention has been made in view of the above circumstances, and has as its object to provide a moving body, a control device, and an imaging method that can reduce image data. [Means for solving the problem]

[0007] A first aspect of the mobile body includes a mobile body main body, an imaging device provided on the mobile body for imaging an object, the imaging device including a calibrated image data acquisition device and a three-dimensional data acquisition device, and a control device that acquires, from the imaging device, unit image data for the object that associates image data acquired for each angle of view of the image data acquisition device with three-dimensional data acquired by the three-dimensional data acquisition device, performs plane estimation for the imaging object based on the three-dimensional data, determines whether the imaging object is planar, and, if it is determined to be planar, determines a first distance until the next unit image data is acquired based on the information about the plane. According to the first aspect, it is possible to reduce image data.

[0008] In the second aspect of the mobile body, when the control device determines that the surface is not flat, the control device determines a second distance, which is shorter than the first distance, as the distance to acquire the next unit of image data. According to the second aspect, by making the second distance shorter than the first distance, it is possible to effectively reduce the amount of image data.

[0009] In the moving body of the third aspect, the imaging device simultaneously acquires image data and three-dimensional data. According to the second aspect, the acquisition of image data and three-dimensional data is facilitated.

[0010] In the mobile body of the fourth aspect, the three-dimensional data acquisition device includes any one of a stereo camera, a laser scanner, and a time-of-flight camera. The fourth aspect specifies a preferred three-dimensional data acquisition device.

[0011] In the moving body of the fifth aspect, the image data is two-dimensional color image data. The fifth aspect specifies a preferred type of image data.

[0012] In the mobile body of the sixth aspect, the mobile body body provided with the imaging device and the control device is an unmanned aerial vehicle. According to the sixth aspect, it becomes easy to capture an image of the object.

[0013] A seventh aspect of the control device is a control device that is provided on a mobile body and controls an imaging device that is equipped with a calibrated image data acquisition device and a three-dimensional data acquisition device and that images an object, and acquires unit image data from the imaging device that associates image data acquired for each angle of view of the image data acquisition device with three-dimensional data acquired by the three-dimensional data acquisition device, performs plane estimation for the imaging object based on the three-dimensional data, determines whether the imaging object is flat, and if it is determined to be flat, determines a first distance until the next unit image data is acquired based on the information about the plane. According to the seventh aspect, it is possible to reduce image data.

[0014] In the control device of the eighth aspect, when it is determined that the surface is not flat, the distance to acquire the next unit image data is determined to be a second distance that is shorter than the first distance. According to the eighth aspect, by making the second distance shorter than the first distance, it is possible to effectively reduce the amount of image data.

[0015] The imaging method of the ninth aspect includes the steps of: acquiring, while moving, unit image data in which image data and three-dimensional data for an object are associated; performing plane estimation for the imaging target based on the three-dimensional data of the unit image data; determining whether the imaging target is planar; and, if it is determined to be planar, determining a first distance until the next unit image data is acquired based on the information about the plane. According to the ninth aspect, it is possible to reduce image data.

[0016] The imaging method of the tenth aspect includes a step of determining, when it is determined that the surface is not flat, a second distance, which is shorter than the first distance, as the distance to acquire the next unit of image data. According to the tenth aspect, by making the second distance shorter than the first distance, it is possible to effectively reduce the amount of image data. [Effects of the Invention]

[0017] According to the present invention, it is possible to reduce the amount of image data and avoid an increase in processing time. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram conceptually illustrating an image processing system. [Figure 2] FIG. 2 is a block diagram showing functions realized by the control device. [Figure 3] FIG. 3 is a block diagram of the controller. [Figure 4] FIG. 4 is a conceptual diagram showing the imaging of an object by the image data acquisition device and the three-dimensional data acquisition device. [Figure 5] FIG. 5 is a conceptual diagram illustrating the correspondence between image data and three-dimensional data. [Figure 6] FIG. 6 is a conceptual diagram showing how an image of an object is captured by an imaging device mounted on a moving body. [Figure 7] FIG. 7 is a flowchart illustrating the image capturing method. [Figure 8] FIG. 8 is a diagram showing how an image of an object is captured by an imaging device while the moving body is flying. [Figure 9] FIG. 9 is a diagram showing an example of image data and three-dimensional data acquired by an imaging device. [Figure 10] FIG. 10 is a conceptual diagram showing how a plane of image data is estimated from three-dimensional data. [Figure 11] FIG. 11 is a conceptual diagram showing how three-dimensional data is added to unit image data. [Figure 12] FIG. 12 is a diagram conceptually showing how a moving object moves a first distance to acquire unit image data. [Figure 13] FIG. 13 is a diagram conceptually showing how a moving object moves a second distance to acquire unit image data. [Figure 14] FIG. 14 is a block diagram of an image processing device. [Figure 15] FIG. 15 is a diagram for explaining the creation of a three-dimensional point cloud. [Figure 16] FIG. 16 is a diagram illustrating the creation of a three-dimensional point cloud. [Figure 17]FIG. 17 is a diagram showing a state in which a three-dimensional point cloud is displayed on an image processing device. [Figure 18] FIG. 18 is a diagram illustrating the creation of another three-dimensional point cloud. [Figure 19] FIG. 19 is a diagram illustrating the creation of another three-dimensional point cloud. DETAILED DESCRIPTION OF THE INVENTION

[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a moving body, a control device, and an imaging method according to the present invention will now be described with reference to the accompanying drawings.

[0020] FIG. 1 is a conceptual diagram of an image processing system including an image processing device 300 and a mobile object 100. The mobile object 100 is, for example, an unmanned aerial vehicle (UAV). The mobile object 100 has a mobile object body 102, a propulsion unit 104 provided on the mobile object body 102, and a control device 120 provided on the mobile object body 102. The mobile object body 102 is a member that forms the main shape of the mobile object 100. In an embodiment, multiple propellers and a propeller drive motor are attached to the mobile object body 102. The propellers and the propeller drive motor form the propulsion unit 104. The mobile object 100 may be a vehicle or a ship. The mobile object 100 may also be a self-propelled robot.

[0021] The moving body 100 is equipped with an imaging device 200. The imaging device 200 can be attached to the moving body main body 102 via, for example, a gimbal (not shown). As will be described later, the moving body 100 also includes an image data acquisition device 202 and a three-dimensional data acquisition device 204 (see FIG. 2). The moving body 100 flies through the atmosphere based on operation by a controller 250. The moving body 100 acquires multiple unit image data of an object using the mounted imaging device 200. The object includes, for example, structures such as bridges, dams, tunnels, and buildings. However, the object is not limited to these structures.

[0022] The image processing device 300 is configured by a computer equipped with a CPU (Central Processing Unit), ROM (Read-Only Memory), RAM (Random Access Memory), etc. The image processing device 300 includes, for example, an operation unit 310 and a display unit 320. The computer constituting the image processing device 300 functions as the image processing device 300 when the CPU executes a structure management program stored in the ROM.

[0023] 2 is a block diagram showing the configuration of the control device 120 provided in the moving body 100. The moving body 100 includes a propeller drive motor 150, a motor driver 152, a sensor unit 154, an airframe wireless communication unit 156, and the control device 120. The control device 120 is configured, for example, by a microcomputer (hereinafter also referred to as a microcomputer).

[0024] The control device 120 includes a main control unit 122, a movement control unit 124, an aircraft-side wireless communication control unit 126, and a camera control unit 128. The main control unit 122 manages the overall functions of the movement control unit 124, the aircraft-side wireless communication control unit 126, and the camera control unit 128. The control device 120 executes a program to cause the main control unit 122, the movement control unit 124, the aircraft-side wireless communication control unit 126, and the camera control unit 128 to function.

[0025] The movement control unit 124 controls the flight (movement) of the moving body 100 by controlling the drive of the propeller drive motors 150 via the motor driver 152. The movement control unit 124 controls the drive of each propeller drive motor 150 based on a control signal transmitted from the controller 250 and information on the flight state of the moving body 100 output from the sensor unit 154, thereby controlling the flight of the moving body 100. For example, when an ascending command is received from the controller 250, the movement control unit 124 controls the drive of each propeller drive motor 150 so that the airframe ascends. Furthermore, when a descending command is received from the controller 250, the movement control unit 124 controls the drive of each propeller drive motor 150 so that the airframe descends. Furthermore, when a turning command is received from the controller 250, the movement control unit 124 controls the drive of each propeller drive motor 150 so that the airframe turns in the instructed direction. Furthermore, during image capture, the movement control unit 124 controls the drive of each propeller drive motor 150 so that the airframe flies at a predetermined speed. A propeller drive motor 150 rotates a propeller (not shown) to provide propulsion to the moving body 100. The moving body 100 is equipped with multiple propeller drive motors 150 and propellers, and by varying the rotational force of each propeller, it is possible to move in various directions. The flight path of the moving body 100 can be set in advance.

[0026] The sensor unit 154 detects the flight state of the mobile object 100. The sensor unit 154 is configured with various sensors such as an IMU (inertial measurement unit) and a GNSS (Global Navigation Satellite System). The IMU is configured by combining, for example, a gyro sensor, a geomagnetic sensor, an acceleration sensor, a speed sensor, etc. on multiple axes. The sensor unit 154 outputs information about the flight state of the mobile object 100 detected by the various sensors to the control device 120.

[0027] Under the control of the control device 120, the aircraft-side wireless communication unit 156 communicates wirelessly with the controller 250, transmitting and receiving various signals to and from the controller 250. For example, when the controller 250 is operated, a control signal based on the operation is transmitted from the controller 250 to the moving object 100. The aircraft-side wireless communication unit 156 receives the control signal transmitted from the controller 250 and outputs it to the moving object 100.

[0028] Although not shown, the control device 120 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and realizes various functions by executing predetermined programs stored in the ROM.

[0029] The camera control unit 128 controls the imaging device 200 based on a control signal transmitted from the controller 250. For example, the camera control unit 128 causes the imaging device 200 to start imaging in response to an imaging start instruction from the controller 250. The camera control unit 128 causes the imaging device 200 to end imaging in response to an imaging end instruction from the controller 250.

[0030] The aircraft-side wireless communication control unit 126 controls communication with the controller 250 via the aircraft-side wireless communication unit 156 .

[0031] The flight plan for the moving body 100 and the imaging conditions for the imaging device 200 can be determined in advance using control software, etc. The flight plan includes, for example, the flight path, speed, and altitude of the moving body 100. The imaging conditions include having the imaging device 200 capture images at equal time intervals and at equal distance intervals. Conditions such as equal time intervals and equal distance intervals are selected as appropriate. The main control unit 122 controls the movement control unit 124 based on the flight plan. The movement control unit 124 controls the drive of the propeller drive motor 150 via the motor driver 152 in accordance with a signal from the main control unit 122. The main control unit 122 controls the camera control unit 128 based on the imaging conditions. The camera control unit 128 controls the imaging device 200. By combining the flight plan and the imaging conditions, the overlap rate of the imaging range on the flight path, the side overlap rate of the imaging range between flight paths, etc. are determined. As will be described later, the moving body 100 of the embodiment can determine the imaging conditions, etc. of the moving body 100 according to the shape of the object to be imaged.

[0032] FIG. 3 is a block diagram showing the electrical configuration of the controller.

[0033] The controller 250 includes a controller operation unit 250A, a controller display unit 250B, a controller side wireless communication unit 250C, and a controller microcomputer 250D.

[0034] The controller operation unit 250A is configured to include various operation members for operating the moving body 100. The operation members for operating the moving body main body 102 having a propulsion unit include, for example, an operation member for instructing the moving body main body 102 to rise or fall, an operation member for instructing the moving body main body 102 to turn, etc. The operation members for operating the imaging device 200 include, for example, an operation member for instructing the start and end of imaging, etc.

[0035] The controller display unit 250B is configured by, for example, an LCD (Liquid Crystal Display). On the controller display unit 250B, for example, information on the flight status of the moving body 100 is displayed.

[0036] The controller side wireless communication unit 250C communicates wirelessly with the moving body 100 under the control of the controller microcomputer 250D, and transmits and receives various signals to and from the moving body 100.

[0037] The controller microcomputer 250D is a control unit that controls the overall operation of the controller 250. The controller microcomputer 250D is equipped with a CPU, ROM, and RAM, and realizes various functions by executing predetermined programs. For example, when the controller operation unit 250A is operated, a control signal corresponding to the operation is generated. The control signal is transmitted to the moving object 100 via the controller-side wireless communication unit 250C. The controller 250 also acquires flight status information from the moving object 100 via the controller-side wireless communication unit 250C and displays it on the controller display unit 250B. The programs are stored in the ROM.

[0038] FIG. 4 is a conceptual diagram of an imaging device including an image data acquisition device and a three-dimensional data acquisition device capturing an image of an object. The imaging device 200 includes an image data acquisition device 202 and a three-dimensional data acquisition device 204. The object includes planar structures A and B and a non-planar structure C. The image data acquisition device 202 acquires two-dimensional image data of the object. The image data acquisition device 202 includes an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) (not shown). The imaging element has a plurality of pixels configured with photoelectric conversion elements arranged two-dimensionally in the x direction (horizontal direction) and the y direction (vertical direction). A color filter (not shown) having, for example, R (red), G (green), and B (blue) filters arranged two-dimensionally in a Bayer pattern is disposed on the upper surface of the plurality of pixels. In this embodiment, the image data acquisition device 202 can acquire two-dimensional color image data. The image data acquisition device 202 acquires image data for each angle of view for each imaging session. The imaging range is determined by the angle of view of the image data acquisition device 202. The image data acquisition device 202 acquires multiple pieces of image data for the target object. The angle of view represents the imaging range when an image is captured by the image data acquisition device 202.

[0039] The three-dimensional data acquisition device 204 acquires three-dimensional data of the object. The three-dimensional data acquisition device 204 is, for example, a stereo camera. A stereo camera is a camera that simultaneously captures image data from multiple cameras arranged at different positions and acquires three-dimensional data up to the object using parallax in the image data. When the three-dimensional data acquisition device 204 is a stereo camera, one of the multiple cameras can be used as the image data acquisition device 202. The image data acquisition device 202 can be provided separately from the three-dimensional data acquisition device 204.

[0040] In the above description, a stereo camera is used as the three-dimensional data acquisition device 204. The three-dimensional data can be acquired using a laser scanner or a time-of-flight (ToF) camera.

[0041] A laser scanner emits a laser pulse to an object and measures the distance based on the time it takes for the laser pulse to reflect off the surface of the object and return. Three-dimensional data of the reflection point of the laser pulse is then obtained based on the measured distance and the angle information of the emission direction of the laser pulse. In other words, the three-dimensional data includes three-dimensional coordinates. Laser scanners are not limited to those that use the time-of-flight method, but can also obtain three-dimensional data using phase difference and trigonometry methods.

[0042] A time-of-flight camera is a camera that acquires three-dimensional data by measuring the time of flight of light.

[0043] FIG. 5 is a conceptual diagram illustrating the correspondence between image data and three-dimensional data. The image data ID includes data of a plurality of pixels P arranged two-dimensionally. The image data ID is data within the angle of view range. The pixel P has values ​​for each of R, G, and B. FIG. 5 shows a pixel P at coordinates (Px, Py) in the image data ID and a point Q having a positional relationship with the pixel P on the object. The point Q has three-dimensional data (x, y, z) which is position information. In other words, the three-dimensional data is three-dimensional coordinates. Since the image data acquisition device 202 and the three-dimensional data acquisition device 204 are calibrated, the pixel P and the point Q are associated. A unit image data UID is acquired in which the pixel of the image data ID is associated with the three-dimensional data TD. Each data PQ of the unit image data UID has information on the three-dimensional data (x, y, z) of the point Q and the (R, G, B) values ​​of the pixel P. According to the flight plan and imaging conditions, the imaging device 200 mounted on the moving body 100 acquires a plurality of unit image data UIDs for the target object. It is preferable that the image data IDs and the three-dimensional data TDs included in the unit image data UIDs are acquired simultaneously, which makes it easy to associate the image data IDs with the three-dimensional data TDs.

[0044] The operation of the moving body 100 will be described with reference to the drawings. As shown in FIG. 6, the moving body 100 equipped with an imaging device 200 flies around an object based on a flight plan. An image data acquisition device 202 (not shown) and a three-dimensional data acquisition device 204 (not shown) provided in the imaging device 200 capture images of the object based on imaging conditions and acquire multiple unit image data UIDs. The flight plan and imaging conditions are input from, for example, the controller 250. The flight plan includes a range for generating a three-dimensional point cloud of the object. The imaging conditions also include an overlap rate and a side overlap rate for generating a three-dimensional point cloud by SfM. The imaging device 200 acquires a large amount of image data in accordance with the imaging conditions.

[0045] Next, a method for capturing an image of an object using the imaging device 200 while flying the moving body 100 will be described. Fig. 7 is a flowchart illustrating an image capturing method using the imaging device 200 of the moving body 100. The image capturing method includes a unit image data acquisition step (step S1), a plane estimation step (step S2), a plane determination step (step S3), a first distance determination step (step S4), a second distance determination step (step S5) that is shorter than the first distance, and a plan completion determination step (step S6).

[0046] In the unit image data acquisition step, unit image data in which image data ID and three-dimensional data TD are associated with an object are acquired while moving (step S1). As shown in Fig. 8, a moving body 100 equipped with an imaging device 200 flies around the object based on a flight plan. An image data acquisition device 202 (not shown) and a three-dimensional data acquisition device 204 (not shown) provided in the imaging device 200 image the object based on imaging conditions while moving within a range of an angle of view θ, and acquire unit image data UID.

[0047] FIG. 9 is a diagram showing an example of image data ID and three-dimensional data TD acquired by the imaging device 200. As shown in FIG. 9, the image data acquisition device 202 (not shown) acquires image data ID, which is two-dimensional color image data. Furthermore, the three-dimensional data acquisition device 204 (not shown) acquires depth data DP to the object. From the image data ID and the depth data DP, a unit image data UID, in which the image data ID and the three-dimensional data TD are associated, is acquired. The depth data DP is displayed in blue the closer the distance, and in red the farther the distance. In this embodiment, the upper right is displayed in a color closer to blue, and the lower left is displayed in a color closer to red. The unit image data UID is input from the imaging device 200 to the control device 120.

[0048] In the plane estimation step, a plane is estimated for the imaging target based on the three-dimensional data TD of the unit image data UID (step S2). The plane estimation is performed by estimating a plane within the angle of view range of the image data acquisition device 202. The plane estimation is performed, for example, by the main control unit 122 in the control device 120. The plane estimation performed based on the three-dimensional data within the angle of view range is performed, for example, by finding the plane equation shown in the following formula.

[0049] [Number 1] a r x+b r y+c r zd r =0 In the formula (1), x, y, and z are three-dimensional data in the three orthogonal axes of the camera coordinate system, and a r , b r , c r , and d r The coefficients a of the plane equation are the minimum squared distance between each point of the 3D data (x, y, z). r , b r , c r , and d r The estimated plane is determined by calculating

[0050] Fig. 10 is a conceptual diagram showing how a plane in image data ID is estimated from three-dimensional data TD. As shown in Fig. 10, in the three-dimensional data TD of the unit image data UID, the area surrounded by a rectangle is estimated to be a plane PL. The plane PL in the image data ID is estimated.

[0051] In the plane determination step, it is determined whether the imaging target is a plane (step S3). It is determined whether most of the area of ​​the imaging target, i.e., within the angle of view range, is a plane PL. The determination of whether it is a plane or not is performed, for example, by the main control unit 122 in the control device 120. For example, the plane estimated from the three-dimensional data is compared with the size of the angle of view. If it is determined to be a plane in the plane determination step (determined as "Y"), the process proceeds to a step of determining the first distance.

[0052] In the step of determining the first distance, if it is determined that the plane is a plane, the first distance until the next unit image data is acquired is determined based on the plane information (step S4). As shown in FIG. 11, for example, the coordinates of the three-dimensional data TD of the unit image data UID are added. These coordinates are shown at the four corners of the estimated plane PL. Here, (-2.0, 1, 3.0), (2.0, 1, 3.5), (2.0, -1, 3.5), and (-2.0, -1, 3.0) are shown. From these four coordinates, the size (width W and height H) of the plane PL is estimated (see FIG. 10).

[0053] The first distance L1 until the next unit image data is acquired can be calculated using Equation 2 when moving horizontally and Equation 3 when moving vertically. As shown in Equation 2 and Equation 3, the smaller the overlap rate, the larger the first distance L1. The overlap rate can be set in advance. Different overlap rates R1 can be set for horizontal and vertical movements.

[0054] [Number 2] L1 = estimated plane width W × (100% - overlap rate R1) [Number 3] L1 = Estimated plane height H × (100% - overlap rate R1) The step of determining the first distance is performed, for example, by the main control unit 122 in the control device 120. The first distance L1 is input, for example, from the main control unit 122 to the movement control unit 124 and the camera control unit 128. The moving body 102 and the imaging device 200 are prepared for acquiring the next unit image data.

[0055] If the plane is determined to be a plane (determined as "N") in the plane determination step (step S3), the process proceeds to a second distance determination step (step S5) in which a second distance shorter than the first distance is determined. The second distance L2 can be determined in advance as a default value by setting an overlap rate R2 (overlap rate and side lap rate) on the premise that a three-dimensional point cloud will be created using SfM. In the second distance determination step, the second distance L2 is determined as a default value. Different overlap rates R2 can be set for horizontal and vertical movement.

[0056] [Number 4] L2 = width of imaging range W × (100% - overlap rate R2) [Number 5] L2 = Height of imaging range H × (100% - overlap rate R2) The overlap rate R1 when calculating the first distance L1 is set to be smaller than the overlap rate R2 when calculating the second distance L2. As a result, the first distance L1 is longer than the second distance L2. After the first distance determination step (step S4) or the second distance determination step (step S5), the process proceeds to the planning completion determination step.

[0057] Next, in a plan completion determination step, it is determined whether or not the plan (flight plan and imaging conditions) previously set for the target object has been completed (step S6).

[0058] If it is determined in the plan completion determination step that the plan has not been completed (determined as "N"), the process proceeds to the unit image data acquisition step (step S1).

[0059] After the first distance determination step (step S4) is executed, in the unit image data acquisition step (step S1), the moving body 100 moves in parallel from the estimated plane by a first distance L1 while maintaining the distance to the estimated plane, as shown in Fig. 12. The imaging device 200 mounted on the moving body 100 acquires the next unit image data of the target object.

[0060] When the second distance determination step (step S5) is executed, in the unit image data acquisition step (step S1), the moving body 100 moves in parallel from the estimated plane by a second distance L2 while maintaining the distance to the estimated plane, as shown in Fig. 13. The imaging device 200 mounted on the moving body 100 acquires the next unit image data for the target object.

[0061] 12 and 13, the first distance L1 is longer than the second distance L2. That is, once a plane is estimated, the distance until the next unit image data is acquired becomes longer, so the number of image data IDs to be acquired for the target object can be reduced.

[0062] The unit image data acquisition step (step S1), plane estimation step (step S2), plane determination step (step S3), first distance determination step (step S4) or second distance determination step (step S5), and plan termination determination step (step S6) are repeatedly executed until the plan termination determination step (step S6) determines that the plan has been terminated (determined as "Y"), and when the plan termination determination step (step S6) determines that the plan has been terminated (determined as "Y"), the moving body 100 stops capturing images using the imaging device 200 and, for example, returns to a predetermined position.

[0063] (Creating 3D point clouds) Next, a first procedure for creating a three-dimensional point cloud using the reduced image data ID will be described. Unit image data UID (image data ID and three-dimensional data TD) of an object acquired by the imaging device 200 of the moving body 100 is input to the image processing device 300. As shown in Fig. 14, the image processing device 300 is composed of, for example, an operation unit 310, a display unit 320, a device input / output unit 330, a device control unit 340, and a recording unit 350.

[0064] The device input / output unit 330 is connected wirelessly or via a wire to input and output information. For example, a plurality of unit image data UIDs acquired by the moving body 100 are input via the device input / output unit 330.

[0065] The device control unit 340 acquires the unit image data UID via the device input / output unit 330 and creates a three-dimensional point cloud. The device control unit 340 also controls recording in the recording unit 350, controls display on the display unit 320, and performs control in response to commands input from the operation unit 310.

[0066] The display unit 320 performs display under the control of the device control unit 340. For example, the display unit 320 displays a three-dimensional point cloud onto which damage is mapped.

[0067] The recording unit 350 records various types of information under the control of the device control unit 340. For example, the recording unit 350 records the created three-dimensional point cloud. The recording unit 350 records various programs that control the device control unit 340.

[0068] In the following embodiment, it is assumed that a plane PL has been estimated for a part of a structure A, as shown in Fig. 15. Image data ID capturing an image of the plane PL has been acquired. The multiple image data IDs are acquired by the imaging device 200 after moving the moving body 100 a first distance L1, as shown in Fig. 12. Furthermore, the multiple image data IDs for parts other than the plane of the structure A are acquired by the imaging device 200 after moving the moving body 100 a second distance L2, as shown in Fig. 13. For parts other than the plane of the structure A, an image group IG containing the multiple image data IDs required for SfM is acquired.

[0069] Next, as shown in FIG. 16, by combining the image data ID of the plane PL, composite image data CID corresponding to the plane of the structure A can be created. The composite image data CID can be created using block matching, a type of pattern matching. In block matching, a block of a predetermined size is set in one image data ID, and that block is scanned in another image data ID to calculate a correlation value. Then, the point with the highest correlation value is considered to be the location where that block overlaps, and adjacent image data ID are connected and combined. Because the plane PL has been estimated, the image data ID can be connected and combined with high accuracy to create composite image data CID.

[0070] Finally, the imaging position, orientation, and coordinates of the object are estimated from the image group IG by SfM. Further, densification is performed by MVS (Multi-view Stereo) processing to create a three-dimensional point cloud. The image group IG does not include a portion corresponding to the estimated plane PL. Therefore, by using point cloud information obtained by SfM and adjacent to the plane PL, the composite image data CID can be arranged to create a three-dimensional point cloud for the object, as shown in FIG. 17. In this embodiment, the image group IG and the composite image data CID are subjected to SfM processing to create a three-dimensional point cloud.

[0071] In SfM, feature points included in the area where the image data IDs of the image group IG overlap are extracted, and the correspondence between feature points between the image data IDs is identified.

[0072] SIFT (Scale-invariant feature transform) features, SURF (Speed-Upped Robust Feature) features, and AKAZE (Accelerated KAZE) features are known as local features that are robust against changes in scaling (differences in imaging distance) and rotation between image data IDs. The number of corresponding points (number of sets) with matching features is preferably several tens or more, and therefore the overlap rate and / or side overlap rate between image data IDs in the image group IG is preferably large.

[0073] Next, a second procedure for creating a three-dimensional point cloud using the reduced image data IDs will be described.

[0074] 18 shows the two unit image data UID-1 and unit image data UID-2 in each camera coordinate system input to the image processing device 300. The camera coordinate system is the coordinate system of the imaging device 200, and the center of the lens has the origin.

[0075] Since the image capture positions of unit image data UID-1 and unit image data UID-2 are different, the origins of the camera coordinate systems are different. In unit image data UID-1 and unit image data UID-2, each coordinate point (x, y, z) has a value of (R, G, B). The device control unit 340 extracts feature points from each of unit image data UID-1 and unit image data UID-2, as indicated by the arrows. The relationship between the feature points of unit image data UID-1 and unit image data UID-2 is determined.

[0076] When a relationship is recognized between the image data UID-1, image data UID-2, and feature points, the image data UID-2 can be projected onto the space of the camera coordinate system of the image data UID-1, as shown in FIG.

[0077] By performing the above-described processing on the estimated unit image data UID of the plane PL, a group of points can be collected as a spatial model of one camera coordinate system.

[0078] Finally, by using the point cloud information obtained by SfM, which is adjacent to the plane PL, and the point cloud collected by the above process, a three-dimensional point cloud for the object shown in Figure 17 can be created, as in the first step.

[0079] In image group data, the number of unit image data UIDs has been reduced, so the burden of processing using image data IDs is small. In image group data, the plane PL has been estimated, so the relationship between feature points between unit image data UIDs can be easily determined.

[0080] The first and second steps are performed by acquiring multiple image data IDs of the object using the image capture device 200 of the moving object 100, extracting multiple feature points from the multiple image data IDs, matching the multiple feature points, and calculating the position and orientation of the image capture device 200 and a three-dimensional point cloud of the feature points. In the embodiment, the amount of image data acquired is reduced by estimating the plane of the object, thereby reducing processing time.

[0081] The hardware for realizing the image processing device according to the present invention can be configured with various processors. These processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs and functions as various processing units; a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture; and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration specifically designed to execute specific processes. A processing unit constituting an image display device may be configured with one of the various processors described above, or may be configured with two or more processors of the same or different types. For example, a processing unit may be configured with multiple FPGAs or a combination of a CPU and an FPGA. Alternatively, multiple processing units may be configured with a single processor. A first example of configuring multiple processing units with a single processor is a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units. Second, there is a form that uses a processor that realizes the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, various processing units are configured as a hardware structure using one or more of the above-mentioned various processors. Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0082] Although the present invention has been described above, the present invention is not limited to the above examples, and various improvements and modifications may be made without departing from the gist of the present invention. [Explanation of symbols]

[0083] 100: Mobile 102: Mobile body 104: Promotion Department 120: Control device 122: Main control unit 124: Movement control unit 126: Aircraft wireless communication control unit 128: Camera control unit 150: Propeller drive motor 152: Motor driver 154: Sensor unit 156: Aircraft side radio communication unit 200: Imaging device 202: Image data acquisition device 204: Three-dimensional data acquisition device 250: Controller 250A: Controller operation section 250B: Controller display 250C: Controller side wireless communication unit 250D: Controller microcomputer 300: Image processing device 310:Operation unit 320:Display section 330: Device input / output unit 340: Device control unit 350: Recording department

Claims

1. a means for estimating a size of a plane of the object included in a single imaging range of an image data acquisition device that acquires two-dimensional image data of the object based on three-dimensional data of the object; a means for determining a first distance when the image data acquisition device is moved parallel to the plane and next two-dimensional image data is captured based on the result of estimating the size of the plane; a means for determining, when it is determined that the object included in the imaging range of one imaging operation is not flat, a second distance, which is shorter than the first distance, as a distance to acquire the next two-dimensional image data; Equipped with an overlap rate when the next two-dimensional image data is acquired using the first distance is set to be smaller than an overlap rate when the next two-dimensional image data is acquired using the second distance; Control device.

2. The three-dimensional data is obtained by associating each pixel in the two-dimensional image data of the object with each point having position information in the three-dimensional data. The control device according to claim 1 .

3. The control device according to claim 1 or 2, wherein the imaging range is determined based on an angle of view of the image data acquisition device.

4. a three-dimensional data acquisition device for acquiring three-dimensional data of the object; a moving body; and the image data acquisition device provided on the moving body; The control device according to any one of claims 1 to 3, moving the moving body based on the first distance and capturing two-dimensional image data of the object; Imaging system.

5. a step of estimating a size of a plane of the object included in a single imaging range of an image data acquisition device that acquires two-dimensional image data of the object based on the three-dimensional data of the object; determining a first distance for capturing next two-dimensional image data by moving the image data acquisition device parallel to the plane based on the result of estimating the size of the plane; determining a second distance, which is shorter than the first distance, as a distance to acquire the next two-dimensional image data when it is determined that the object included in the imaging range of the single imaging is not flat; Including, an overlap rate when the next two-dimensional image data is acquired using the first distance is set to be smaller than an overlap rate when the next two-dimensional image data is acquired using the second distance; Imaging method.

Images