Control device, control method, and program

Abstract

This control device comprises a first processor. The first processor: acquires, at a timing prior to a first mobile object on which an imaging device is mounted arriving at a target location, the brightness of a target imaging region to be captured by the imaging device from the target location; acquires a shutter speed corresponding to the brightness and a travel speed corresponding to the shutter speed; and at the target location, causes the imaging device to capture the target imaging region at the above shutter speed while causing the first mobile object to travel at the above travel speed.

Description

Control device, control method, and program

[0001] The technology of the present disclosure relates to a control device, a control method, and a program.

[0002] Japanese Patent Application Laid-Open Publication No. 2021-96865 discloses an information processing device that instructs control of flight by an aircraft. The information processing device includes a processing unit. The processing unit acquires the illuminance around the aircraft detected by an illuminance sensor equipped in the aircraft, and acquires relationship information indicating the relationship between the illuminance and the upper limit of the flight speed of the aircraft. The processing unit derives the upper limit of the flight speed of the aircraft corresponding to the acquired illuminance based on the relationship information, and instructs control of the flight speed so that the flight speed of the aircraft is equal to or less than the upper limit of the flight speed of the aircraft.

[0003] Japanese Patent Application Laid-Open Publication No. 2020-50261 discloses an information processing device that instructs control of flight by an air vehicle. The information processing device includes a processing unit. The processing unit acquires information about a shutter speed for capturing images by the air vehicle and acquires information about the distance in real space per pixel in the captured image captured by the air vehicle. The processing unit determines an upper limit for the flight speed of the air vehicle based on the shutter speed and the distance in real space per pixel, and instructs control of the flight speed so that the flight speed is equal to or less than the upper limit for the flight speed of the air vehicle.

[0004] International Publication No. 2021 / 014752 discloses an information processing device including a map creation unit, a shape extraction unit, a composition setting unit, and a route determination unit. The map creation unit creates a map of a movement range, which is a range in which a moving object equipped with an imaging device captures images while moving. The shape extraction unit extracts shapes present in the map. The composition setting unit sets the composition of an image to be captured by the imaging device. The route determination unit determines a movement route within the movement range of the moving object based on the shape and composition.

[0005] JP 2018-535487 A discloses a method for controlling a movable object, which includes the steps of estimating a reference point of the movable object and one or more movement characteristics of the movable object at the reference point based on a target direction, and generating a movement path of the movable object from the position of the movable object to the reference point based on the position of the movable object and the one or more movement characteristics of the movable object at the reference point.

[0006] JP 2019-507924 A discloses a system for modifying the autonomous flight of an unmanned aerial vehicle. The system includes a first user interface and a second user interface. The first user interface is configured to receive a first user input, the first user input providing one or more commands for performing the autonomous flight of the UAV. The second user interface is configured to receive a second user input, the second user input providing one or more commands for modifying the autonomous flight of the UAV.

[0007] International Publication No. WO 2018 / 167893 discloses a shape generation method. The shape generation method includes a step of acquiring information about multiple imaging positions of an aircraft having multiple imaging devices. The shape generation method includes a step of selecting an imaging device to be used for imaging at each of the multiple imaging positions from among the multiple imaging devices. The shape generation method includes a step of capturing images at each imaging position using the selected imaging device. The shape generation method includes a step of restoring the shape of the subject based on images captured by each imaging device. The shape generation method includes a step of synthesizing the restored shapes for each imaging device. The step of selecting an imaging device includes a step of selecting at least one imaging device from the multiple imaging devices based on a proportion of an area within an imaging area at the imaging position that has less than a predetermined amount of light.

[0008] Japanese Patent Application Laid-Open Publication No. 2019-87073 discloses a mobile manipulator including a mobile device for moving from a first position to a second position, a manipulator connected to the mobile device, a controller for controlling the mobile device and the manipulator, and an environmental acquisition sensor for acquiring environmental data associated with the position of the mobile device. The controller is configured to control at least one of the mobile device or the manipulator based on the environmental data acquired by the environmental acquisition sensor and associated with the destination position or any point on the movement path between the first position and the second position.

[0009] As an example, one embodiment of the technology disclosed herein provides a control device, a control method, and a program that can move a first moving body at a target position at a movement speed corresponding to the shutter speed, while causing an imaging device to capture an image of an image target area at a shutter speed corresponding to the brightness of the image target area.

[0010] A first aspect of the technology of the present disclosure is a control device that includes a first processor, and the first processor acquires the brightness of an imaging target area to be imaged by the imaging device from the target position at a timing before a first moving body equipped with an imaging device reaches the target position, acquires a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed, and causes the imaging device to image the imaging target area at the shutter speed while moving the first moving body at the movement speed at the target position.

[0011] A second aspect of the technology of the present disclosure is a control device according to the first aspect, wherein the first processor acquires a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed based on relationship information representing the relationship between brightness, shutter speed, and movement speed.

[0012] A third aspect of the technology of the present disclosure is the control device according to the second aspect, further comprising a memory that stores related information.

[0013] A fourth aspect of the technology of the present disclosure is a control device according to any one of the first to third aspects, wherein the brightness is detected using a sensor from a target position.

[0014] A fifth aspect of the technique of the present disclosure is the control device according to the fourth aspect, wherein the sensor is a control device mounted on the second moving body.

[0015] A sixth aspect of the technology of the present disclosure is a control device according to the fifth aspect, wherein the target position is set on a movement route, and the brightness is the brightness detected while the second moving body moves along the movement route at a speed higher than the movement speed.

[0016] A seventh aspect of the technology of the present disclosure is a control device according to any one of the first to third aspects, wherein the brightness is detected using an overhead camera that overlooks a subject including an imaging target area.

[0017] According to an eighth aspect of the technique of the present disclosure, in the control device according to the seventh aspect, the subject is a control device including a plurality of imaging target areas.

[0018] A ninth aspect of the technology of the present disclosure is a control device according to any one of the first to eighth aspects, wherein the target position is set on a movement route and the brightness is the brightness detected at a timing before the first moving body starts moving along the movement route.

[0019] A tenth aspect of the technology of the present disclosure is a control device according to any one of the first to eighth aspects, wherein the target position is set on a movement route, and the brightness is the brightness detected at a timing after the first moving body starts moving along the movement route and before the first moving body reaches the target position.

[0020] An eleventh aspect of the technology of the present disclosure is a control device relating to any one of the first to tenth aspects, in which a plurality of target positions are set as destinations to which a first moving body will move, and a first processor is a control device that causes an imaging device to capture an image of an imaging target area at a constant aperture value at the plurality of target positions.

[0021] A twelfth aspect of the technology of the present disclosure is a control device relating to any one of the first to tenth aspects, in which a plurality of target positions are set as destinations to which a first moving body will move, and the first processor is a control device that causes an imaging device to capture an image of an imaging target area at each target position with an aperture value corresponding to brightness and / or shutter speed.

[0022] A thirteenth aspect of the technology of the present disclosure is a control device relating to any one of the first to twelfth aspects, in which a plurality of target positions are set as destinations to which a first moving body will move, and a first processor acquires a synthesis image obtained by having an imaging device capture an image of an imaging target area for each target position, and the synthesis image is an image in which adjacent synthesis images partially overlap.

[0023] A fourteenth aspect of the technology of the present disclosure is a control device according to the thirteenth aspect, further comprising a second processor that generates a composite image by combining adjacent images to be combined, and the second processor is a control device that performs specific processing on the composite image based on brightness.

[0024] A fifteenth aspect of the technique of the present disclosure is the control device according to the fourteenth aspect, wherein the specific processing includes a correction processing for correcting the luminance of the composite image.

[0025] A sixteenth aspect of the technique of the present disclosure is the control device according to the fifteenth aspect, wherein the correction process includes a process of correcting a difference in luminance between the images to be combined.

[0026] A seventeenth aspect of the technology of the present disclosure is a control device according to the fifteenth or sixteenth aspect, wherein the correction process includes a process of correcting the luminance distribution of the composite image based on the brightness distribution.

[0027] An 18th aspect of the technology of the present disclosure is a control device relating to any one of the 14th to 17th aspects, wherein the specific processing includes an alarm processing that issues an alarm when the difference in brightness between the imaging target areas exceeds a predetermined value.

[0028] A 19th aspect of the technology of the present disclosure is a control method that includes acquiring the brightness of an imaging target area to be imaged by the imaging device from a target position at a timing before a first moving body equipped with an imaging device reaches the target position, acquiring a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed, and causing the imaging device to image the imaging target area at the shutter speed while moving the first moving body at the movement speed at the target position.

[0029] A twentieth aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring the brightness of an imaging target area to be imaged by the imaging device from a target position at a timing before a first moving body equipped with an imaging device reaches the target position, acquiring a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed, and causing the imaging device to image the imaging target area at the shutter speed while moving the first moving body at the movement speed at the target position.

[0030] 1 is a front view showing an example of a plurality of imaging target areas and an imaging system according to the first embodiment. FIG. 1 is a block diagram showing an example of the hardware configuration of a flying imaging device according to the first embodiment. FIG. 1 is a block diagram showing the hardware configuration of an imaging device according to the first embodiment. FIG. 2 is a block diagram showing an example of a functional configuration for realizing flying imaging processing according to the first embodiment. FIG. 3 is an explanatory diagram illustrating an example of the operation of a data request unit and a brightness acquisition unit according to the first embodiment. FIG. 4 is an explanatory diagram illustrating an example of the operation of a brightness acquisition unit and an imaging condition acquisition unit according to the first embodiment. FIG. 5 is an explanatory diagram illustrating an example of the operation of an imaging condition acquisition unit and a flight speed control unit according to the first embodiment. FIG. 6 is an explanatory diagram illustrating an example of the operation of an imaging condition acquisition unit, a reach determination unit, and an imaging control unit according to the first embodiment. FIG. 7 is a flowchart showing an example of the flow of flying imaging processing according to the first embodiment. FIG. 8 is an explanatory diagram illustrating an example of the operation of a data request unit and a brightness acquisition unit according to the second embodiment. FIG. 9 is an explanatory diagram illustrating an example of the operation of a data request unit and a brightness acquisition unit according to the third embodiment. FIG. 10 is an explanatory diagram illustrating an example of the operation of a data request unit and a brightness acquisition unit according to the fourth embodiment. FIG. 11 is a block diagram showing an example of a functional configuration for realizing composite image generation processing according to the fifth embodiment. FIG. 12 is an explanatory diagram illustrating an example of the operation of an image synthesis unit, a brightness information acquisition unit, and a correction processing unit according to the fifth embodiment.

[0031] Hereinafter, an example of an embodiment of a control device, a control method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0032] First, the terms used in the following description will be explained.

[0033] I / F is an abbreviation for "Interface". RAM is an abbreviation for "Random Access Memory". CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". HDD is an abbreviation for "Hard Disk Drive". SSD is an abbreviation for "Solid State Drive". DRAM is an abbreviation for "Dynamic Random Access Memory". SRAM is an abbreviation for "Static Random Access Memory". GNSS is an abbreviation for "Global Navigation Satellite System". GPS is an abbreviation for "Global Positioning System". LiDAR is an abbreviation for "light detection and ranging". NVM is an abbreviation for "Non-Volatile Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". FPGA is an abbreviation for "Field-Programmable Gate Array". PLD is an abbreviation for "Programmable Logic Device". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". RGB is an abbreviation for "Red Green Blue". CIE is an abbreviation for "Commission Internationale de l'Eclairage". TPU is an abbreviation for "Tensor Processing Unit". USB is an abbreviation for "Universal Serial Bus". SoC is an abbreviation for "System-on-a-Chip". IC is an abbreviation for "Integrated Circuit".

[0034] In the description of this specification, the term "vertical direction" refers to a perfectly vertical direction as well as a vertical direction generally accepted in the technical field to which the technology of the present disclosure belongs, with a degree of tolerance that does not deviate from the spirit of the technology of the present disclosure. In the description of this specification, the term "horizontal direction" refers to a perfectly horizontal direction as well as a horizontal direction generally accepted in the technical field to which the technology of the present disclosure belongs, with a degree of tolerance that does not deviate from the spirit of the technology of the present disclosure. In the description of this specification, the term "rectangle" refers to a perfectly horizontal direction as well as a rectangle generally accepted in the technical field to which the technology of the present disclosure belongs, with a degree of tolerance that does not deviate from the spirit of the technology of the present disclosure. In the description of this specification, the term "vertical" refers to a perfectly vertical direction as well as a vertical direction generally accepted in the technical field to which the technology of the present disclosure belongs, with a degree of tolerance that does not deviate from the spirit of the technology of the present disclosure. In the description of this specification, "constant" refers not only to absolute constancy, but also to constancy that is generally accepted in the technical field to which the technology of the present disclosure belongs, and that includes an error that does not contradict the intent of the technology of the present disclosure.

[0035] First Embodiment First, the first embodiment will be described.

[0036] 1, the imaging system S includes a flying imaging device 10 and an overhead camera 160. The flying imaging device 10 has a flying function and an imaging function, and captures an image of a wall surface 2A of an object 2 while flying.

[0037] As an example, the wall surface 2A is a plane. A plane refers to a two-dimensional surface (i.e., a surface extending in two dimensions). In addition, in the description of this specification, the concept of "plane" does not include the meaning of a mirror surface. In this embodiment, for example, the wall surface 2A is a plane defined by the horizontal and vertical directions (i.e., a surface extending in the horizontal and vertical directions). The wall surface 2A includes irregularities. The irregularities referred to here include, for example, irregularities caused by the material forming the wall surface 2A, as well as irregularities caused by defects and / or flaws. As an example, the object 2 having the wall surface 2A is a pier provided on a bridge. The pier is made of, for example, reinforced concrete. Here, a pier is given as an example of the object 2, but the object 2 may be an object other than a pier (e.g., a tunnel or a dam).

[0038] The flight imaging device 10 includes an air vehicle 20 and an imaging device 60. The air vehicle 20 is, for example, an unmanned aerial vehicle such as a drone. The flight function of the flight imaging device 10 is realized by the air vehicle 20. The air vehicle 20 has multiple propellers 42, and flies when the multiple propellers 42 rotate. The flight of the air vehicle 20 is synonymous with the flight of the flight imaging device 10. The air vehicle 20 is an example of a "moving body" according to the technology of the present disclosure.

[0039] The imaging device 60 is, for example, a digital camera or a video camera. The imaging function of the flying imaging device 10 is realized by the imaging device 60. The imaging device 60 is mounted on the flying object 20. As an example, the imaging device 60 is provided on the bottom of the flying object 20. The imaging device 60 is an example of an "imaging device" according to the technology of the present disclosure.

[0040] A flight route 4 is set for the target object 2. In the example shown in FIG. 1 , the flight route 4 is set on a virtual surface opposite the wall surface 2A. As an example, the flight route 4 is set in a zigzag pattern, with a horizontal route extending horizontally and a vertical route extending vertically connected. As will be described later, the flight imaging device 10 stores flight route information 116 (see FIG. 8 ) indicating the flight route 4, and is capable of autonomously flying along the flight route 4 without relying on a flight instruction signal from a transmitter (not shown) or a base station (not shown), etc.

[0041] The flying imaging device 10 flies along the flight route 4, moving in a zigzag pattern by alternately repeating horizontal and vertical movement. A plurality of waypoints 5 are set on the flight route 4. At each waypoint 5, the flying imaging device 10 captures an image of the wall surface 2A. The flight route 4 is an example of a "travel route" according to the technology of the present disclosure, and the waypoint 5 is an example of a "target position" according to the technology of the present disclosure.

[0042] Note that although an example is given here in which the flight imaging device 10 flies independently along the flight route 4, the flight imaging device 10 may also fly along the flight route 4 based on a flight instruction signal from a transmitter or a base station, etc.

[0043] The flight imaging device 10 captures images of the wall surface 2A at each waypoint 5, thereby sequentially capturing images of multiple imaging target areas 3 on the wall surface 2A. Each imaging target area 3 corresponds to a respective waypoint 5. The imaging target area 3 is an area determined by the angle of view of the flight imaging device 10. In the example shown in FIG. 1 , a rectangular area is shown as an example of the imaging target area 3. The flight imaging device 10 moves in a zigzag pattern by alternately moving horizontally and vertically, thereby sequentially capturing images of multiple imaging target areas 3 connected in a zigzag pattern.

[0044] A plurality of compositing images 132 are obtained by sequentially capturing images of a plurality of imaging target areas 3 using the imaging device 60. A composite image 130 is generated by combining the plurality of compositing images 132. The plurality of compositing images 132 are combined such that adjacent compositing images 132 partially overlap each other. An example of the composite image 130 is a two-dimensional panoramic image. The two-dimensional panoramic image is merely an example, and a three-dimensional image (e.g., a three-dimensional panoramic image) may be generated as the composite image 130 in the same manner as a two-dimensional panoramic image is generated as the composite image 130.

[0045] The composite image 130 may be generated each time a second or subsequent frame of a compositing image 132 is obtained, or may be generated after a plurality of compositing images 132 are obtained for the wall surface 2A. The process of generating the composite image 130 may be executed by the flight imaging device 10, or may be executed by an external device (not shown) communicatively connected to the flight imaging device 10. The composite image 130 is used, for example, to inspect or measure the wall surface 2A of the object 2.

[0046] 1 shows an example in which each imaging target area 3 is imaged by the imaging device 60 with the optical axis OA (see FIG. 2) of the imaging device 60 perpendicular to the wall surface 2A. The following description is based on the assumption that each imaging target area 3 is imaged by the imaging device 60 with the optical axis OA of the imaging device 60 perpendicular to the wall surface 2A.

[0047] The multiple imaging target areas 3 are imaged so that adjacent imaging target areas 3 partially overlap. The multiple imaging target areas 3 are imaged so that adjacent imaging target areas 3 partially overlap in order to synthesize compositing images 132 corresponding to the adjacent imaging target areas 3 based on feature points included in the overlapping parts of the adjacent imaging target areas 3. Hereinafter, the partial overlap of adjacent imaging target areas 3 and the partial overlap of adjacent compositing images 132 may each be referred to as "overlap."

[0048] The overhead camera 160 is a digital camera or video camera with a wider angle than the imaging device 60. The overhead camera 160 is positioned, for example, opposite the wall surface 2A. The overhead camera 160 is also installed at a position farther away from the wall surface 2A than the flying imaging device 10. The overhead camera 160 has an angle of view that allows it to capture the entire wall surface 2A. The overhead camera 160 is equipped with an imaging lens, an image sensor, a processor, storage, RAM, etc., all of which are not shown in the figures. The overhead camera 160 is an example of a "bird's-eye view camera" according to the technology of the present disclosure.

[0049] 2, the flying object 20 includes a flight device 22, an input / output I / F 24, a computer 26, a positioning unit 28, an acceleration sensor 30, and a communication device 32. The computer 26 is an example of a "control device" and a "computer" according to the techniques of the present disclosure.

[0050] The computer 26 includes a processor 34, a storage 36, and a RAM 38. The processor 34, the storage 36, and the RAM 38 are connected to one another via a bus 40, and the bus 40 is connected to the input / output I / F 24. The positioning unit 28, the acceleration sensor 30, and the communication device 32 are also connected to the input / output I / F 24.

[0051] The processor 34 has, for example, a CPU and controls the entire flight imaging device 10. Here, an example in which the processor 34 has a CPU is given, but this is merely one example. For example, the processor 34 may have a CPU and a GPU. In this case, for example, the GPU operates under the control of the CPU and is responsible for executing image processing. The processor 34 is an example of a "first processor" according to the technology of the present disclosure.

[0052] The storage 36 is a non-volatile storage device that stores various programs, various parameters, etc. Examples of the storage 36 include an HDD and an SSD. Note that the HDD and SSD are merely examples, and a flash memory, a magnetoresistive memory, and / or a ferroelectric memory may be used instead of the HDD and / or SSD, or together with the HDD and / or SSD.

[0053] The RAM 38 is a memory that temporarily stores information and is used as a work memory by the processor 34. The RAM 38 may be, for example, a DRAM and / or an SRAM.

[0054] The communication device 32 is connected to the overhead camera 160 so as to be able to communicate with it via wired or wireless communication. The communication device 32 is responsible for sending and receiving information to and from the overhead camera 160. For example, the communication device 32 transmits information to the overhead camera 160 in response to a request from the processor 34. The communication device 32 also receives data transmitted from the overhead camera 160 and outputs the received data to the processor 34 via the bus 40.

[0055] The flight device 22 has multiple propellers 42, multiple motors 44, and a motor driver 46. The motor driver 46 is connected to the processor 34 via the input / output I / F 24 and the bus 40. The motor driver 46 individually controls the multiple motors 44 in accordance with instructions from the processor 34. The number of the multiple motors 44 is the same as the number of the multiple propellers 42.

[0056] A propeller 42 is fixed to the rotation shaft of each motor 44. Each motor 44 rotates the propeller 42. The rotation of the multiple propellers 42 causes the flying object 20 to fly. Note that, while the number of multiple propellers 42 (in other words, the number of multiple motors 44) provided to the flying object 20 is four, this is merely an example, and the number of multiple propellers 42 may be, for example, three, or five or more.

[0057] The positioning unit 28 is a device that detects the position of the air vehicle 20. The position of the air vehicle 20 is detected, for example, using GNSS (e.g., GPS). The positioning unit 28 has a GNSS receiver (not shown). The GNSS receiver receives, for example, radio waves transmitted from multiple satellites. The positioning unit 28 detects the position of the air vehicle 20 based on the radio waves received by the GNSS receiver and outputs positioning data 48 (e.g., data indicating latitude, longitude, and altitude) corresponding to the detected position.

[0058] The acceleration sensor 30 detects acceleration in each of the axial directions of the pitch axis, yaw axis, and roll axis of the air vehicle 20. The acceleration sensor 30 outputs acceleration data 50 corresponding to the acceleration in each axial direction of the air vehicle 20. The processor 34 acquires the position of the air vehicle 20 based on the positioning data 48 and / or the acceleration data 50.

[0059] Note that if the processor 34 acquires the position of the flying object 20 based only on the positioning data 48, the acceleration sensor 30 may be omitted. On the other hand, if the processor 34 acquires the position of the flying object 20 based only on the acceleration data 50, the positioning unit 28 may be omitted.

[0060] When the processor 34 acquires the position of the flying object 20 based on the positioning data 48, the position in the absolute coordinate system is derived based on the positioning data 48. On the other hand, when the processor 34 acquires the imaging position based on the acceleration data 50, the amount of change in position relative to a reference position defined in the relative coordinate system is derived based on the acceleration data 50.

[0061] Furthermore, the flying object 20 may be equipped with other devices for detecting the position of the flying object 20 instead of or in addition to the positioning unit 28 and / or the acceleration sensor 30. Examples of such other devices include a LiDAR scanner, a stereo camera, a magnetic compass, a barometric altimeter, or an ultrasonic sensor.

[0062] As an example, as shown in FIG. 3, the imaging device 60 includes an imaging lens 62, an aperture 64, an aperture actuator 66, a mechanical shutter 68, a shutter actuator 70, a controller 72, an image sensor 74, and an image sensor driver 76.

[0063] The controller 72 and the image sensor driver 76 are connected to the processor 34 via the input / output I / F 24 and the bus 40. The imaging lens 62 includes, for example, an objective lens (not shown) and a focus lens (not shown). The imaging device 60 also includes a zoom lens (not shown). The imaging lens 62 is disposed on the object side of the aperture 64, and the zoom lens is disposed between the aperture 64 and the mechanical shutter 68.

[0064] The aperture actuator 66 has a power transmission mechanism (not shown) and an aperture motor (not shown). The aperture 64 has an aperture 64A, and the size of the aperture 64A is variable. The aperture 64A is formed by multiple blades (not shown). The multiple blades are connected to the power transmission mechanism. An aperture motor is connected to the power transmission mechanism, and the power transmission mechanism transmits the power of the aperture motor to the multiple blades. The multiple blades change the size of the aperture 64A by operating in response to the power transmitted from the power transmission mechanism. The aperture 64 adjusts exposure by changing the size of the aperture 64A. The aperture value is determined by the size of the aperture 64A. The aperture actuator 66 is connected to a controller 72.

[0065] The controller 72 is a device having a computer including, for example, a CPU, NVM, RAM, etc. (All of these are omitted from the drawing.) Note that although a computer is used here as an example, this is merely one example, and devices including an ASIC, FPGA, and / or PLD may also be used. Furthermore, the controller 72 may be implemented by combining hardware and software, for example. The controller 72 controls the aperture actuator 66 in accordance with instructions from the processor 34.

[0066] The image sensor 74 includes a photoelectric conversion element 78 and a signal processing circuit 80. The image sensor 74 is, for example, a CMOS image sensor. In the present embodiment, a CMOS image sensor is used as the image sensor 74, but the technology of the present disclosure is not limited to this. For example, the technology of the present disclosure can be applied even if the image sensor 74 is another type of image sensor, such as a CCD image sensor. The photoelectric conversion element 78 is connected to an image sensor driver 76. The image sensor driver 76 controls the photoelectric conversion element 78 in accordance with instructions from the processor 34.

[0067] The photoelectric conversion element 78 has a light receiving surface 78A on which a plurality of pixels (not shown) are provided. The photoelectric conversion element 78 outputs electrical signals output from the plurality of pixels to the signal processing circuit 80 as imaging data 82. The signal processing circuit 80 digitizes the analog imaging data 82 input from the photoelectric conversion element 78. The signal processing circuit 80 is connected to the input / output I / F 24. The digitized imaging data 82 is image data representing the synthesis image 132, and is stored in the storage 36 after undergoing various processes by the processor 34.

[0068] The mechanical shutter 68 is, for example, a focal plane shutter, and is disposed between the diaphragm 64 and the light receiving surface 78A. The mechanical shutter 68 includes a front curtain 68A and a rear curtain 68B. For example, the front curtain 68A and the rear curtain 68B each include a plurality of blades (not shown). The front curtain 68A is disposed closer to the subject than the rear curtain 68B.

[0069] The shutter actuator 70 has a link mechanism (not shown), a leading curtain solenoid (not shown), and a trailing curtain solenoid (not shown). The leading curtain solenoid is the drive source for the leading curtain 68A and is mechanically linked to the leading curtain 68A via the link mechanism. The trailing curtain solenoid is the drive source for the trailing curtain 68B and is mechanically linked to the trailing curtain 68B via the link mechanism. The shutter actuator 70 is connected to a controller 72. The controller 72 controls the shutter actuator 70 in accordance with instructions from the processor 34.

[0070] The first curtain solenoid generates power under the control of the controller 72 and applies the generated power to the first curtain 68A to selectively wind up or down the first curtain 68A. The second curtain solenoid generates power under the control of the controller 72 and applies the generated power to the second curtain 68B to selectively wind up or down the second curtain 68B. In the imaging device 60, the opening and closing of the first curtain 68A and the second curtain 68B are controlled by the processor 34 to adjust the amount of exposure to the image sensor 74. The shutter speed of the mechanical shutter 68 is determined by the time the first curtain 68A and the second curtain 68B are open.

[0071] Although a focal plane shutter has been described as an example of the mechanical shutter 68, this is merely an example, and the mechanical shutter 68 may also be a lens shutter. Also, although an example has been described in which the shutter speed of the mechanical shutter 68 is derived and set, this is merely an example. For example, the shutter speed of an electronic shutter (e.g., an electronic front-curtain shutter or a fully electronic shutter) may also be derived and set in the same manner as above.

[0072] 2 and 3, the computer 26 is common to the flying object 20 and the imaging device 60, but the computer 26 may be configured by a first computer provided in the flying object 20 and a second computer provided in the imaging device 60. The computer 26 is mounted on the flying object 20, but may also be mounted on the imaging device 60. The computer 26 may also be mounted on a transmitter or a base station. An example of the hardware configuration of the flying imaging device 10 has been described above.

[0073] Next, a problem of the first embodiment will be described. When the aircraft 20 equipped with the imaging device 60 reaches a target waypoint 5 (hereinafter referred to as a "target waypoint 5"), the following technique is assumed as a technique for causing the imaging device 60 to capture an image of the imaging target area 3 corresponding to the target waypoint 5.

[0074] That is, in the assumed technology (hereinafter referred to as the "expected technology"), an area corresponding to the flight position of the aircraft 20 (hereinafter referred to as the "flight position corresponding area") is captured by the imaging device 60 before the aircraft 20 reaches the target waypoint 5, and the brightness of the flight position corresponding area is detected based on the image obtained by capturing the image. Next, a shutter speed corresponding to the detected brightness and a flight speed corresponding to the shutter speed are calculated. Then, at the target waypoint 5, the imaging target area 3 is captured by the imaging device 60 at the calculated shutter speed while the aircraft 20 flies at the calculated flight speed.

[0075] In the above-described assumed technology, the flight position corresponding area is imaged by the imaging device 60 before the flying object 20 reaches the target waypoint 5, and the brightness of the flight position corresponding area is detected based on the image obtained by imaging. The flight position corresponding area imaged by the imaging device 60 is an area located in front of the imaging target area 3 corresponding to the target waypoint 5. For this reason, the flight position corresponding area imaged by the imaging device 60 may have a different brightness than the imaging target area 3 corresponding to the target waypoint 5. When the brightness of the flight position corresponding area and the brightness of the imaging target area 3 differ in this way, it is not possible to image the imaging target area 3 at a shutter speed corresponding to the brightness of the imaging target area 3.

[0076] Therefore, the following method can be considered to capture an image of the imaging target area 3 at a shutter speed corresponding to the brightness of the imaging target area 3. That is, when the flying vehicle 20 reaches the target waypoint 5, the flying vehicle 20 is temporarily stopped at the target waypoint 5, and in this state, the imaging device 60 captures the imaging target area 3, and based on the resulting image, the brightness of the imaging target area 3 is detected, and a shutter speed corresponding to the detected brightness can be calculated.

[0077] However, the above-described method has a problem in that, because the flying object 20 is stopped temporarily at the target waypoint 5, it is not possible to have the imaging device 60 capture an image of the imaging target area 3 while the flying object 20 is flying. Therefore, in the first embodiment, the flying object 20 is caused to fly at the target waypoint 5 at a flight speed corresponding to the shutter speed, and the imaging device 60 is caused to capture an image of the imaging target area 3 at a shutter speed corresponding to the brightness of the imaging target area 3. To achieve this, in the first embodiment, the processor 34 performs the flight imaging process (see FIGS. 4 to 9) described below.

[0078] 4 as an example, a flight imaging program 90 is stored in the storage 36. The processor 34 reads the flight imaging program 90 from the storage 36 and executes the read flight imaging program 90 on the RAM 38. The processor 34 performs flight imaging processing for imaging the imaging target area 3 while the flight imaging device 10 is flying, in accordance with the flight imaging program 90 executed on the RAM 38. The flight imaging processing is realized by the processor 34 operating as a data request unit 92, a brightness acquisition unit 94, an imaging condition acquisition unit 96, a flight speed control unit 98, a reach determination unit 100, and an imaging control unit 102 in accordance with the flight imaging program 90.

[0079] 5 as an example, the overhead camera 160 captures an image of the entire wall surface 2A including multiple imaging target areas 3 at a timing before the flying object 20 reaches the target waypoint 5 (for example, a timing before the flying object 20 starts flying along the flight route 4). The entire wall surface 2A is captured by the overhead camera 160, so that an image 162 (hereinafter referred to as "wall surface image 162") including an image of the entire wall surface 2A is obtained.

[0080] The overhead camera 160 derives the brightness of each imaging target area 3 corresponding to each waypoint 5 based on the wall surface image 162. For example, the wall surface image 162 is an RGB image formed using the three primary colors of light, and the overhead camera 160 generates an image 164 based on the RGB image. The image 164 is an image obtained by converting the color space of the RGB image into a color space different from RGB. Hereinafter, the image 164 will be referred to as the "converted image 164." The converted image 164 includes a brightness component that indicates the degree of brightness. Examples of color spaces for the converted image 164 include the Lab color space, the Lch color space, and the HSV color space. The overhead camera 160 derives the brightness of each imaging target area 3 corresponding to each waypoint 5 based on the brightness component included in the converted image 164.

[0081] The brightness of the imaging target area 3 may be derived based on a standard light source (e.g., D50 or D65) defined by the CIE. The brightness of the imaging target area 3 may be a representative value of the overall brightness of the imaging target area 3 (e.g., the minimum value, maximum value, or median value), or may be the average value of the overall brightness of the imaging target area 3. The brightness of the imaging target area 3 may be the overall brightness of the imaging target area 3, or the brightness of a portion of the imaging target area 3. The brightness of a portion of the imaging target area 3 may be the brightness of the center of the imaging target area 3, or the brightness of a portion different from the center of the imaging target area 3.

[0082] The overhead camera 160 then generates information regarding the brightness of each image capture target area 3 corresponding to each waypoint 5 as brightness information 166, and stores the generated brightness information 166. For example, the brightness information 166 is information that represents the relationship between a number indicating the order of each waypoint 5 (hereinafter referred to as a "waypoint number") and the brightness of the image capture target area 3. Figure 5 shows, as an example of the brightness information 166, an example of a specific value of the brightness of the image capture target area 3 corresponding to each waypoint number. Brightness is represented, for example, by a value that indicates the degree of brightness.

[0083] The data request unit 92 transmits a request signal 104 to the overhead camera 160 via the communication device 32 to request brightness data 168. The brightness data 168 is data indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5. The request signal 104 includes information indicating the waypoint number corresponding to the target waypoint 5.

[0084] When the overhead camera 160 receives the request signal 104, it extracts the brightness corresponding to the waypoint number indicated by the request signal 104 from the brightness information 166 and transmits brightness data 168 indicating the extracted brightness to the communication device 32. In the example shown in Figure 5, the waypoint number indicated by the request signal 104 is "No. 1," and brightness data 168 indicating "20," which is the degree of brightness corresponding to "No. 1," is transmitted to the communication device 32 by the overhead camera 160.

[0085] The brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 168 received by the communication device 32. In the example shown in FIG. 5, the brightness acquired by the brightness acquisition unit 94 is the brightness detected at the timing before the flying object 20 starts moving along the flight route 4.

[0086] 6 , a table 110 is stored in the storage 36. The table 110 is an example of "relationship information" according to the technology of the present disclosure, and the storage 36 is an example of "memory" according to the technology of the present disclosure. The table 110 includes a first table 112 and a second table 114. The first table 112 indicates the relationship between brightness, shutter speed, and aperture value, and the second table 114 indicates the relationship between shutter speed and flight speed.

[0087] 6 shows an example of specific values ​​of brightness, shutter speed, and aperture value for the first table 112, and an example of specific values ​​of shutter speed and flight speed for the second table 114. The unit of shutter speed is "seconds (s)," and the unit of flight speed is "meters per second (m / s)."

[0088] The shutter speed is set, for example, to a value that causes image blurring that occurs during exposure on the light-receiving surface 78A to fall within an acceptable range. Image blurring is defined, for example, by the distance the image moves on the light-receiving surface 78A (for example, a distance corresponding to the number of pixels). The acceptable range is set, for example, based on the image quality required for the composite image 132 (for example, image quality that allows feature points to be identified). The aperture value is set based on brightness and / or shutter speed. The flight speed is set, for each shutter speed, to a value that causes image blurring to fall within an acceptable range.

[0089] 6, the table 110 includes a first table 112 and a second table 114, but may be a single table showing the relationships between brightness, shutter speed, aperture value, and flight speed. Also, in the example shown in FIG. 6, the first table 112 and the second table 114 are stored in the storage 36, but a first relational expression showing the relationship between brightness, shutter speed, and aperture value may be stored instead of the first table 112, and a second relational expression showing the relationship between shutter speed and flight speed may be stored instead of the second table 114. Also, the first table 112 stores different aperture values ​​depending on the brightness and shutter speed, but the aperture value may be constant.

[0090] The imaging condition acquisition unit 96 acquires imaging conditions based on the brightness acquired by the brightness acquisition unit 94. The imaging conditions include a shutter speed, an aperture value, and a flight speed. The imaging condition acquisition unit 96 acquires a shutter speed and an aperture value corresponding to the brightness acquired by the brightness acquisition unit 94 based on a first table 112. The imaging condition acquisition unit 96 also acquires a flight speed corresponding to the shutter speed acquired based on the first table 112 based on a second table 114.

[0091] 7 , the flight speed control unit 98 controls the plurality of motors 44 via the motor driver 46 based on the flight speed acquired by the imaging condition acquisition unit 96, thereby setting the flight speed of the flying object 20 to the flight speed acquired by the imaging condition acquisition unit 96. This adjusts the rotation speed of the plurality of propellers 42, and the flight speed of the flying object 20 is set to the flight speed acquired by the imaging condition acquisition unit 96.

[0092] 8 , the storage 36 stores flight route information 116 indicating the flight route 4. The flight route information 116 includes position information indicating the positions of a plurality of waypoints 5 set on the flight route 4.

[0093] The arrival determination unit 100 acquires the position of the aircraft 20 based on the positioning data 48 input from the positioning unit 28 and / or the acceleration data 50 input from the acceleration sensor 30. Then, the arrival determination unit 100 determines whether the aircraft 20 has reached the target waypoint 5 based on the acquired position of the aircraft 20 and the position of the target waypoint 5 indicated by the flight route information 116.

[0094] When the arrival determination unit 100 determines that the flying object 20 has reached the target waypoint 5, the imaging control unit 102 causes the imaging device 60 to capture an image of the imaging target area 3 corresponding to the target waypoint 5 at the shutter speed and aperture value acquired by the imaging condition acquisition unit 96.

[0095] Specifically, the imaging control unit 102 controls the shutter actuator 70 via the controller 72 to open and close the front curtain 68A and the rear curtain 68B at a time corresponding to the shutter speed acquired by the imaging condition acquisition unit 96. Also, the imaging control unit 102 controls the diaphragm actuator 66 via the controller 72 to adjust the size of the opening 64A of the diaphragm 64 to a size corresponding to the f-stop value acquired by the imaging condition acquisition unit 96. Also, the imaging control unit 102 controls the image sensor 74 via the image sensor driver 76 to output imaging data 82 from the image sensor 74. The imaging data 82 is image data representing the compositing image 132 obtained by capturing an image of the imaging target area 3, and is stored in the storage 36 after undergoing various processes by the processor 34 (see FIG. 2 ).

[0096] The flight speed of the flying object 20 is maintained at the flight speed acquired by the imaging condition acquisition unit 96 until the imaging target area 3 is imaged by the imaging control unit 102. After the imaging target area 3 is imaged by the imaging control unit 102, the flight speed of the flying object 20 may be changed to a flight speed higher than the flight speed acquired by the imaging condition acquisition unit 96 until the flight speed is again controlled by the flight speed control unit 98 in accordance with the next target waypoint 5. Changing the flight speed to a speed higher than the flight speed acquired by the imaging condition acquisition unit 96 makes it possible to improve the efficiency of the imaging work.

[0097] Next, the operation of the flight imaging device 10 according to the first embodiment will be described with reference to Fig. 9. Fig. 9 shows an example of the flow of flight imaging processing according to the first embodiment.

[0098] 9 , first, in step ST10, the data request unit 92 transmits a request signal 104 to the overhead camera 160 via the communication device 32 to request brightness data 168 indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5 (see FIG. 5 ). After the processing of step ST10 is executed, the flight imaging processing proceeds to step ST12.

[0099] In step ST12, the brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 168 transmitted from the overhead camera 160 and received by the communication device 32 (see FIG. 5 ). After the processing of step ST12 is executed, the flight imaging processing proceeds to step ST14.

[0100] In step ST14, the imaging condition acquisition unit 96 acquires the shutter speed and aperture value corresponding to the brightness acquired in step ST12 based on the first table 112 stored in the storage 36 (see FIG. 6). The imaging condition acquisition unit 96 also acquires the flight speed corresponding to the shutter speed acquired based on the first table 112 based on the second table 114 stored in the storage 36 (see FIG. 6). After the processing of step ST14 is executed, the flight imaging processing proceeds to step ST16.

[0101] In step ST16, the flight speed control unit 98 controls the motors 44 via the motor drivers 46 to set the flight speed of the flying object 20 to the flight speed acquired in step ST14 (see FIG. 7). This adjusts the rotation speed of the propellers 42, and sets the flight speed of the flying object 20 to the flight speed acquired in step ST14. After the processing of step ST16 is executed, the flight imaging processing proceeds to step ST18.

[0102] In step ST18, the arrival determination unit 100 acquires the position of the aircraft 20 based on the positioning data 48 input from the positioning unit 28 and / or the acceleration data 50 input from the acceleration sensor 30 (see FIG. 8 ). Then, the arrival determination unit 100 determines whether the aircraft 20 has reached the target waypoint 5 based on the acquired position of the aircraft 20 and the position of the waypoint 5 indicated by the flight route information 116 stored in the storage 36 (see FIG. 8 ). In step ST18, if the aircraft 20 has reached the target waypoint 5, the determination is affirmative, and the flight imaging process proceeds to step ST20. In step ST18, if the aircraft 20 has not reached the target waypoint 5, the determination is negative, and the flight imaging process executes the process of step ST18 again.

[0103] In step ST20, the imaging control unit 102 causes the imaging device 60 to capture an image of the imaging target area 3 corresponding to the target waypoint 5 at the shutter speed and aperture value acquired in step ST14 (see FIG. 8 ). After the processing of step ST20 is executed, the flight imaging processing proceeds to step ST22.

[0104] In step ST22, the processor 34 determines whether a condition for terminating the flight imaging process (termination condition) is met. Examples of the termination condition include a condition in which the imaging target area 3 corresponding to the final waypoint 5 has been imaged by the imaging device 60, or a condition in which the user has given the flight imaging device 10 an instruction to terminate the flight imaging process. If the termination condition is not met in step ST22, the determination is negative, and the flight imaging process proceeds to step ST10. If the termination condition is met in step ST22, the determination is positive, and the flight imaging process terminates. The control method described above as the operation of the flight imaging device 10 is an example of a "control method" according to the technology of the present disclosure.

[0105] As described above, in the flight imaging device 10 according to the first embodiment, the processor 34 acquires the brightness of the imaging target area 3 to be imaged by the imaging device 60 from the target waypoint 5 at a timing before the flying object 20 reaches the target waypoint 5 (see FIG. 5 ). The processor 34 also acquires a shutter speed corresponding to the acquired brightness and a flight speed corresponding to the shutter speed (see FIG. 6 ). The processor 34 then causes the imaging device 60 to image the imaging target area 3 at the acquired shutter speed while flying the flying object 20 at the target waypoint 5 at the acquired flight speed (see FIG. 7 ) (see FIG. 8 ). Therefore, while flying the flying object 20 at the target waypoint 5 at a flight speed corresponding to the shutter speed, the processor 34 can cause the imaging device 60 to image the imaging target area 3 at a shutter speed corresponding to the brightness of the imaging target area 3.

[0106] Furthermore, the processor 34 acquires the shutter speed corresponding to the brightness and the flight speed corresponding to the shutter speed based on a table 110 that represents the relationship between brightness, shutter speed, and flight speed (see FIG. 6). Therefore, the shutter speed and flight speed that are predetermined by the table 110 can be acquired based on the brightness.

[0107] The computer 26 also includes a storage 36 that stores a table 110 (see FIG. 6 ). Therefore, for example, the shutter speed and the flight speed can be obtained based on the table 110 stored in the storage 36 that is directly connected to the processor 34.

[0108] The brightness is the brightness detected using the overhead camera 160 (see FIG. 5). Therefore, for example, the brightness of the multiple imaging target areas 3 can be detected from a position farther from the wall surface 2A than the flying imaging device 10.

[0109] The overhead camera 160 also captures an image of the entire wall surface 2A including the multiple imaging target areas 3 (see FIG. 5 ). Therefore, the number of times images are captured can be reduced compared to when the brightness of each imaging target area 3 is detected individually based on images obtained by capturing images of the multiple imaging target areas 3 one by one, thereby improving the workability when detecting the brightness of the multiple imaging target areas 3.

[0110] The brightness is the brightness detected at the timing before the flying object 20 starts flying along the flight route 4 (see FIG. 5 ). As a result, a shutter speed corresponding to the brightness and a flight speed corresponding to the shutter speed are acquired based on the brightness detected at the timing before the flying object 20 starts flying along the flight route 4. Therefore, for example, compared to when the brightness is detected at the timing when the flying imaging device 10 reaches the target waypoint 5, it is possible to have the flying imaging device 10 capture images with an appropriate shutter speed and flight speed set at the target waypoint 5 (for example, a shutter speed and flight speed that do not cause image blur).

[0111] Furthermore, a plurality of waypoints 5 are set on the flight route 4, and the processor 34 causes the imaging device 60 to capture an image of the imaging target area 3 at an aperture value (see FIG. 6 ) corresponding to the brightness and / or shutter speed for each waypoint 5. Therefore, even if the conditions related to the brightness and / or shutter speed differ for each waypoint 5, the imaging target area 3 can be captured at an aperture value corresponding to the brightness and / or shutter speed.

[0112] Furthermore, for each waypoint 5, the imaging target area 3 is imaged by the imaging device 60 to obtain a compositing image 132 (see FIG. 8 ). The compositing images 132 are images in which adjacent compositing images 132 partially overlap. Therefore, based on feature points included in the overlapping portions of the adjacent imaging target areas 3, it is possible to composite the compositing images 132 corresponding to the adjacent imaging target areas 3.

[0113] In the first embodiment, the processor 34 may cause the imaging device 60 to capture images of the imaging target area 3 at a constant aperture value at multiple waypoints 5. In this case, the process of operating the aperture 64 for each waypoint 5 can be omitted.

[0114] Second Embodiment Next, a second embodiment will be described.

[0115] In the first embodiment, the overhead camera 160 captures an image of the entire wall surface 2A, including the multiple imaging target areas 3, at a timing before the flying object 20 starts flying along the flight route 4. In the second embodiment, the timing at which the wall surface 2A is captured by the overhead camera 160 differs from that in the first embodiment. This will be specifically described below.

[0116] 10 , after the flying object 20 starts moving along the flight route 4 and before the flying object 20 reaches the target waypoint 5, the data request unit 92 transmits a request signal 104 to the overhead camera 160 via the communication device 32 to request brightness data 168. The brightness data 168 is data indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5. The request signal 104 includes information indicating the waypoint number corresponding to the target waypoint 5.

[0117] When the overhead camera 160 receives the request signal 104, it captures an image of the entire wall surface 2A, including the multiple imaging target areas 3. This results in a wall surface image 162. The overhead camera 160 also converts the wall surface image 162 into a converted image 164, and derives the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness component contained in the converted image 164. The method for deriving the brightness of the imaging target area 3 is the same as in the first embodiment. The overhead camera 160 then transmits brightness data 168 indicating the derived brightness to the communication device 32.

[0118] The overhead camera 160 may extract an image that includes only the imaging target area 3 that corresponds to the waypoint number indicated by the request signal 104 from the wall surface image 162 obtained by capturing an image of the entire wall surface 2A. The overhead camera 160 may then convert the extracted image into a converted image 164, and derive the brightness of only the imaging target area 3 that corresponds to the target waypoint 5 based on the brightness component contained in the converted converted image 164.

[0119] Alternatively, overhead camera 160 may capture only the imaging target area 3 corresponding to the waypoint number indicated by request signal 104. Then, overhead camera 160 may convert the captured image into converted image 164, and derive the brightness of only the imaging target area 3 corresponding to target waypoint 5 based on the brightness component contained in converted image 164.

[0120] The brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 168 received by the communication device 32 .

[0121] As described above in detail, in the second embodiment, the brightness acquired by the brightness acquisition unit 94 is the brightness detected at a timing after the aircraft 20 starts moving along the flight route 4 and before the aircraft 20 reaches the target waypoint 5. As a result, a shutter speed corresponding to the brightness and a flight speed corresponding to the shutter speed are acquired based on the brightness detected at a timing after the aircraft 20 starts moving along the flight route 4 and before the aircraft 20 reaches the target waypoint 5. Therefore, for example, compared to when the brightness is detected at a timing when the flight imaging device 10 reaches the target waypoint 5, it is possible to cause the flight imaging device 10 to perform imaging with an appropriate shutter speed and flight speed set at the target waypoint 5 (for example, a shutter speed and flight speed that do not cause image blur).

[0122] Furthermore, even if the brightness changes after the aircraft 20 starts moving along the flight route 4, the changed brightness can be detected at a timing before the aircraft 20 reaches the target waypoint 5. This allows for improved brightness detection accuracy compared to, for example, when the brightness is detected before the aircraft 20 starts moving along the flight route 4.

[0123] Third Embodiment Next, a third embodiment will be described.

[0124] In the first embodiment, the brightness of each imaging target area 3 corresponding to each waypoint 5 is detected based on a wall surface image 162 obtained by capturing an image using an overhead camera 160. In the third embodiment, the method for detecting the brightness of each imaging target area 3 corresponding to each waypoint 5 differs from that of the first embodiment. This will be described in detail below.

[0125] 11 as an example, in the third embodiment, a flight imaging device 210 separate from the flight imaging device 10 is used instead of the overhead camera 160. Hereinafter, when it is necessary to distinguish between the flight imaging devices 10 and 210, the flight imaging device 10 will be referred to as the "first flight imaging device 10" and the flight imaging device 210 will be referred to as the "second flight imaging device 210."

[0126] The second flight imaging device 210 has the same hardware configuration as the first flight imaging device 10. Specifically, the second flight imaging device 210 includes an air vehicle 220 and an imaging device 260. The air vehicle 220 has the same configuration as the air vehicle 20 included in the first flight imaging device 10, and the imaging device 260 has the same configuration as the imaging device 60 included in the first flight imaging device 10. The air vehicle 220 included in the second flight imaging device 210 is an example of a "second moving body" according to the technology of the present disclosure. The imaging device 260 included in the second flight imaging device 210 is an example of a "sensor" according to the technology of the present disclosure.

[0127] The second flight imaging device 210 flies along the flight route 4 at a timing before the first flight imaging device 10 reaches the target waypoint 5 (for example, a timing before the first flight imaging device 10 starts flying along the flight route 4), and captures an image of the imaging target area 3 corresponding to each waypoint 5 each time it reaches a waypoint 5. By capturing an image of each imaging target area 3, an image 262 (hereinafter referred to as an "imaging target area image 262") including an image of the imaging target area 3 is obtained.

[0128] As an example, the second flight imaging device 210 flies along the flight route 4 at a flight speed higher than the flight speed of the first flight imaging device 10 when flying along the flight route 4. The flight speed may be calculated, for example, as the average flight speed from the start to the end of flight along the flight route 4. By setting the flight speed of the second flight imaging device 210 higher than the flight speed of the first flight imaging device 10, it is possible to improve the efficiency of the imaging work by the second flight imaging device 210.

[0129] The second flight imaging device 210 derives the brightness of each imaging target area 3 corresponding to each waypoint 5 based on an imaging target area image 262 obtained each time the flight imaging device 210 reaches a waypoint 5. For example, the imaging target area image 262 is an RGB image formed using the three primary colors of light, and the second flight imaging device 210 converts the RGB image into an image 264. Hereinafter, the image 264 will be referred to as the "converted image 264." The converted image 264 is similar to the converted image 164 (see FIG. 5 ) of the first embodiment. The second flight imaging device 210 then derives the brightness of each imaging target area 3 corresponding to each waypoint 5 based on the brightness components contained in the converted image 264. The method for deriving the brightness of each imaging target area 3 is the same as in the first embodiment.

[0130] The second flight imaging device 210 also generates brightness information 266, which is information related to the brightness of each imaging target area 3 corresponding to each waypoint 5, and stores the generated brightness information 266. The brightness information 266 is similar to the brightness information 166 of the first embodiment (see FIG. 5 ). FIG. 11 shows, as an example of the brightness information 266, an example of a specific value of the brightness of the imaging target area 3 corresponding to each waypoint number.

[0131] The data request unit 92 transmits a request signal 104 to the second flight imaging device 210 via the communication device 32 to request the brightness data 268. The brightness data 268 is data indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5. The request signal 104 includes information indicating the waypoint number corresponding to the target waypoint 5.

[0132] When the second flight imaging device 210 receives the request signal 104, it extracts the brightness corresponding to the waypoint number indicated by the request signal 104 from the brightness information 266 and transmits brightness data 268 indicating the extracted brightness to the communication device 32 of the first flight imaging device 10. In the example shown in Figure 11, the waypoint number indicated by the request signal 104 is "NO. 1", and brightness data 268 indicating "20", which is the degree of brightness corresponding to "NO. 1", is transmitted to the communication device 32 by the second flight imaging device 210.

[0133] The brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 268 received by the communication device 32 .

[0134] As described above in detail, in the third embodiment, the brightness acquired by the brightness acquisition unit 94 is the brightness detected at the timing before the first flight imaging device 10 starts moving along the flight route 4. As a result, a shutter speed corresponding to the brightness and a flight speed corresponding to the shutter speed are acquired based on the brightness detected at the timing before the flying body 20 starts flying along the flight route 4. Therefore, for example, compared to when the brightness is detected at the timing when the flight imaging device 10 reaches the target waypoint 5, it is possible to have the flight imaging device 10 perform imaging with an appropriate shutter speed and flight speed set at the target waypoint 5 (for example, a shutter speed and flight speed that do not cause image blur).

[0135] The brightness acquired by the brightness acquisition unit 94 is the brightness detected using the imaging device 60 of the second flight imaging device 210 from the target waypoint 5. Therefore, the accuracy of brightness detection can be improved compared to, for example, brightness detected using the imaging device 60 from a position farther from the wall surface 2A than the target waypoint 5.

[0136] Furthermore, the imaging device 60 is mounted on the second flying imaging device 210. Therefore, while flying the second flying imaging device 210, the brightness of the imaging target area 3 can be detected at each waypoint 5.

[0137] Furthermore, the second flight imaging device 210 flies along the flight route 4 at a higher flight speed than the first flight imaging device 10. Therefore, the time required for the second flight imaging device 210 to fly along the flight route 4 can be made shorter than the time required for the first flight imaging device 10 to fly along the flight route 4.

[0138] In the third embodiment, the imaging device 260 is used to detect the brightness of the imaging target area 3, but an illuminance sensor (not shown) may be used instead of the imaging device 260. The illuminance sensor is an example of a "sensor" according to the technology of the present disclosure.

[0139] In addition, in the third embodiment, a second flight imaging device 210 separate from the first flight imaging device 10 is used, but the first flight imaging device 10 may be used instead of the second flight imaging device 210. In this case, the flying body 20 provided in the first flight imaging device 10 is an example of the "first moving body" and "second moving body" according to the technology of the present disclosure.

[0140] Fourth Embodiment Next, a fourth embodiment will be described.

[0141] In the third embodiment, the second flight imaging device 210 images the imaging target area 3 corresponding to each waypoint 5 while flying along the flight route 4, each time it reaches a waypoint 5, at a timing before the first flight imaging device 10 starts flying along the flight route 4. In the fourth embodiment, the timing at which the imaging target area 3 corresponding to each waypoint 5 is imaged by the second flight imaging device 210 differs from that in the third embodiment. This will be explained in detail below.

[0142] 12 as an example, the second flying imaging device 210 flies ahead of the first flying imaging device 10. The distance between the centers of the first flying imaging device 10 and the second flying imaging device 210 along the flight route 4 may be set to a distance shorter than the distance between the centers of adjacent waypoints 5, or may be set to a distance longer than the distance between the centers of adjacent waypoints 5.

[0143] The second flight imaging device 210 flies along the flight route 4 ahead of the first flight imaging device 10, and captures an image of the imaging target area 3 corresponding to each waypoint 5 each time it reaches a waypoint 5. By capturing an image of each imaging target area 3, an imaging target area image 262 is obtained.

[0144] The second flight imaging device 210 converts the imaging target area image 262 obtained each time a waypoint 5 is reached into a converted image 264, and derives the brightness of each imaging target area 3 corresponding to each waypoint 5 based on the brightness component contained in the converted image 264. The method for deriving the brightness of each imaging target area 3 is the same as in the third embodiment.

[0145] The second flight imaging device 210 also generates brightness information 266, which is information about the brightness of each imaging target area 3 corresponding to each waypoint 5, and stores the generated brightness information 266. The brightness information 266 is the same as in the third embodiment. Figure 12 shows, as an example of the brightness information 266, an example of a specific value of the brightness of the imaging target area 3 corresponding to each waypoint number.

[0146] After the first flight imaging device 10 starts moving along the flight route 4 and before the first flight imaging device 10 reaches the target waypoint 5, the data request unit 92 transmits a request signal 104 to the second flight imaging device 210 via the communication device 32 to request brightness data 268. The brightness data 268 is data indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5. The request signal 104 includes information indicating the waypoint number corresponding to the target waypoint 5.

[0147] When the second flight imaging device 210 receives the request signal 104, it extracts the brightness corresponding to the waypoint number indicated by the request signal 104 from the brightness information 266 and transmits brightness data 268 indicating the extracted brightness to the communication device 32 of the first flight imaging device 10. In the example shown in Figure 12, the waypoint number indicated by the request signal 104 is "No. 3", and brightness data 268 indicating "60", which is the degree of brightness corresponding to "No. 3", is transmitted to the communication device 32 by the second flight imaging device 210.

[0148] The brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 268 received by the communication device 32 .

[0149] As described above in detail, in the fourth embodiment, the brightness acquired by the brightness acquisition unit 94 is the brightness detected at a timing after the first flight imaging device 10 starts moving along the flight route 4 and before the first flight imaging device 10 reaches the target waypoint 5. As a result, a shutter speed corresponding to the brightness and a flight speed corresponding to the shutter speed are acquired based on the brightness detected at a timing after the flight body 20 starts moving along the flight route 4 and before the flight body 20 reaches the target waypoint 5. Therefore, for example, compared to when the brightness is detected at a timing when the flight imaging device 10 reaches the target waypoint 5, it is possible to have the flight imaging device 10 perform imaging with an appropriate shutter speed and flight speed set at the target waypoint 5 (for example, a shutter speed and flight speed that do not cause image blur).

[0150] Furthermore, even if the brightness changes after the aircraft 20 starts moving along the flight route 4, the changed brightness can be detected at a timing before the aircraft 20 reaches the target waypoint 5. This allows for improved brightness detection accuracy compared to, for example, when the brightness is detected before the aircraft 20 starts moving along the flight route 4.

[0151] Fifth Embodiment Next, a fifth embodiment will be described.

[0152] In the fifth embodiment, the processor 34 provided in the flight imaging device 10 acquires a plurality of compositing images 132 and generates a composite image 130 by compositing adjacent compositing images 132. The processor 34 also performs a correction process on the composite image 130. The correction process is referred to as a "specific process" according to the technology of the present disclosure and will be described in detail below.

[0153] 13 as an example, a composite image generation program 120 is stored in the storage 36. The processor 34 reads the composite image generation program 120 from the storage 36 and executes the read composite image generation program 120 on the RAM 38. The processor 34 performs a composite image generation process in accordance with the composite image generation program 120 executed on the RAM 38. The composite image generation process is realized by the processor 34 operating as an image synthesis unit 122, a brightness information acquisition unit 124, and a correction processing unit 126 in accordance with the composite image generation program 120. The processor 34 is an example of a "second processor" according to the technology of the present disclosure.

[0154] As an example, as shown in Figure 14, the image synthesis unit 122 generates a synthetic image 130 by synthesizing adjacent synthesis images 132 obtained by capturing images of each imaging target area 3.

[0155] The brightness information acquisition unit 124 acquires brightness information 166 generated by the overhead camera 160. As described in the first embodiment, the brightness information 166 is information generated based on the brightness components contained in the converted image 164, and is information indicating the brightness of each imaging target area 3 corresponding to each waypoint 5.

[0156] The correction processing unit 126 performs a correction process on the composite image 130 synthesized by the image synthesis unit 122. Specifically, the correction process is a process of correcting the luminance of the composite image 130 based on the brightness information 166 acquired by the brightness information acquisition unit 124. In the correction process, the luminance is corrected for areas of the composite image 130 excluding areas indicating feature points.

[0157] For example, the correction process may involve reducing the brightness of the brightest region of the composite image 130 (for example, a region where blown-out highlights occur). Alternatively, the correction process may involve correcting the difference in brightness between adjacent compositing images 132. Alternatively, the correction process may involve correcting the difference in brightness between compositing images 132 that are located apart. Examples of the compositing images 132 that are located apart include compositing images 132 that are located in a positional relationship such that one compositing image 132 is placed between compositing images 132 that are located apart, or compositing images 132 that are located in two of the four corners of the composite image 130.

[0158] An example of the process for correcting the difference in luminance between the compositing images 132 is a process for bringing the luminance between the compositing images 132 closer together. Furthermore, for example, as a correction process, a process for correcting the luminance distribution of the composite image 130 is executed based on the brightness distribution of the multiple imaging target areas 3. An example of the process for correcting the luminance distribution of the composite image 130 is a process for correcting unevenness in luminance. The above correction process is executed on the composite image 130. The correction process is an example of a "specific process" and a "correction process" according to the technology of the present disclosure.

[0159] The composite image generation process may be performed, for example, after all of the composite images 132 have been obtained for the wall surface 2A, or may be performed each time the second or subsequent frame of the composite image 132 is obtained.

[0160] Next, the operation of the flying imaging device 10 according to the fifth embodiment will be described with reference to Fig. 15. Fig. 15 shows an example of the flow of a composite image generation process according to the fifth embodiment.

[0161] 15 , first, in step ST30, the image composition unit 122 generates a composite image 130 by combining adjacent composite images 132 with respect to a plurality of composite images 132 obtained by capturing images of each imaging target area 3. After the processing of step ST30 is executed, the composite image generation processing proceeds to step ST32.

[0162] In step ST32, the brightness information acquisition unit 124 acquires the brightness information 166 generated by the overhead camera 160. After the process of step ST32 is executed, the composite image generation process proceeds to step ST34.

[0163] In step ST34, correction processing unit 126 performs correction processing based on brightness information 166 acquired in step ST32, thereby correcting the brightness of composite image 130 generated in step ST30. This results in composite image 130 with corrected brightness. After the processing of step ST34 is executed, the composite image generation processing ends.

[0164] As described above in detail, in the fifth embodiment, the composite image 130 is generated by combining adjacent compositing images 132. Then, the luminance of the composite image 130 is corrected by performing a correction process on the composite image 130 based on the brightness information 166 indicating the brightness of each imaging target area 3. Therefore, it is possible to prevent the appearance of the composite image 130 from being deteriorated due to unevenness in brightness that occurs when adjacent compositing images 132 are combined, for example.

[0165] The correction process also includes a process of correcting the difference in luminance between the compositing images 132. Therefore, it is possible to obtain a composite image 130 in which the difference in luminance between the compositing images 132 has been corrected based on the brightness of each imaging target area 3.

[0166] The correction process also includes a process of correcting the luminance distribution of the composite image 130 based on the brightness distribution of the plurality of imaging target areas 3. Therefore, it is possible to obtain the composite image 130 whose luminance distribution has been corrected based on the brightness distribution of the plurality of imaging target areas 3.

[0167] In the above description, the composite image generation process is executed in the flight imaging device 10, but it may also be executed in an external device (not shown) communicatively connected to the flight imaging device 10.

[0168] The identification process may also include a notification process that issues a notification when the difference in brightness between the image capture target areas 3 exceeds a predetermined value. The predetermined value is set, for example, to an upper limit of the brightness difference that does not affect the inspection or measurement of the wall surface 2A based on the composite image 130. For example, the notification process may include a process that issues an alarm sound. The notification process is an example of the "identification process" and "notification process" according to the technology of the present disclosure. In this way, when the notification process that issues a notification when the difference in brightness between the image capture target areas 3 exceeds the predetermined value is executed, the user or the like can be made aware that the difference in brightness between the image capture target areas 3 has exceeded the predetermined value.

[0169] Although the above embodiment illustrates the flying imaging device 10 as an example of a moving object, any moving object that moves along a moving route may be used, such as a car, a motorcycle, a bicycle, a dolly, a gondola, an airplane, a flying object, or a ship.

[0170] Furthermore, in the above embodiment, the term "plurality of waypoints 5" refers to all of the waypoints 5 set on the flight route 4, but may refer to some of the waypoints 5 among all of the waypoints 5 set on the flight route 4. Furthermore, the number of waypoints 5 may be any number.

[0171] In addition, in the above embodiment, a waypoint 5 set on the flight route 4 is used as an example of a target position, but a target position that is a position with a different concept from the waypoint 5 may also be used.

[0172] Furthermore, in the above embodiment, the imaging device 60 images the imaging target area 3 in order to obtain the image to be synthesized 132 , but the imaging target area 3 may also be imaged for a purpose other than obtaining the image to be synthesized 132 .

[0173] In addition, in the above embodiment, processor 34 is exemplified, but at least one other CPU, at least one GPU, and / or at least one TPU may be used instead of processor 34 or together with processor 34.

[0174] Furthermore, in the above embodiment, an example was described in which the flight imaging program 90 and the composite image generation program 120 are stored in the storage 36, but the technology of the present disclosure is not limited to this. For example, the flight imaging program 90 and / or the composite image generation program 120 may be stored in a portable, non-transitory, computer-readable storage medium (hereinafter simply referred to as a "non-transitory storage medium") such as an SSD or USB memory. The flight imaging program 90 and / or the composite image generation program 120 stored in the non-transitory storage medium may be installed in the computer 26 of the flight imaging device 10.

[0175] In addition, the flight imaging program 90 and / or the composite image generation program 120 may be stored in a storage device such as another computer or server device connected to the flight imaging device 10 via a network, and the flight imaging program 90 and / or the composite image generation program 120 may be downloaded and installed on the computer 26 in response to a request from the flight imaging device 10.

[0176] Furthermore, it is not necessary to store all of the flight imaging program 90 and / or composite image generation program 120 in a storage device such as another computer or server device connected to the flight imaging device 10, or in the storage 36; only part of the flight imaging program 90 and / or composite image generation program 120 may be stored.

[0177] Furthermore, although the flying imaging device 10 has a built-in computer 26, the technology of the present disclosure is not limited to this, and for example, the computer 26 may be provided outside the flying imaging device 10.

[0178] In addition, in the above embodiment, the computer 26 includes the processor 34, the storage 36, and the RAM 38, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 26. Furthermore, a combination of a hardware configuration and a software configuration may be used instead of the computer 26.

[0179] Furthermore, the following various processors can be used as hardware resources for executing the various processes described in the above embodiments. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing various processes by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, and ASICs, which are processors with a circuit configuration designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute various processes.

[0180] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0181] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes various processes. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute various processes, on a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.

[0182] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The above-described gaze detection process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0183] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0184] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0185] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A first processor is provided, The first processor, acquiring brightness of an imaging target area imaged by the imaging device from a first moving body mounted with the imaging device at a timing before the first moving body reaches a target position; acquiring a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed; While moving the first moving body at the moving speed in the target position, the imaging device is caused to capture an image of the imaging target area at the shutter speed. Control device.

2. The first processor acquires the shutter speed corresponding to the brightness and the movement speed corresponding to the shutter speed based on relationship information indicating a relationship between the brightness, the shutter speed, and the movement speed. The control device according to claim 1 .

3. The relationship information is stored in a memory. The control device according to claim 2.

4. The brightness is the brightness detected from the target position using a sensor. The control device according to claim 1 .

5. The sensor is mounted on a second moving body. The control device according to claim 4.

6. The target position is set on a moving route, The brightness is detected while the second moving object is moving along the moving route at a speed higher than the moving speed. The control device according to claim 5.

7. The brightness is detected using an overhead camera that overlooks a subject including the imaging target area. The control device according to claim 1 .

8. The subject includes a plurality of the imaging target regions. The control device according to claim 7.

9. The target position is set on a moving route, The brightness is detected at a timing before the first moving body starts moving along the moving route. The control device according to claim 1 .

10. The target position is set on a moving route, The brightness is detected at a timing after the first moving object starts moving along the moving route and before the first moving object reaches the target position. The control device according to claim 1 .

11. A plurality of target positions are set as destinations to which the first moving body moves, The first processor causes the imaging device to capture the imaging target area at a constant aperture value at a plurality of the target positions. The control device according to claim 1 .

12. A plurality of target positions are set as destinations to which the first moving body moves, The first processor causes the imaging device to capture the imaging target area at an aperture value corresponding to the brightness and / or the shutter speed for each of the target positions. The control device according to claim 1 .

13. A plurality of target positions are set as destinations to which the first moving body moves, The first processor acquires, for each of the target positions, an image to be synthesized obtained by causing the imaging device to capture an image of the imaging target area; The images to be combined are images in which adjacent images to be combined partially overlap each other. The control device according to claim 1 .

14. A second processor is further provided for generating a composite image by combining adjacent images for combination, The second processor performs a specific process on the composite image based on the brightness. The control device according to claim 13.

15. The specific processing includes a correction processing for correcting the luminance of the composite image. The control device according to claim 14.

16. The correction process includes a process of correcting the difference in luminance between the images to be combined. The control device according to claim 15.

17. The correction process includes a process of correcting a luminance distribution of the composite image based on the brightness distribution. The control device according to claim 15.

18. The identification process includes a notification process for notifying when the brightness difference between the imaging target regions exceeds a preset value. The control device according to claim 14.

19. acquiring brightness of an imaging target area imaged by the imaging device from a target position at a timing before a first moving body having an imaging device mounted thereon reaches the target position; Obtaining a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed; and While moving the first moving body at the moving speed in the target position, the imaging device captures an image of the imaging target area at the shutter speed. A control method comprising:

20. acquiring brightness of an imaging target area imaged by the imaging device from a target position at a timing before a first moving body having an imaging device mounted thereon reaches the target position; Obtaining a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed; and While moving the first moving body at the moving speed in the target position, the imaging device captures an image of the imaging target area at the shutter speed. A program for causing a computer to execute a process including the steps of: