Flight information control method, control device, and computer program

Abstract

To provide a novel and enhanced flight information generation method capable of improving efficiency in imaging of a target region with a drone which is capable of implementing imaging.SOLUTION: A flight information control method which is executed by a computer includes steps of: identifying a target region of an imaging target; generating flight information including a flight course for a flight imaging apparatus, which flies on the basis of flight information and performs shooting, to image the target region in order to shoot the target region in accordance with the identification of the target region of the imaging target; transmitting the generated flight information to the flight imaging apparatus; and transmitting a landing instruction to the flight imaging apparatus. The flight information includes location information where the flight imaging apparatus executes imaging processing.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a flight information control method, a control device, and a computer program. [Background technology]

[0002] There is disclosed a technology for a photographic method in which a camera is attached to a wirelessly controlled flying object and images are taken with the camera (see, for example, Patent Document 1). By attaching a camera to the flying object, it becomes possible to take photographs from the sky or from places where a tripod cannot be set up. Furthermore, attaching a camera to the flying object to take images brings various advantages, such as lower costs than using a real airplane or helicopter, safer imaging, the ability to take images at low altitudes or in narrow spaces, and the ability to get close to a target to take images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-27448 Summary of the Invention [Problem to be solved by the invention]

[0004] If an aircraft equipped with such a camera can be used to effectively capture images of places that are difficult for humans to enter, it is believed that this will be extremely useful for inspecting structures that are difficult for humans to approach, such as social infrastructure such as bridges, tunnels, dams, and roads over rivers and seas, and industrial infrastructure such as airports, buildings, warehouses, factories, and plants.

[0005] Therefore, the present disclosure proposes a new and improved flight information generation method, information processing device, and computer program that can efficiently capture images of a target area using an aircraft capable of performing image capture. [Means for solving the problem]

[0006] According to the present disclosure, there is provided a flight information control method executed by a computer, in which a flight imaging device identifies a target area of ​​an object to be imaged and flies and photographs based on the flight information, generates flight information including a flight path for imaging the target area for photographing the target area in accordance with the identification of the target area of ​​the object to be imaged, transmits the generated flight information to the flight imaging device, and transmits a takeoff instruction to the flight imaging device, and the flight information includes position information where the flight imaging device performs the imaging process.

[0007] Furthermore, according to the present disclosure, there is provided a control device having a target area identification unit that identifies a target area of ​​an object to be imaged, a flight imaging device that flies and photographs based on flight information, a generation unit that generates flight information including a flight path for imaging the target area for photographing the target area in accordance with the identification of the target area of ​​the object to be imaged, and a communication unit that is capable of sending and receiving information via wireless communication with the flight imaging device and transmits the generated flight information and takeoff instructions to the flight imaging device, wherein the flight information includes position information where the flight imaging device performs the imaging process.

[0008] Furthermore, according to the present disclosure, a computer program is provided that causes a flying imaging device to identify a target area of ​​an object to be imaged, fly based on flight information to perform imaging, generate flight information including a flight path for imaging the target area in accordance with the identification of the target area of ​​the object to be imaged, transmit the generated flight information to the flying imaging device, and execute a process of transmitting a takeoff instruction to the flying imaging device, wherein the flight information includes position information where the flying imaging device performs the imaging process. [Effects of the Invention]

[0009] As described above, the present disclosure makes it possible to provide a new and improved flight information generation method, control device, and computer program that can efficiently capture images of a target area using an aircraft capable of performing image capture.

[0010] The above effects are not necessarily limiting, and any of the effects described in this specification or other effects that can be understood from this specification may be achieved in addition to or instead of the above effects. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an embodiment of the present disclosure. [Figure 2] 1 is an explanatory diagram illustrating an example of a system configuration of an inspection system 10 according to an embodiment of the present disclosure. [Figure 3] 1 is an explanatory diagram illustrating an example of a functional configuration of a hovering camera 100 according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is an explanatory diagram illustrating an example of a functional configuration of a control terminal 200 according to an embodiment of the present disclosure. [Figure 5] 4 is a flowchart illustrating an example of the operation of the inspection system 10 according to an embodiment of the present disclosure. [Figure 6] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 7] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 8] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 9] 1 is an explanatory diagram conceptually showing how the hovering camera 100 captures an image of the bottom surface of a bridge 1. FIG. [Figure 10] 1 is an explanatory diagram conceptually showing the operation of a hovering camera 100 in an inspection system 10 according to an embodiment of the present disclosure. [Figure 11] 1 is an explanatory diagram conceptually showing the operation of a hovering camera 100 in an inspection system 10 according to an embodiment of the present disclosure. [Figure 12] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 13] FIG. 10 is an explanatory diagram showing an overview of inspecting the bottom surface of a bridge girder 3. [Figure 14] 1 is an explanatory diagram showing an example of an image 20 obtained by stitching still images captured by a hovering camera 100. FIG. [Figure 15] FIG. 2 is an explanatory diagram illustrating an example of a functional configuration of an information processing device 300 according to an embodiment of the present disclosure. [Figure 16] 10 is a flowchart illustrating an example of operation of the information processing device 300 according to an embodiment of the present disclosure. [Figure 17] 10 is a flowchart illustrating an example of the operation of the control terminal 200 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] The explanation will be given in the following order. 1. One embodiment of the present disclosure Overview 1.2. System configuration example 1.3. Functional configuration example 1.4. Example of operation 1.5. Example of damage data generation 1.5.1 Functional configuration example 1.5.2 Example of operation 2. Summary

[0014] 1. One embodiment of the present disclosure [1.1. Overview] Before describing an embodiment of the present disclosure in detail, an overview of the embodiment of the present disclosure will first be described.

[0015] Human inspection of the condition of structures such as roads, bridges, tunnels, buildings, etc. is essential for the maintenance and management of these structures. Normally, visual inspection of such structures involves workers approaching the structure and visually checking for damage such as corrosion or cracks, or loose bolts or other connecting members, and also conducting a tapping test to check for the presence of these abnormalities.

[0016] In order to maintain bridges, especially concrete bridges, workers must perform visual and hammering inspections of bridge girders and piers, and it has become necessary to set up scaffolding on the undersides of the piers and girders, for example, or to close some or all lanes to ensure the safety of the workers and to position work vehicles. These factors can cause problems not only in the cost of inspections, but also in the cost of arranging road guides due to road closures, and further in the traffic congestion that can occur on detour routes due to road closures.

[0017] Furthermore, there are bridges where it is difficult or impossible to set up scaffolding because they are built over rivers or the sea. In light of these circumstances, there is a need for technology that enables safe, low-cost inspection of structures without affecting traffic.

[0018] In light of the above circumstances, the present inventors have investigated technology that enables safe, low-cost inspection of structures without affecting traffic. As a result, as will be explained below, the present inventors have devised a technology that enables safe, low-cost inspection using an aircraft equipped with an imaging device (in the following explanation, an aircraft equipped with an imaging device will also be referred to as a "hovering camera").

[0019] Fig. 1 is an explanatory diagram illustrating an overview of an embodiment of the present disclosure. Fig. 1 schematically illustrates a bridge 1 constructed of, for example, concrete. In the past, as described above, when inspecting the bridge 1 constructed of concrete, workers had to set up scaffolding on the undersides of the piers 2 and bridge girders 3 to visually inspect the bridge for damage such as cracks and corrosion, and some or all lanes had to be closed to ensure the safety of the workers and to accommodate work vehicles.

[0020] In one embodiment of the present disclosure, a hovering camera 100 is used when inspecting a bridge 1. The hovering camera 100 is an aircraft equipped with an imaging device and configured to fly automatically according to preset flight information (which in this embodiment includes information on the flight path and the imaging position of the still image). Note that the information on the imaging position of the still image may include, for example, the position where the imaging process is performed, the imaging direction, the travel time to the position where the next imaging process is performed, etc.

[0021] For example, when inspecting the back side (bottom surface) of a bridge girder 3, the hovering camera 100 is automatically flown and the hovering camera 100 is made to capture an image of the back side of the bridge girder 3. By having the hovering camera 100 capture an image of the back side of the bridge girder 3, it is not necessary to erect scaffolding on the back side of the pier 2 or the bridge girder 3 for inspecting the bridge girder 3, and the frequency of lane closures can be reduced or road closures can be eliminated. Furthermore, when inspecting the side (side surface) of the bridge girder 3, for example, the hovering camera 100 is automatically flown and the hovering camera 100 is made to capture an image of the side of the bridge girder 3. Therefore, by automatically flying the hovering camera 100 and having the hovering camera 100 capture an image of the back side or side of the bridge girder 3, the safety of workers can be ensured, and the bridge 1 can be inspected at low cost without affecting traffic.

[0022] In order to automatically fly the hovering camera 100 and capture an image of the underside of the bridge girder 3, it is necessary to set the flight path of the hovering camera 100 and to set information on the capture position of a still image at the underside of the bridge girder 3. In one embodiment of the present disclosure, an object is to enable efficient inspection of the bridge 1 by efficiently creating flight information to be set in the hovering camera 100 using information on the general condition of the bridge 1.

[0023] An overview of an embodiment of the present disclosure has been described above. Next, a configuration example of an inspection system according to an embodiment of the present disclosure will be described.

[0024] [1.2. System configuration example] Fig. 2 is an explanatory diagram showing an example system configuration of an inspection system 10 according to an embodiment of the present disclosure. The inspection system 10 according to an embodiment of the present disclosure shown in Fig. 2 is a system intended to efficiently inspect a structure, for example, a bridge 1. Hereinafter, an example system configuration of the inspection system 10 according to an embodiment of the present disclosure will be described with reference to Fig. 2.

[0025] As shown in FIG. 2, the inspection system 10 according to one embodiment of the present disclosure includes a hovering camera 100, a control terminal 200, an information processing device 300, a wireless relay node 400, a position estimation node 500, a base station 600, a charging station 700, and a server device 800.

[0026] The hovering camera 100 is an example of an imaging device of the present disclosure, and is an air vehicle equipped with an imaging device as described above. The hovering camera 100 is an air vehicle configured to fly automatically based on a specified flight path and capture still images with the imaging device at a specified imaging position. The hovering camera 100 can fly using, for example, four rotors, and by controlling the rotation of each rotor, it can fly while ascending, descending, and moving horizontally. Of course, the number of rotors is not limited to the example.

[0027] The flight path from the flight start position to the flight end position and the imaging positions set in the hovering camera 100 are set as, for example, GPS (Global Positioning System) position information. Therefore, the hovering camera 100 may be equipped with a GPS receiver that receives radio waves from GPS satellites and calculates the current position. The flight path set in the hovering camera 100 may have all of the latitude, longitude, and altitude set as GPS position information, or only the latitude and longitude may be set as GPS position information, with the altitude set as, for example, a relative height from a base station 600 described below.

[0028] The control terminal 200 is an example of a control device of the present disclosure, and is a terminal that executes control related to the flight of the hovering camera 100. The control terminal 200 controls the flight of the hovering camera 100, for example, by generating flight information to be sent to the hovering camera 100, issuing takeoff instructions to the hovering camera 100, issuing return instructions to the base station 600 (described later), and maneuvering the hovering camera 100 when the hovering camera 100 is unable to fly automatically for some reason. The process of generating flight information for the hovering camera 100 by the control terminal 200 will be described in detail later, but here, a brief example will be described.

[0029] When generating flight information for the hovering camera 100, the control terminal 200 reads information about the general condition of the bridge 1 to be inspected, for example, a general condition map of the bridge 1 to be inspected, and displays it on the screen. Points on this general condition map of the bridge 1 are associated in advance with points on map data containing detailed GPS information. This association is preferably performed for at least two pairs of points. By associating points on map data containing detailed GPS information in advance with the general condition map of the bridge 1, the flight path of the hovering camera 100 is defined as GPS values. The control terminal 200 then generates the flight path of the hovering camera 100 based on this general condition map of the bridge 1. The flight path of the hovering camera 100 is displayed superimposed on the general condition map so that it is easy for the user (the worker inspecting the structure) to understand.

[0030] When generating flight information for the hovering camera 100, the control terminal 200 may take into consideration the structure and dimensions of the bridge 1 and the parts of the bridge 1 that are to be imaged by the hovering camera 100. When generating flight information for the hovering camera 100, the control terminal 200 may generate flight information that causes the hovering camera 100 to image in detail parts that are considered to have a high probability of being damaged.

[0031] As described above, the flight path set for the hovering camera 100 may have all of the latitude, longitude, and altitude set as GPS position information, but there may be cases where altitude data does not exist in the overview map of the bridge 1. If altitude data does not exist in the overview map of the bridge 1, the flight path set for the hovering camera 100 may have only the latitude and longitude set as GPS position information, and the altitude may be set as a relative height from the base station 600, for example.

[0032] When setting flight information to be set for the hovering camera 100, the control terminal 200 preferably generates flight information that keeps a constant distance from the surface of the object to be imaged when the hovering camera 100 images the bridge 1. By generating flight information that keeps a constant distance from the surface of the object to be imaged when the hovering camera 100 images the bridge 1, the control terminal 200 can cause the hovering camera 100 to generate images at the same scale.

[0033] The control terminal 200 is a portable device such as a notebook computer or tablet terminal, and transmits and receives information wirelessly to and from the hovering camera 100. The control terminal 200 may perform wireless communication with the hovering camera 100 directly, but since the hovering camera 100 may fly beyond the communication range of the control terminal 200 when inspecting a structure, particularly a bridge 1, the communication may be performed via a wireless relay node 400 that is installed during the inspection.

[0034] The control terminal 200 acquires images captured by an imaging device while the hovering camera 100 is flying, and displays them as necessary. The control terminal 200 may also acquire and display, by streaming, moving images captured by an imaging device while the hovering camera 100 is flying. By acquiring and displaying, by streaming, moving images captured by an imaging device while the hovering camera 100 is flying, the control terminal 200 can present to the user the position at which the hovering camera 100 is flying.

[0035] The information processing device 300 is a device that processes various types of information, and may be, for example, a device having a function of processing information, such as a personal computer (PC) or a game console. In this embodiment, the information processing device 300 is a device that has a function of displaying images captured by the hovering camera 100, in particular, to allow a user to check the condition of the bridge 1. The information processing device 300 also has a function of calculating the absolute damage position of the bridge girder 3 from the images captured by the hovering camera 100 and generating damage data, which will be described later. The information processing device 300 may also have a function of transmitting the generated damage data to the server device 800. Note that the control terminal 200 may have the function of calculating the absolute damage position of the bridge girder 3 from the images captured by the hovering camera 100 and generating the damage data, which will be described later.

[0036] The information processing device 300 acquires the images captured by the hovering camera 100, for example, from the control terminal 200. The timing at which the information processing device 300 acquires the images captured by the hovering camera 100 is not limited to a specific timing. For example, the information processing device 300 may acquire the images captured by the hovering camera 100 from the control terminal 200 at the timing at which one flight of the hovering camera 100 ends.

[0037] The wireless relay node 400 is a device that relays wireless communication between the hovering camera 100 and the control terminal 200. As described above, when inspecting a structure, particularly a bridge 1, the hovering camera 100 may fly beyond the communication range of the control terminal 200. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 can be performed via the wireless relay node 400 that is installed when inspecting the structure. The number of wireless relay nodes 400 is not limited to one, and multiple nodes may be installed depending on the inspection range of the bridge 1. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 can be performed via multiple wireless relay nodes 400. The hovering camera 100 can switch its communication destination between the control terminal 200 and the wireless relay node 400 depending on the radio wave conditions.

[0038] The wireless relay node 400 can be installed at an appropriate location on the bridge surface (preferably on the sidewalk) when inspecting the bridge 1. The wireless relay node 400 may also be installed so as to be suspended from the parapet of the bridge girder 3. Before inspecting the bridge 1, it is desirable to confirm that the wireless relay node 400 is operating normally using a predetermined method, for example, using the control terminal 200.

[0039] The position estimation node 500 is a device that causes the hovering camera 100 to estimate its current position. As described above, the flight path of the hovering camera 100 is set as, for example, GPS position information. In this case, if there is nothing blocking the radio waves from the GPS satellites, the hovering camera 100 can determine its current position with extremely high accuracy. However, it is inevitable that the hovering camera 100 will end up going under the bridge girder 3. If the bridge girder 3 blocks the radio waves from the GPS satellites or if multipath occurs due to radio wave reflection by the bridge 1, the hovering camera 100 may not be able to determine its current position with high accuracy.

[0040] Therefore, in this embodiment, a position estimation node 500 is provided under the bridge girder 3 to allow the hovering camera 100 to accurately acquire the current position. As the position estimation node 500, for example, an AR (Augmented Reality) marker or a GPS signal transmitter may be used.

[0041] When AR markers are used as the position estimation nodes 500, in order to have the hovering camera 100 recognize its current position, for example, the position estimation nodes 500 are hung from both ends of the bridge 1, and the hovering camera 100 is made to capture an image of the position estimation nodes 500. Then, the hovering camera 100 that has captured the image of the position estimation nodes 500 is made to fly between the specified position estimation nodes 500. The hovering camera 100 can grasp the position between the position estimation nodes 500, for example, by using an integrated value of a sensor (e.g., an IMU (Inertial Measurement Unit) sensor) provided in the hovering camera 100 and the distance to the destination position estimation node 500 calculated from the captured image. Therefore, by capturing an image of the position estimation nodes 500, the hovering camera 100 can accurately obtain its current position even under the bridge girder 3.

[0042] Furthermore, when a GPS signal transmitter is used as the position estimation node 500, in order to allow the hovering camera 100 to recognize its current position, the position estimation node 500 is installed, for example, at a diagonal or four corners of the bridge 1. By receiving the GPS signal emitted from the position estimation node 500, the hovering camera 100 can accurately obtain its current position even under the bridge girder 3.

[0043] The base station 600 is a device provided for takeoff and landing of the hovering camera 100. The base station 600 is equipped with a GPS receiver and calculates its current position by receiving radio waves from GPS satellites. The current position calculated by the base station 600 is sent to the control terminal 200. By sending the current position calculated by the base station 600 to the control terminal 200, the control terminal 200 becomes able to display the position of the base station 600 on the overview map of the bridge 1.

[0044] The base station 600 may have a function for checking the operation of the hover camera 100. The operation check of the hover camera 100 performed by the base station 600 may include, for example, checking the communication function, the imaging function, the flight function, and calibration of various sensors. Needless to say, the method for calibrating the sensors of the hover camera 100 is not limited to the method using the base station 600. For example, a method for calibrating the sensors of the hover camera 100 may involve fixing the hover camera 100 to a jig dedicated to calibration and rotating the hover camera 100 in the pitch direction and roll direction to calibrate the sensors.

[0045] The charging station 700 charges the secondary battery provided in the hovering camera 100. The hovering camera 100 is powered by a battery, and consumes power stored in the battery during flight and image capture. If the battery provided in the hovering camera 100 is a secondary battery, the charging station 700 can restore power consumed by the hovering camera 100 by charging the battery. The charging station 700 may charge the hovering camera 100 by connecting a cable or the like to the hovering camera 100 and supplying power to the hovering camera 100, or may charge the hovering camera 100 by supplying power to the hovering camera 100 via a contactless power transmission method.

[0046] The server device 800 is a device that stores various types of data. In this embodiment, the server device 800 may store the damage data generated by the information processing device 300.

[0047] 2, the inspection system 10 according to an embodiment of the present disclosure can have the hovering camera 100 capture an image of the bridge 1 to acquire an image of the bridge 1. By having the hovering camera 100 capture an image of the bridge 1, the inspection system 10 according to an embodiment of the present disclosure can eliminate the need to set up scaffolding on the bridge piers and girders, reduce the frequency of closing some or all lanes to traffic to ensure the safety of workers, or even eliminate the need for road closures, thereby enabling low-cost and efficient inspection of the bridge 1.

[0048] An example of the system configuration of the inspection system 10 according to an embodiment of the present disclosure has been described. Next, an example of the functional configuration of the hovering camera 100 and the control terminal 200 that constitute the inspection system 10 according to an embodiment of the present disclosure will be described.

[0049] [1.3. Functional configuration example] First, a functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure will be described. Fig. 3 is an explanatory diagram showing a functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure. Hereinafter, the functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure will be described with reference to Fig. 3.

[0050] As shown in FIG. 3, the hovering camera 100 according to one embodiment of the present disclosure includes an imaging device 101, rotors 104a to 104d, motors 108a to 108d, a control unit 110, a communication unit 120, a sensor unit 130, a position information acquisition unit 132, a memory unit 140, and a battery 150.

[0051] The control unit 110 controls the operation of the hovering camera 100. For example, the control unit 110 can control the adjustment of the rotation speed of the rotors 104a to 104d by adjusting the rotation speed of the motors 108a to 108d, the image capturing process by the image capturing device 101, the process of transmitting and receiving information to and from other devices (for example, the control terminal 200) via the communication unit 120, and the storage and reading of information into the memory unit 140.

[0052] In this embodiment, the control unit 110 controls flight by adjusting the rotation speeds of the motors 108a to 108d based on flight information transmitted from the control terminal 200, and controls the execution of still image capture processing for the imaging device 101. By controlling the motors 108a to 108d and the imaging device 101 based on the flight information transmitted from the control terminal 200, the control unit 110 becomes able to provide the control terminal 200 with an image based on a request from the control terminal 200.

[0053] The imaging device 101 is composed of a lens, an imaging element such as a CCD image sensor or a CMOS image sensor, a flash, etc. The imaging device 101 provided in the hovering camera 100 captures still images or moving images under the control of the control terminal 200. The images captured by the imaging device 101 are transmitted from the communication unit 120 to the control terminal 200. In this embodiment, the imaging device 101 performs imaging processing based on information about the imaging position of the still image included in the flight information transmitted from the control terminal 200. The images obtained by the imaging processing of the imaging device 101 may be stored in the storage unit 140 or transmitted from the communication unit 120 to the control terminal 200. When capturing an image of the underside of the bridge 1 with the hovering camera 100, sunlight may be blocked by the bridge 1, resulting in insufficient brightness. Therefore, it is preferable that the hovering camera 100 emits a flash when capturing an image of the underside of the bridge 1.

[0054] The imaging device 101 can change its imaging direction to any direction, for example, under control of the control unit 110. For example, when the horizontal direction of the hovering camera is set to 0 degrees, the imaging direction can be adjusted to a range of ±90 degrees vertically. By changing the imaging direction, the imaging device 101 can capture images in a predetermined direction and provide the captured images to the control terminal 200. The control unit 110 then associates the position information of the hovering camera 100 at the time the imaging device 101 captured a still image (this may include position information determined by GPS or positioning using the position estimation node 500; positioning using the position estimation node 500 will be described later), aircraft information at the time of image capture (e.g., yaw angle, pitch angle, acceleration, angular velocity), and imaging direction information as metadata for the still image. The associated metadata can be stored in an additional information field of the still image data (e.g., a specific field in the Exif format), or as separate data, such as by recording the metadata in a separate file from the image file.

[0055] The rotors 104a to 104d generate lift through rotation, causing the hovering camera 100 to fly. The rotation of the rotors 104a to 104d is achieved by the rotation of the motors 108a to 108d. The motors 108a to 108d rotate the rotors 104a to 104d. The rotation of the motors 108a to 108d can be controlled by the control unit 110.

[0056] The communication unit 120 performs processing for transmitting and receiving information via wireless communication with the control terminal 200. The hovering camera 100 transmits images captured by the imaging device 101 from the communication unit 120 to the control terminal 200. The hovering camera 100 also receives flight-related instructions from the control terminal 200 via the communication unit 120.

[0057] The sensor unit 130 is a group of devices that acquires the status of the hovering camera 100, and may be composed of, for example, an acceleration sensor, a gyro sensor, an ultrasonic sensor, a barometric pressure sensor, an optical flow sensor, a laser range finder, etc. The sensor unit 130 converts the acquired status of the hovering camera 100 into a predetermined signal and may provide the signal to the control unit 110 as needed. The position information acquisition unit 132 acquires information about the current position of the hovering camera 100 using, for example, a GPS or a vision sensor. The position information acquisition unit 132 may provide the acquired information about the current position of the hovering camera 100 to the control unit 110 as needed. The control unit 110 uses the information about the current position of the hovering camera 100 acquired by the position information acquisition unit 132 to perform flight control of the hovering camera 100 based on the flight information received from the control terminal 200.

[0058] The sensor unit 130 also detects obstacles that may interfere with flight. By detecting obstacles, the hovering camera 100 can provide information about the detected obstacles to the control terminal 200.

[0059] The storage unit 140 stores various information. Examples of information stored in the storage unit 140 include flight information of the hovering camera 100 transmitted from the control terminal 200, images captured by the imaging device 101, and the like.

[0060] The battery 150 stores power for operating the hovering camera 100. The battery 150 may be a primary battery that can only discharge, or a secondary battery that can also be charged. If the battery 150 is a secondary battery, the battery 150 may receive a supply of power from, for example, a charging station 700 shown in FIG. 2 .

[0061] The hovering camera 100 according to one embodiment of the present disclosure has a configuration as shown in FIG. 3, and can fly automatically based on the flight path included in the flight information transmitted from the control terminal 200, and can perform imaging processing based on information on the imaging position of the still image included in the flight information transmitted from the control terminal 200.

[0062] An example of the functional configuration of the hovering camera 100 according to an embodiment of the present disclosure has been described above with reference to Fig. 3. Next, an example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure will be described.

[0063] 4 is an explanatory diagram showing an example of a functional configuration of the control terminal 200 according to an embodiment of the present disclosure. Hereinafter, the example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure will be described with reference to FIG.

[0064] As shown in FIG. 4, the control terminal 200 according to an embodiment of the present disclosure includes a display unit 210, a communication unit 220, a control unit 230, and a storage unit 240.

[0065] The display unit 210 is formed of a flat panel display device such as a liquid crystal display device or an organic EL display device. The display unit 210 can display, for example, images captured by the imaging device 101 and information for controlling the operation of the hovering camera 100. The display unit 210 is provided with a touch panel, and a user can directly operate the information displayed on the display unit 210 by touching the display unit 210 with a finger or the like.

[0066] The communication unit 220 transmits and receives information to and from the hovering camera 100 via wireless communication. The control terminal 200 receives images captured by the imaging device 101 from the hovering camera 100 via the communication unit 220. The control terminal 200 also transmits instructions related to the flight of the hovering camera 100 from the communication unit 220 to the hovering camera 100. Commands related to the flight of the hovering camera 100 can be generated by the control unit 230.

[0067] The control unit 230 controls the operation of the control terminal 200. For example, the control unit 230 can control the display processing of characters, figures, images, and other information on the display unit 210, and the transmission and reception processing of information to and from other devices (e.g., the hovering camera 100) via the communication unit 220. The control unit 230 is also configured to include a flight information generation unit 232 and a display control unit 234.

[0068] The flight information generating unit 232 generates flight information to be transmitted to the hovering camera 100. When generating the flight information, the flight information generating unit 232 uses, for example, information about the structure to be inspected that is stored in the storage unit 240 described below. After generating the flight information, the flight information generating unit 232 causes the communication unit 220 to transmit the generated flight information before the hovering camera 100 takes off.

[0069] The process of generating flight information by the flight information generating unit 232 will be described in detail later, but an example of the process of generating flight information by the flight information generating unit 232 will be briefly described below. When generating flight information for the hovering camera 100, the flight information generating unit 232 reads a general map of the bridge 1 to be inspected. The read general map of the bridge 1 is displayed on the display unit 210 by the display control unit 234. As described above, points on this general map of the bridge 1 are previously associated with points on map data containing detailed GPS information. It is desirable that this association be performed for at least two pairs of points. By previously associating points on map data containing detailed GPS information with the general map of the bridge 1, the flight path of the hovering camera 100 is defined as GPS values ​​(a set of latitude and longitude).

[0070] The flight information generation unit 232 then generates a flight path for the hovering camera 100 based on this overview map of the bridge 1. When generating the flight path for the hovering camera 100, the flight information generation unit 232 uses information about the bridge 1's construction method, width, span length, and other structural information, the available flight time for the hovering camera 100, and information about the bridge 1 inspection method. Concrete bridges are classified into reinforced concrete (RC) and prestressed concrete (PC) bridges depending on the reinforcement method, and are classified into, for example, RCT girder bridges, PCT girder bridges, PC hollow slab bridges, RC box girder bridges, and PC box girder bridges depending on the shape of the girders. Therefore, if the construction method of the bridge 1 to be inspected is known, the flight information generation unit 232 can generate a flight path appropriate for the construction method of the bridge 1. The flight information generation unit 232 then displays the flight path of the hovering camera 100 superimposed on the overview map of the bridge 1.

[0071] As described above, the flight information generating unit 232 defines the flight path of the hovering camera 100 as GPS values ​​(a set of latitude and longitude). By the flight information generating unit 232 defining the flight path of the hovering camera 100 as GPS values, the hovering camera 100 can determine, based on the GPS values, at what position during flight it should perform imaging processing.

[0072] The display control unit 234 controls the display of characters, figures, images, and other information on the display unit 210. The display of characters, figures, symbols, images, and other information displayed on the display unit 210 in the drawings referred to in the following description is controlled by the display control unit 234. For example, when the flight information generation unit 232 generates flight information to be transmitted to the hovering camera 100, the display control unit 234 controls the display of an overview diagram of the structure (bridge 1) to be inspected and the generated flight information on the display unit 210.

[0073] The storage unit 240 stores various types of information. For example, information stored in the storage unit 240 includes information about the structure (bridge 1) to be inspected. The information about the structure to be inspected may include, for example, an overview map of the structure to be inspected (bridge 1) and the construction method of the structure to be inspected. Furthermore, if locations of the structure to be inspected that are thought to be prone to damage are known in advance, the information about the structure to be inspected may include information about parts that are thought to be highly likely to be damaged.

[0074] The control terminal 200 may receive information about the structure (bridge 1) to be inspected from, for example, the information processing device 300 when inspecting the structure, without having to store the information in advance in the storage unit 240.

[0075] The control terminal 200 according to one embodiment of the present disclosure has a configuration as shown in FIG. 4, and generates flight information to be transmitted to the hovering camera 100 based on information about the structure (bridge 1) to be inspected, and can acquire images captured based on the flight information by the hovering camera 100, which flies based on the flight information.

[0076] An example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure has been described above using Fig. 4. Next, an example of the operation of the inspection system 10 according to an embodiment of the present disclosure will be described.

[0077] [1.4. Example of operation] Fig. 5 is a flow chart showing an example of operation of the inspection system 10 according to an embodiment of the present disclosure. Fig. 5 shows an example of operation of the inspection system 10 according to an embodiment of the present disclosure when inspecting the bridge 1 by flying the hovering camera 100 and having the hovering camera 100 capture an image of the bridge 1. Note that when inspecting the bridge 1 using the hovering camera 100, it is assumed that the wireless relay node 400 and the position estimation node 500 have been installed in advance at appropriate positions on the bridge 1. Hereinafter, an example of operation of the inspection system 10 according to an embodiment of the present disclosure will be described using Fig. 5.

[0078] The control terminal 200, which generates flight information for the hovering camera 100, reads information about the bridge 1, including a schematic diagram of the bridge 1 to be inspected, and displays the schematic diagram of the bridge 1 on the display unit 210 (step S101). The reading of the information about the bridge 1 is performed, for example, by the flight information generation unit 232, and the display of the schematic diagram of the bridge 1 on the display unit 210 is performed, for example, by the display control unit 234. The control terminal 200, which is displaying the schematic diagram of the bridge 1 on the display unit 210, prompts the user to specify an area of ​​the bridge 1 to be inspected using the schematic diagram of the bridge 1 displayed on the display unit 210 (step S102). The process of prompting the user to specify in step S102 is performed, for example, by the flight information generation unit 232.

[0079] For example, when a portion of the bridge 1 is to be inspected, the control terminal 200 allows the user to specify the area to be inspected in the overview map of the bridge 1 displayed on the display unit 210. Also, when the entire bridge 1 is to be inspected, the control terminal 200 allows the user to specify the area of ​​all parts of the bridge 1 in the overview map of the bridge 1 displayed on the display unit 210.

[0080] FIG. 6 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 6 shows an example of a screen displayed on the display unit 210 when the user is prompted to specify the area of ​​the bridge 1 to be inspected in step S102 above. FIG. 6 shows a screen that is displayed on the display unit 210 when a bridge girder is specified as the area of ​​the bridge 1 to be inspected. The control terminal 200 has, for example, a touch panel as an input unit (not shown), and can allow the user to specify the area of ​​the bridge 1 by, for example, having the user trace the screen or select a span to be inspected. Of course, the method of allowing the user to specify the area of ​​the bridge 1 to be inspected is not limited to the example shown in FIG. 6. Furthermore, the display of the area specified by the user is not limited to the example shown in FIG. 6.

[0081] 6 also shows a state in which a mark B1 indicating the position of the base station 600 is superimposed on the general view of the bridge 1. As described above, the base station 600 is equipped with a GPS receiver and can calculate its current position by receiving radio waves from GPS satellites. Therefore, the control terminal 200 can display a mark B1 indicating the position of the base station 600 superimposed on the general view of the bridge 1 based on the information on the current position calculated by the base station 600.

[0082] After the user specifies the area of ​​the bridge 1 to be inspected, the control terminal 200 then generates flight information for the hovering camera 100 in the area to be inspected specified by the user, based on information about the bridge 1 (step S103). The flight information generation process in step S103 is executed by, for example, the flight information generation unit 232.

[0083] When generating flight information for the hovering camera 100 in step S103, the control terminal 200 uses information about the bridge 1's structure, such as its construction method, width, and span length, the time the hovering camera 100 can fly, and information about the inspection method for the bridge 1. For example, if a T-girder is used as the construction method for the bridge 1, the control terminal 200 generates, as flight information, a flight path in which the hovering camera 100 repeatedly rises and falls on the underside of the bridge 1. Furthermore, when generating flight information for the hovering camera 100 in step S103, the control terminal 200 may use information about the surface of the bridge 1 to be imaged. For example, if the user selects to image the side of the bridge 1, the control terminal 200 generates, as flight information, a flight path that follows the side of the bridge 1, and if the user selects to image the underside of the bridge 1, the control terminal 200 generates, as flight information, a flight path that makes a round trip along the underside of the bridge 1.

[0084] An example of flight information generated by the control terminal 200 will be described below. The flight information may specify, for example, a list for each position where image capture processing is to be performed in the following format. ID: (relative coordinates of the imaging point, imaging direction, imaging speed, travel time to the next imaging point, etc.) The relative coordinates of the imaging point are specified by three points on the X, Y, and Z axes. The X axis is the latitude direction, the Y axis is the longitude direction, and the Z axis is the height direction. Other information may include, for example, information for controlling special imaging. Information for controlling special imaging may include, for example, information for capturing images from multiple imaging directions at the same position, information on parameters for bracket imaging (capturing images at the same position and imaging direction with different exposures, shutter speeds, ISO sensitivity, etc.), and information on the infrared wavelength used for imaging. According to this format, the flight information generated by the control terminal 200 may be composed of a list of values ​​such as the following: 0:(0,0,0,0,0,2,1.0) 1:(5,0,0,0,0,2,1.0) 2:(7,0,0,0,0,2,1.0) 3:(9,0,0,0,0,2,1.0) The imaging points included in the flight information generated by the control terminal 200 may be specified by relative coordinates from a reference point, such as the absolute coordinates of the base station 600 or the absolute coordinates of an arbitrary position, such as the position where the first imaging was performed. The hovering camera 100 may convert the relative coordinates from the absolute coordinates of the reference point into absolute coordinates and refer to the converted coordinates during flight. Furthermore, the imaging points included in the flight information generated by the control terminal 200 may be specified by absolute coordinates instead of relative coordinates. Furthermore, information controlling special imaging included in the flight information generated by the control terminal 200 may store predetermined values. For example, the information controlling special imaging may store values ​​such as 1: imaging in multiple imaging directions, 2: bracket imaging (changing the shutter speed), 3: bracket imaging (changing the ISO sensitivity), etc. The control terminal 200 may include information controlling special imaging in the flight information for locations of the bridge girder 3 that are considered to be prone to damage, which are stored in the storage unit 240, for example.

[0085] During the process of generating flight information in step S103, the control terminal 200 may generate flight information that causes the hovering camera 100 to capture images of the underside of the bridge girder 3 of the bridge 1 at equal intervals. Therefore, during the process of generating flight information in step S103, the control terminal 200 may generate flight information such that the capturing positions of still images are at equal intervals.

[0086] When generating flight information for the hovering camera 100 in step S103 above, if information about the portion considered to be highly likely to be damaged is stored in advance in the storage unit 140, the control terminal 200 may read out the stored information and generate flight information that causes the hovering camera 100 to capture detailed images of that portion. When causing the hovering camera 100 to capture images of the portion considered to be highly likely to be damaged, the control terminal 200 may include information for controlling the above-mentioned special imaging in the flight information. Of course, the information about the portion considered to be highly likely to be damaged does not have to be stored in advance in the storage unit 140, in which case the user may input information about the portion considered to be highly likely to be damaged during inspection.

[0087] When flying the hover camera 100 over an area of ​​the bridge 1 to be inspected, it may be impossible to fly the hover camera 100 over that area in one go, depending on the flight time of the hover camera 100. The flight time of the hover camera 100 can be calculated in advance from the capacity of the battery 150, the power consumption of the motors 108a-108d for moving the rotors 104a-104d, the power consumption of the image capture device 101, the control unit 110, and the communication unit 120, etc. When generating flight information, it is also possible to estimate the time required for one inspection flight of the hover camera 100 from the planned travel time from the start position (e.g., base station 600) to the first image capture point, the planned travel time between image capture points, the planned travel time from the last image capture point to return to the start position, etc. Therefore, if the flight path for the area of ​​the bridge 1 to be inspected cannot be flown by the hover camera 100 in one inspection flight, the control terminal 200 may divide the generated flight path into several parts.

[0088] Furthermore, when generating flight information for the hovering camera 100 in step S103, the control terminal 200 may generate multiple flight paths and display the flight paths on the display unit 210. FIG. 7 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 7 shows an example of a state in which multiple flight paths are generated when flight information for the hovering camera 100 is generated in step S103, and the flight paths R1 and R2 are displayed on the display unit 210. The control terminal 200 displays the multiple flight paths on the display unit 210 and allows the user to select one flight path. The control terminal 200 generates flight information based on the user's selection of the flight path.

[0089] After generating the flight information of the hovering camera 100 in step S103, the control terminal 200 subsequently transmits the generated flight information to the hovering camera 100 and transmits a takeoff instruction to the hovering camera 100 (step S104). The transmission of the generated flight information and the takeoff instruction is performed by the flight information generation unit 232 via the communication unit 220, for example.

[0090] FIG. 8 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 8 shows an example of a screen displayed on the display unit 210 of the control terminal 200 when a takeoff instruction is transmitted to the hovering camera 100. The user can transmit a takeoff instruction from the control terminal 200 to the hovering camera 100 by touching a takeoff instruction button 211 displayed on the display unit 210. When transmitting a takeoff instruction from the control terminal 200 to the hovering camera 100, the flight information generated in step S103 above may be transmitted from the control terminal 200 to the hovering camera 100 before the takeoff instruction. Alternatively, the flight information generated in step S103 above may be transmitted from the control terminal 200 to the hovering camera 100 after the takeoff instruction is transmitted from the control terminal 200 to the hovering camera 100.

[0091] The hovering camera 100 receives flight information and a takeoff command from the control terminal 200 and takes off from the base station 600. The hovering camera 100 flies based on the flight information sent from the control terminal 200 and executes an imaging process to obtain a still image (step S105). The hovering camera 100 acquires position information and aircraft information at the time of the imaging process when the imaging process to obtain the still image is executed, and associates the information with the still image. The aircraft information at the time of the imaging process may include, for example, yaw angle, pitch angle, acceleration, and angular velocity. The hovering camera 100 may also stream video captured by the imaging device 101 during flight to the control terminal 200. The control terminal 200 streams and displays the video captured by the imaging device during flight of the hovering camera 100, allowing the control terminal 200 to present to the user the location where the hovering camera 100 is flying.

[0092] When performing imaging processing, it is desirable for the hovering camera 100 to maintain a constant distance from the surface to be imaged (for example, the side or bottom surface of the bridge girder 3) at all imaging points. By maintaining a constant distance from the surface to be imaged at all imaging points, the hovering camera 100 can obtain still images captured at the same size.

[0093] If a part that is considered to have a high possibility of being damaged is included in the flight path of hover camera 100, hover camera 100 may capture multiple still images of that part by changing the imaging direction of the imaging device, applying infrared light of different wavelengths, or changing the shutter speed. Also, if a part that is considered to have a high possibility of being damaged is included in the flight path of hover camera 100, hover camera 100 may narrow the intervals between positions where imaging processing is performed for that part compared to other parts.

[0094] 9 is an explanatory diagram conceptually illustrating the operation of the hovering camera 100 in the inspection system 10 according to an embodiment of the present disclosure. When the hovering camera 100 flies over the underside of the bridge 1 based on flight information, the hovering camera 100 stops at time t1, captures an image of the underside of the bridge 1, flies to a position where the image is to be captured at time t2, stops at time t2, captures an image of the underside of the bridge 1 at a different position, and thereafter repeats the flight, stop, and image capture for capturing the underside of the bridge 1 until time t n The hovering camera 100 repeatedly flies, stops, and captures images, thereby obtaining an image of the bottom surface of the bridge 1.

[0095] When flying based on flight information, the hovering camera 100 can accurately determine its current position if it can receive radio waves from GPS satellites without interference. However, in locations where it is difficult to receive radio waves from GPS satellites, such as under a bridge 1, it becomes difficult for the hovering camera 100 to accurately determine its current position. Therefore, in this embodiment, by using a position estimation node 500, the hovering camera 100 can accurately determine its current position even in locations where it is difficult to receive radio waves from GPS satellites.

[0096] Fig. 10 is an explanatory diagram conceptually illustrating the operation of the hovering camera 100 in the inspection system 10 according to an embodiment of the present disclosure. For example, when a route is set such that the hovering camera 100 flies from Start to Goal in Fig. 10, the hovering camera 100 moves back and forth between a GPS positioning area 40 where it receives radio waves from GPS satellites 30 without interference to determine its position, and a sensor positioning area 50 where it estimates its current position using, for example, a vision sensor.

[0097] In a GPS positioning area 40, the hovering camera 100 determines its current position using radio waves from GPS satellites 30. In a sensor positioning area 50, the hovering camera 100 determines its position between position estimation nodes 500, and if the position estimation node 500 is an AR marker, it determines its current position using an integrated value of a sensor (e.g., an IMU sensor) provided in the hovering camera 100 and the distance to the destination position estimation node 500 calculated from an image captured by the imaging device 101. If the position estimation node 500 is a GPS signal transmitter, the hovering camera 100 determines its current position using a signal transmitted from the position estimation node 500.

[0098] By using the position estimation node 500 in this way, the hovering camera 100 can accurately determine its current position even when it moves to a location where it is difficult to receive radio waves from a GPS satellite.

[0099] When the hovering camera 100 completes the imaging process at the last imaging point, it automatically flies to the base station 600 and returns to the base station 600 (step S106). Then, the control terminal 200 acquires from the hovering camera 100 the images captured by the hovering camera 100 that has returned to the base station 600 (step S107). Note that the images captured by the hovering camera 100 may be acquired after the hovering camera 100 has returned to the base station 600 in this manner, but the control terminal 200 may also acquire still images one by one each time the hovering camera 100 executes the imaging process and obtains the still image.

[0100] In an inspection system 10 according to one embodiment of the present disclosure, the hovering camera 100 and the control terminal 200 perform the operations shown in FIG. 5, so that the control terminal 200 generates flight information to be sent to the hovering camera 100 based on information about the structure (bridge 1) to be inspected, the hovering camera 100 flies based on the flight information, captures images based on the flight information, and the control terminal 200 acquires the images captured by the hovering camera 100.

[0101] It should be noted that, while the hovering camera 100 is flying, the user may find a part that he or she wishes to capture in detail while viewing the video captured by the hovering camera 100. In this case, for example, the user may operate the control terminal 200 to send an instruction to the hovering camera 100 to stop automatic flight and switch to manual operation.

[0102] In the above example, flight information is generated by the control terminal 200, and the hovering camera 100 automatically flies based on the generated flight information and performs image capture processing. However, it is possible that an obstacle that cannot be seen in the overview map of the bridge 1 exists on the flight path.

[0103] Fig. 11 is an explanatory diagram conceptually illustrating the operation of the hovering camera 100 in the inspection system 10 according to an embodiment of the present disclosure. Fig. 11 shows a tree 4 growing under a bridge girder 3. This tree 4 is an obstacle that does not appear on the overview diagram of the bridge 1, and its presence may become apparent only when the hovering camera 100 is flying.

[0104] Therefore, in this embodiment, the hovering camera 100 may be test-flew once based on the flight information generated by the control terminal 200 to check whether there are any obstacles on the flight path included in the flight information.

[0105] When the hover camera 100 is test-flew once based on the flight information generated by the control terminal 200, the control terminal 200 may receive streaming video captured by the hover camera 100, and the user may check whether there are any obstacles on the flight path included in the flight information while viewing the video. Alternatively, the sensor unit 130 of the hover camera 100 may detect the obstacles. The precise location of the obstacle can be determined by providing a stereo camera as the imaging device 101 of the hover camera 100, determining the distance to the obstacle through imaging by the stereo camera, or identifying the direction of the obstacle based on the orientation of the hover camera 100. Note that if an obstacle is found on the flight path during the test flight of the hover camera 100, the hover camera 100 may stop automatic flight and enter a hovering state, wait for a user operation, or automatically return to the base station 600.

[0106] If it is determined that an obstacle exists on the flight path included in the flight information, the control terminal 200 may register the location of the obstacle in the overview map of the bridge 1. The location of the obstacle may be manually input by the user, or, if the hovering camera 100 detects an obstacle using the sensor unit 130, the location of the detected obstacle may be acquired from the hovering camera 100 and the location of the obstacle may be registered in the overview map of the bridge 1.

[0107] Fig. 12 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. Fig. 12 shows an example of a screen displayed on the display unit 210 when the test flight of the hovering camera 100 reveals that an obstacle exists on the flight path. If the test flight of the hovering camera 100 reveals that an obstacle exists on the flight path, the control terminal 200 displays a mark O1 indicating the location of the obstacle superimposed on the overview map of the bridge 1.

[0108] Once the location of the obstacle is known, the control terminal 200 regenerates flight information so that the flight path avoids the location of the obstacle, and transmits the regenerated flight information to the hovering camera 100. By flying based on the flight information regenerated by the control terminal 200, the hovering camera 100 can fly and capture images while avoiding the obstacle (tree 4).

[0109] The method of determining the location of an obstacle by flying the hovering camera 100 is not limited to the above example. For example, the hovering camera 100 may fly along a simple route around the periphery of the flight path generated by the control terminal 200 while capturing moving images with the imaging device 101, thereby checking whether there is an obstacle under the bridge girder 3.

[0110] [1.5. Example of damage data generation] By flying the hovering camera 100 and having it capture images of the bridge 1, it becomes possible to grasp the state of places that workers cannot easily approach, such as the bottom of the bridge girder 3. A still image captured by the hovering camera 100 is associated with, for example, the position information of the hovering camera 100 that captured the still image (which may include position information obtained by GPS positioning or positioning using the position estimation node 500), aircraft information at the time of capturing the image (e.g., yaw angle, pitch angle, acceleration, angular velocity), and information on the capturing direction. Furthermore, by capturing images while maintaining a constant distance from the target surface at all capturing points, the relative position of the location where damage has occurred in the image can be determined. Therefore, if a damaged portion of the bridge girder 3 is included in a still image captured by the hovering camera 100, the absolute location of the damaged portion can be determined. For example, the position information of the damaged part can be obtained by calculating the relative value of the damaged location with the center of the still image as the origin, and then calculating the relative value to the position information of the hovering camera 100 when the image was captured. Note that the following data, for example, can be recorded as the position information of the damaged part. (1) Record the information on the capture position of the still image as the position of the damaged part (without recording a relative value (offset)). (2) Information on the imaging position of the still image and a relative value (offset) corresponding to the damaged part are recorded as the position of the damaged part. (3) The reference absolute value (for example, as described below, the capture position of a still image of the four corners where the accuracy of the position information is considered to be high, or the coordinates of the position estimation node 500) and the relative value (offset) are recorded as the position of the damaged part. (4) The calculated absolute values ​​(e.g., latitude, longitude, altitude) are recorded as the location of the damaged part.

[0111] Techniques are known for determining the physical size of the imaging range by using numerical values ​​such as the focal length of the lens, the size of the image sensor, and the distance to the imaging target. Therefore, when identifying a damaged portion, the physical size of the imaging range of the hovering camera 100 may be estimated using information about the distance from the hovering camera 100 to the imaging target (e.g., the back or side of the bridge girder 3) and information about the angle of view of the imaging device 101. The center position of the captured image (the position where the hovering camera 100 captured the image) is set as the origin, and the physical relative position from this origin to the damaged portion is estimated. The position coordinates of the origin of the captured image are added to this relative position to determine the physical position information of the damaged portion. Note that the distance information and angle of view information may be acquired via a sensor possessed by the hovering camera 100 when capturing an image and recorded in association with the image, or values ​​previously set in the hovering camera 100 or the imaging device 101 may be used. Furthermore, in addition to the imaging position information, distance information, and angle of view information, the position information of the damaged portion may be calculated using aircraft information at the time of imaging (for example, yaw angle, pitch angle, acceleration, angular velocity) and imaging direction information.

[0112] The detection of damaged parts from the still images captured by the hovering camera 100 may be performed visually by the user, or may be performed automatically by image processing, for example, by the information processing device 300. When the detection of damaged parts is automated, an image processing technique such as pattern matching processing may be used.

[0113] The data structure of the damage data is defined, for example, in the following format: (Image ID, damage ID, location information of damaged part, coordinates of damaged part on image, damage type ID, damage level) The damage type ID refers to an ID assigned to each type of damage, such as cracks, peeling, water leakage, and free lime. The damage level field can record the maximum width of the damage, the length of the damaged portion in the image, and the like. The inspection system 10 according to this embodiment can generate damage data in the above-described format from still images captured by the hovering camera 100, either manually input by the user or automatically processed by the information processing device 300. The damage data generated by the inspection system 10 according to this embodiment can then be used to place an order with a contractor to repair the damage to the bridge 1.

[0114] However, the hovering camera 100 captures a large number of still images during one inspection flight, and therefore, it is a heavy burden on the user to check each and every still image captured by the hovering camera 100 during an inspection flight.

[0115] Therefore, the still images captured by the hovering camera 100 are joined (stitched) together to obtain a single image. By stitching the still images captured by the hovering camera 100, for example, the state of the bottom surface of one span of the bridge girder 3 can be obtained as a single image. Then, by checking the image of the bottom surface of one span of the bridge girder 3 obtained by stitching the still images captured by the hovering camera 100, the user can confirm whether there is any damage to the bottom surface of the bridge girder 3. The still image stitching process may be performed by the control terminal 200 or the information processing device 300.

[0116] 13 is an explanatory diagram showing an overview of inspecting the bottom surface of a bridge girder 3 based on still images captured by a hovering camera 100. A portion of the bottom surface of the bridge girder 3 (for example, the length of one span of the bridge girder 3) is captured by the hovering camera 100, and the still images captured by the hovering camera 100 are stitched together to obtain a single image 20 of the bottom surface of the bridge girder 3. Note that the reference numeral 21 indicates an image captured by the hovering camera 100 in one imaging process.

[0117] When determining the absolute location of the damaged portion from an image obtained by stitching still images captured by the hovering camera 100, position information from the stitched images with relatively high accuracy at the time of capture can be selected as a reference point. Position information from the hovering camera 100 at the time of capture of the still images of the four corners that form the basis of the stitched image may also be used as the reference point. The still images of the four corners that form the basis of the stitched image have the least distortion, and GPS positioning areas have less error in position information, making them GPS positioning areas. It is considered preferable to use position information from the capture of the four corners that are close to the GPS positioning area as the reference point. Therefore, by determining the absolute location of the damaged portion from the position information of the hovering camera 100 corresponding to the still images of the four corners, the location of the damaged portion can be determined with high accuracy. Note that, for example, positioning status information in the GPS positioning data (information indicating whether 2D positioning is in progress, 3D positioning is in progress, or positioning is unavailable, and data such as the number of received satellites) may be used to indicate the accuracy of each position information.

[0118] 14 is an explanatory diagram showing an example of image 20 obtained by stitching still images captured by hovering camera 100. Centers G1 to G4 of still images C1 to C4 at the four corners that form the basis of image 20 correspond to the position of hovering camera 100 when each still image was captured. In this embodiment, the absolute position of each damaged portion in image 20 is calculated using position information of hovering camera 100 that corresponds to these still images C1 to C4 at the four corners.

[0119] When generating damage data from stitched images, the data structure of the damage data is defined, for example, in the following format: That is, the image ID is deleted from the above-mentioned damage data. (Damage ID, location information of the damaged part, coordinates of the damaged part on the image, damage type ID, damage level) Alternatively, an image ID of the stitched image may be generated and included in the damage data. The inspection system 10 according to this embodiment can generate damage data in the above-described format from the stitched images, either manually input by the user or automatically processed by the information processing device 300.

[0120] [1.5.1. Functional configuration example] Fig. 15 is an explanatory diagram showing an example functional configuration of an information processing device 300 according to an embodiment of the present disclosure. Fig. 15 shows an example functional configuration of an information processing device 300 according to an embodiment of the present disclosure, which has a function of determining the absolute position of damage to a bridge girder 3 from a still image captured by a hovering camera 100 and generating damage data. Hereinafter, the example functional configuration of the information processing device 300 according to an embodiment of the present disclosure will be described with reference to Fig. 15.

[0121] As shown in FIG. 15, an information processing device 300 according to an embodiment of the present disclosure includes a display unit 310, a communication unit 320, a control unit 330, and a storage unit 340.

[0122] The display unit 310 is composed of a flat panel display device such as a liquid crystal display device, an organic EL display device, etc. The display unit 310 can display, for example, images captured by the imaging device 101 of the hovering camera 100, information about damage to the bridge 1 obtained from the images captured by the imaging device 101, etc.

[0123] The communication unit 320 transmits and receives information via wireless communication with, for example, the control terminal 200. The information processing device 300 receives, via the communication unit 320, an image captured by the hovering camera 100 from the control terminal 200 together with information on the absolute imaging position of the image.

[0124] The control unit 330 controls the operation of the information processing device 300. For example, the control unit 330 can control the display processing of characters, figures, images, and other information on the display unit 310, and the transmission and reception processing of information to and from other devices (e.g., the control terminal 200) via the communication unit 320. The control unit 330 is also configured to include an imaging position information acquisition unit 332, a damage position calculation unit 334, an image synthesis unit 336, and a damage data generation unit 338.

[0125] The imaging position information acquisition unit 332 acquires information about the imaging position at the time of imaging, which is acquired by the hovering camera 100 when the hovering camera 100 images the bridge 1. The damage position calculation unit 334 detects damaged parts of the bridge 1 from the image captured by the hovering camera 100 using an image processing technique such as pattern matching processing, and calculates the absolute position of the damaged parts using the imaging position information acquired by the imaging position information acquisition unit 332.

[0126] The image synthesis unit 336 executes image stitching processing to generate a single image by joining together still images captured by the hovering camera 100. When stitching the still images captured by the hovering camera 100, the image synthesis unit 336 may use information on the imaging positions at which each still image was captured.

[0127] When calculating the damage position, the damage position calculation unit 334 may use information on the imaging positions of corner captured images (for example, each of the four corners) in the captured images that form the basis of the image stitched by the image synthesis unit 336. As described above, it is considered that the still images at the four corners in the captured images that form the basis of the stitched image have the least distortion, so the damage position calculation unit 334 can determine the damage position more accurately by using information on the imaging positions of the corner captured images in the captured images that form the basis of the stitched image.

[0128] The damage data generation unit 338 generates the above-mentioned damage data using the absolute position of the damaged portion of the bridge 1 calculated by the damage position calculation unit 334. The damage data generation unit 338 may generate damage data for each still image, or may generate one piece of damage data for the images stitched by the image synthesis unit 336.

[0129] The storage unit 340 stores various types of information. The information stored in the storage unit 340 may include, for example, information on a still image captured by the imaging device 101 of the hovering camera 100, information on the absolute imaging position of the hovering camera 100 when the still image was captured, and information on damage data generated by the damage data generation unit 338.

[0130] The information processing device 300 according to an embodiment of the present disclosure has the configuration shown in FIG. 15 , which enables it to generate damage data from still images captured by the hovering camera 100. Therefore, the information processing device 300 according to an embodiment of the present disclosure can efficiently generate inspection results for the bridge 1, which is a structure to be inspected. As described above, the damage data may be generated by the control terminal 200 rather than the information processing device 300. Therefore, the configuration of the control unit 330 of the information processing device 300, which performs the operations shown in FIG. 15, may be included in the control terminal 200. Furthermore, the inspection results for the bridge 1, which is a structure to be inspected, can be accumulated in and utilized in a public or private database. As described above, the damage data may be generated by the control terminal 200 rather than the information processing device 300. Therefore, the configuration of the control unit 330 of the information processing device 300, which performs the operations shown in FIG. 15, may be included in the control terminal 200.

[0131] An example of the functional configuration of the information processing device 300 according to an embodiment of the present disclosure has been described above with reference to Fig. 15. Next, an example of the operation of the information processing device 300 according to an embodiment of the present disclosure will be described.

[0132] [1.5.2. Example of operation] Fig. 16 is a flow chart showing an example operation of the information processing device 300 according to an embodiment of the present disclosure. Fig. 16 shows an example operation of the information processing device 300 according to an embodiment of the present disclosure when determining the absolute position of damage to the bridge girder 3 from a still image captured by the hovering camera 100 and generating damage data. Hereinafter, the example operation of the information processing device 300 according to an embodiment of the present disclosure will be described with reference to Fig. 16.

[0133] The information processing device 300 first acquires an image associated with information about the image capturing position, the image being captured by the hovering camera 100 while flying around the periphery of the bridge 1 (step S301). After acquiring the image associated with the information about the image capturing position in step S301, the information processing device 300 then detects damaged portions from the image using an image processing technique such as pattern matching processing (step S302). The process of detecting damaged portions in step S302 can be executed by the damage position calculation unit 334, for example.

[0134] After detecting a damaged portion from the image in step S302, the information processing device 300 subsequently calculates the absolute position of the damaged portion (step S303). The calculation process in step S303 may be performed by, for example, the damage position calculation unit 334. The information processing device 300 calculates the absolute position of the damaged portion in step S303 based on information about the imaging position of the still image captured by the hovering camera 100. When calculating the absolute position of the damaged portion, the information processing device 300 may estimate the physical size of the imaging range of the hovering camera 100 based on information about the distance from the hovering camera 100 to the imaging target (e.g., the back or side of the bridge girder 3) and information about the angle of view of the imaging device 101. The information processing device 300 can determine the physical position information of the damaged portion by estimating the physical relative position from the center of the captured image to the damaged portion and adding the position coordinates of the origin of the captured image to this relative position. Then, the information processing device 300 generates damage data including the absolute position of the damaged portion (step S304). The damage data generation process of step S304 can be executed by the damage data generation unit 338, for example.

[0135] By performing the operation shown in Fig. 16 , the information processing device 300 according to an embodiment of the present disclosure can generate damage data from a still image captured by the hovering camera 100, and therefore the information processing device 300 according to an embodiment of the present disclosure can efficiently generate inspection results for the bridge 1, which is the structure to be inspected. As described above, the damage data may be generated by the control terminal 200, rather than the information processing device 300. Therefore, the operation shown in Fig. 16 may be performed by the control terminal 200.

[0136] Fig. 17 is a flow chart showing an example of operation of the control terminal 200 according to an embodiment of the present disclosure. Fig. 17 shows an example of a process for generating flight information by the control terminal 200 using damage data generated by the information processing device 300. Hereinafter, an example of operation of the control terminal 200 according to an embodiment of the present disclosure will be described with reference to Fig. 17.

[0137] The control terminal 200 displays a still image captured by the hovering camera 100, and when the user specifies the location of damage on the still image (step S311), the control terminal 200 acquires the damage position of the specified location from the damage data generated by the information processing device 300 (step S312). The method of specifying the location of damage is not important, but for example, a still image may be displayed and the user may specify the location of damage by touching the touch panel of the control terminal 200 with a finger or the like.

[0138] After acquiring the damage location at the location specified by the user from the damage data, the control terminal 200 then generates flight information for flying the hovering camera 100 to the damage location acquired from the damage data and capturing an image of the damage location (step S313). The processing of step S313 is executed, for example, by the flight information generating unit 232. The flight information generated by the control terminal 200 in step S313 is for checking the damage location in detail, and therefore may instruct the hovering camera 100 to narrow the intervals between image capturing positions or to perform the above-mentioned special image capturing at each image capturing position, compared to the flight information generated by the control terminal 200 described above in the description of FIG. 5. When special image capturing is performed, the flight information includes information for controlling the special image capturing, as described above.

[0139] When the flight information is generated in step S313, the control terminal 200 transmits the generated flight information to the hovering camera 100, and the hovering camera 100 performs flight and imaging processing based on the flight information, as shown in steps S104 and S105 of Fig. 5. Then, when the hovering camera 100 completes the imaging processing at the last imaging point, it automatically flies to the base station 600 and returns to the base station 600, and the control terminal 200 acquires the images captured by the hovering camera 100 from the hovering camera 100, as shown in steps S106 and S107 of Fig. 5.

[0140] By performing the operations shown in FIG. 17, the control terminal 200 according to one embodiment of the present disclosure can use the damage data generated by the information processing device 300 to generate flight information that allows the hovering camera 100 to capture detailed images of the damaged portion of the bridge girder 3.

[0141] <2. Summary> As described above, one embodiment of the present disclosure provides a hovering camera 100 that flies automatically based on set flight information and captures images of a structure to be inspected, and an inspection system 10 that can check the state of damage to the structure based on still images captured by the hovering camera 100.

[0142] The inspection system 10 according to an embodiment of the present disclosure uses information about the structure to be inspected when the control terminal 200 generates flight information to be transmitted to the hovering camera 100. By using the information about the structure to be inspected, the control terminal 200 can generate flight information for flying the hovering camera 100 to efficiently inspect the structure to be inspected.

[0143] In the above embodiment, the images captured by the hovering camera 100 are still images, and an example of the inspection system 10 in which the still images are used to inspect the damage state of the bridge 1 has been described; however, the present disclosure is not limited to such an example. The hovering camera 100 may capture moving images of the bridge 1 while flying, and the information processing device 300 may generate damage data using the moving images captured by the hovering camera 100. The hovering camera 100 periodically acquires location information during the execution of the moving image capturing process, and by linking the capturing time of the moving images with the acquisition time of the location information, the information processing device 300 can generate damage data using the moving images.

[0144] The steps in the processes performed by each device in this specification do not necessarily have to be processed in chronological order according to the order shown in the sequence diagrams or flowcharts. For example, the steps in the processes performed by each device may be processed in an order different from the order shown in the flowcharts, or may be processed in parallel.

[0145] Computer programs can also be created to cause hardware such as the CPU, ROM, and RAM built into each device to perform functions equivalent to those of the above-described device configurations. Storage media storing such computer programs can also be provided. Furthermore, by configuring each functional block shown in the functional block diagram with hardware or hardware circuits, a series of processes can be realized with hardware or hardware circuits. Furthermore, some or all of the functional blocks shown in the functional block diagrams used in the above description may be realized in a server device connected via a network such as the Internet. Furthermore, the configuration of each functional block shown in the functional block diagrams used in the above description may be realized in a single device or in a system in which multiple devices work together. A system in which multiple devices work together may include, for example, a combination of multiple server devices, a combination of a server device and a terminal device, etc.

[0146] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0147] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0148] The following configurations also fall within the technical scope of the present disclosure. (1) an imaging position information acquisition unit that acquires imaging position information when an image of the structure is captured, the imaging position information being acquired by an imaging device that flies around the periphery of the structure and captures an image of the structure based on predetermined flight information; a damage data generation unit that generates data related to damage to the structure, including location information of the damage to the structure included in the captured image, using the captured image of the structure captured by the imaging device and the image capture location information; and An information processing device comprising: (2) The information processing device according to (1), further comprising a damage position calculation unit that calculates a position of damage to the structure included in the captured image. (3) further comprising an image synthesis unit that generates a synthetic image by stitching together a plurality of the captured images; The information processing device described in (2), wherein the damage position calculation unit calculates the position of damage to the structure shown in the captured image using information on the image capturing position of a captured image associated with information on an image capturing position with relatively high positional accuracy among the captured images that are the basis for generation by the image synthesis unit. (4) The information processing device described in (3), wherein the damage position calculation unit calculates the position of damage to the structure shown in the captured image using information on the capturing position of the captured image corresponding to a corner part of the composite image as the capturing position with relatively high positional accuracy. (5) The information processing device according to any one of (2) to (4), wherein the damage data generation unit generates data relating to damage to the structure using the position of the damage to the structure calculated by the damage position calculation unit. (6) The information processing device according to any one of (2) to (5), wherein the damage position calculation unit calculates an absolute position of damage to the structure shown in the captured image. (7) The information processing device described in any one of (1) to (6) further includes a flight information generation unit that generates flight information using the data generated by the damage data generation unit to cause the imaging device to capture the location of the damage corresponding to the data. (8) Acquiring information about an imaging position when imaging the structure from an imaging device that flies around the periphery of the structure and images the structure based on predetermined flight information; generating data relating to damage to the structure, including position information of the damage to the structure shown in the captured image, using the captured image of the structure captured by the imaging device and information on the capturing position of the captured image; An information processing method, including: (9) On the computer, Acquiring information about an imaging position when imaging the structure from an imaging device that flies around the periphery of the structure and images the structure based on predetermined flight information; generating data relating to damage to the structure, including position information of the damage to the structure shown in the captured image, using the captured image of the structure captured by the imaging device and information on the capturing position of the captured image; A computer program that executes [Explanation of symbols]

[0149] 10 Inspection System 100 Hovering Camera 101 Imaging device 104a~104d rotors 108a~108d Motor 110 control section 120 Communications Department 130 Sensor unit 132 Location information acquisition unit 140 Storage section 150 Battery 200 Control Terminal 300 Information processing device 400 wireless relay nodes 500 location estimation nodes 600 Base Station 700 charging stations

Claims

1. A computer-implemented flight information control method, comprising: Identifying a target area of ​​the object to be imaged based on information input by a user before the start of flight; A flying imaging device that flies and photographs based on flight information generates the flight information including a flight path for photographing the target area in accordance with the identification of the target area of ​​the imaging object, displaying the flight path of the generated flight information on a display unit; Transmitting the generated flight information to the flight imaging device; Sending a takeoff command to the flying imaging device; The flight information includes position information where the flying imaging device performs imaging processing, The display includes a line indicating the flight path superimposed on a general view of the imaged object. Flight information control methods.

2. The flying imaging device has a sensor unit and provides sensor information detected by the sensor unit; Identifying the status of an obstacle based on the sensor information provided by the flying image capture device, and controlling the flying image capture device based on the status of the obstacle The flight information control method according to claim 1 .

3. The control of the flight imaging device regenerates flight information so that the flight path avoids the location of the obstacle, and transmits the regenerated flight information to the flight imaging device. The flight information control method according to claim 2.

4. The display includes displaying an overview of the image capture object on the display unit, The generation generates the flight information including a flight path for the flight imaging device to capture the target area in response to identification of the target area based on an input from a user with respect to the overview map of the imaging object. The flight information control method according to claim 1 .

5. When the flight path for capturing an image of the target area cannot be flown in a single flight, the flight path is divided and the flight information including the divided flight paths is generated; The display is a diagram of the object to be imaged, superimposed on a plurality of divided flight paths. The flight information control method according to claim 1 .

6. The generation includes, when generating the flight information, generating a plurality of flight paths for photographing the target area by different paths, and generating the flight information including a flight path selected from the plurality of flight paths; The display displays the plurality of flight paths in a selectable manner by superimposing them on an overview diagram of the imaged object. The flight information control method according to claim 1 .

7. The generation of the flight information includes a flight path along which the flight imaging device travels back and forth in one direction along the target area at equal intervals in a direction intersecting the one direction to capture the target area. The flight information control method according to claim 1 .

8. The input information by the user is a touch input to the screen of the display unit of the terminal. The flight information control method according to claim 1 .

9. The flight path is defined by latitude, longitude, and altitude. The flight information control method according to claim 1 .

10. The altitude of the flight path is expressed as a relative altitude from the takeoff and landing point of the flying image capture device. The flight information control method according to claim 1 .

11. The flight information includes information about an estimated time required for the flight based on the planned flight path. The flight information control method according to claim 1 .

12. a target area specifying unit that specifies a target area of ​​an image capture object based on input information by a user before the start of flight; A flight imaging device that flies and photographs based on flight information generates the flight information including a flight path for photographing the target area in accordance with the identification of the target area of ​​the imaging object, and a display unit that displays the flight path of the generated flight information; A communication unit that is capable of transmitting and receiving information via wireless communication with the flight imaging device and transmits the generated flight information and takeoff instructions to the flight imaging device, The flight information includes position information where the flying imaging device performs imaging processing, The display unit displays a line indicating the flight path superimposed on a general view of the imaged object. Control device.

13. On the computer, Identifying a target area of ​​the object to be imaged based on information input by a user before the start of flight; A flying imaging device that flies and photographs based on flight information generates the flight information including a flight path for photographing the target area in accordance with the identification of the target area of ​​the imaging object, displaying the flight path of the generated flight information on a display unit; Transmitting the generated flight information to the flight imaging device; sending a takeoff command to the flying imaging device; Execute the process, The flight information includes position information where the flying imaging device performs imaging processing, The display includes a line indicating the flight path superimposed on a general view of the imaged object. Computer program.

Images