Information processing method, information processing device and computer program

Abstract

To improve efficiency of a check using a flying body which is capable of implementing imaging.SOLUTION: Flight information of a flight imaging apparatus including a flight path including setting information of a camera, which performs imaging, is created and based on data of an IMU (Inertial Measurement Unit) sensor and image data obtained by imaging with the camera, a position of the flight imaging apparatus is determined. On the basis of the flight information and the position of the flight imaging apparatus, a flight of the flight imaging apparatus is controlled and based on information on a detected obstacle that may disturbs the flight, control is performed to stop the flight and shift to a hovering state.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing method, an information processing device, and a computer program. [Background technology]

[0002] There is disclosed a technology for a photographic method in which a camera is attached to a wirelessly controlled flying object and photographs are taken with the camera (see, for example, Patent Document 1, etc.). By attaching a camera to the flying object, it becomes possible to take photographs from the sky or from places where a tripod cannot be set up. Furthermore, by attaching a camera to the flying object to take photographs, various advantages are obtained, such as lower costs than when using a real airplane or helicopter, safer photography, the ability to take photographs at low altitudes or in narrow places, and the ability to approach a target and take photographs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-27448 A Summary of the Invention [Problem to be solved by the invention]

[0004] If an aircraft equipped with such a camera can be used to effectively capture images of places that humans cannot easily enter, it is believed that this will be extremely useful for inspecting structures that are difficult for humans to approach, such as social infrastructure such as bridges, tunnels, dams, and roads over rivers and seas, and industrial infrastructure such as airports, buildings, warehouses, factories, and plants.

[0005] Therefore, the present disclosure proposes a new and improved information processing method, information processing device, and computer program that can improve the efficiency of inspections using aircraft capable of performing imaging. [Means for solving the problem]

[0006] According to the present disclosure, a control device is provided that includes an acquisition unit that acquires information regarding an overall condition of a structure, and a flight information generation unit that generates flight information for an aircraft that flies around the edge of the structure and images the structure, based on the information acquired by the acquisition unit.

[0007] In addition, according to the present disclosure, there is provided an imaging device including a control unit that controls the device to fly around the edge of a structure based on flight information generated based on information regarding the general condition of the structure, and an imaging unit that performs imaging processing of the structure while flying based on the flight information.

[0008] The present disclosure also provides a control method that includes acquiring information regarding an overall condition of a structure, and generating flight information for an aircraft that flies around the edge of the structure and images the structure based on the acquired information.

[0009] The present disclosure also provides an imaging method that includes controlling flight around the edge of a structure based on flight information generated based on information regarding the general condition of the structure, and performing imaging processing of the structure while flying based on the flight information.

[0010] In addition, according to the present disclosure, a computer program is provided that causes a computer to acquire information regarding an overall condition of a structure, and based on the acquired information, generate flight information for an aircraft that flies around the edge of the structure and images the structure.

[0011] In addition, according to the present disclosure, a computer program is provided that causes a computer to control the drone to fly around the edge of a structure based on flight information generated based on information regarding the general condition of the structure, and to perform imaging processing of the structure while flying based on the flight information. Effect of the Invention

[0012] As described above, the present disclosure provides a new and improved information processing method, information processing device, and computer program that can improve the efficiency of inspections using an aircraft capable of performing image capture.

[0013] It should be noted that the above effects are not necessarily limiting, and any of the effects shown in this specification, or other effects that can be understood from this specification, may be achieved in addition to or instead of the above effects. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an embodiment of the present disclosure. [Diagram 2] 1 is an explanatory diagram illustrating an example of a system configuration of an inspection system 10 according to an embodiment of the present disclosure. [Diagram 3] 1 is an explanatory diagram illustrating an example of a functional configuration of a hovering camera 100 according to an embodiment of the present disclosure. [Figure 4] 2 is an explanatory diagram illustrating an example of a functional configuration of a control terminal 200 according to an embodiment of the present disclosure. FIG. [Diagram 5] 4 is a flowchart showing an example of the operation of the inspection system 10 according to an embodiment of the present disclosure. [Figure 6] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 7] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 8] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 9] 1 is an explanatory diagram conceptually showing how the underside of a bridge 1 is imaged by a hovering camera 100. FIG. [Figure 10] 1 is an explanatory diagram conceptually showing the operation of a hovering camera 100 in an inspection system 10 according to an embodiment of the present disclosure. [Figure 11] 1 is an explanatory diagram conceptually showing the operation of a hovering camera 100 in an inspection system 10 according to an embodiment of the present disclosure. [Figure 12] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 13] FIG. 13 is an explanatory diagram showing an overview of inspecting the bottom surface of a bridge girder 3. [Figure 14] 1 is an explanatory diagram showing an example of an image 20 obtained by stitching still images captured by a hovering camera 100. FIG. [Figure 15] 2 is an explanatory diagram illustrating an example of a functional configuration of an information processing device 300 according to an embodiment of the present disclosure. [Figure 16] 4 is a flowchart showing an example of operation of the information processing device 300 according to an embodiment of the present disclosure. [Figure 17] 4 is a flowchart showing an example of the operation of the control terminal 200 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0016] The explanation will be given in the following order. 1. One embodiment of the present disclosure 1.1. Overview 1.2. System configuration example 1.3. Example of functional configuration 1.4. Example of operation 1.5. Example of Damage Data Generation 1.5.1 Functional configuration example 1.5.2 Example of operation 2. Summary

[0017] 1. One embodiment of the present disclosure [1.1. Overview] Before describing an embodiment of the present disclosure in detail, an overview of the embodiment of the present disclosure will be first described.

[0018] In the maintenance and management of structures such as roads, bridges, tunnels, buildings, etc., human inspection of the structure's condition is essential. Normally, visual inspection of such structures is done by workers approaching the structure and visually checking for damage such as corrosion or cracks, or loosening of connecting members such as bolts, and also by conducting a tapping test to check for the presence or absence of such abnormalities.

[0019] In order to maintain bridges, especially concrete bridges, it has become necessary to erect scaffolding on the undersides of bridge piers and girders for workers to carry out visual and hammering inspections of the girders and piers, and to close some or all of the lanes to ensure the safety of the workers and to position work vehicles. These factors can cause problems not only in the cost required for inspections, but also in the cost required to arrange road guides for road closures, and further in the traffic congestion on detours that may occur due to road closures.

[0020] In addition, there are bridges where it is not easy or even possible to set up scaffolding because they are built over rivers or the sea. In light of these circumstances, there is a need for technology that enables safe, low-cost inspection of structures without affecting traffic.

[0021] In light of the above circumstances, the present inventors have investigated technology that would enable safe, low-cost inspection of structures without affecting traffic. As described below, the present inventors have developed a technology that uses an aircraft equipped with an imaging device (in the following description, an aircraft equipped with an imaging device is also referred to as a "hovering camera") to perform inspection of structures safely, without affecting traffic. This resulted in the development of a technology that enables inspection work to be performed quickly and at low cost.

[0022] Fig. 1 is an explanatory diagram for explaining an outline of an embodiment of the present disclosure. Fig. 1 shows a schematic diagram of a bridge 1 constructed of concrete, for example. When inspecting the bridge 1 constructed of concrete, as described above, it has been necessary to set up scaffolding on the back surface of the pier 2 or bridge girder 3 so that workers can visually inspect whether there is any damage such as cracks or corrosion, and it has been necessary to close some or all of the lanes to ensure the safety of workers or to place work vehicles.

[0023] In one embodiment of the present disclosure, a hovering camera 100 is used when inspecting a bridge 1. The hovering camera 100 is an aircraft equipped with an imaging device, configured to fly automatically according to preset flight information (which in this embodiment includes information on a flight path and an imaging position of a still image). Note that the information on the imaging position of a still image may include, for example, a position where an imaging process is performed, an imaging direction, a travel time to a position where the next imaging process is performed, and the like.

[0024] For example, when inspecting the back side (bottom surface) of the bridge girder 3, the hovering camera 100 is automatically flown and the hovering camera 100 is caused to capture an image of the back side of the bridge girder 3. By causing the hovering camera 100 to capture an image of the back side of the bridge girder 3, it becomes unnecessary to set up scaffolding on the back side of the pier 2 or the bridge girder 3 for inspecting the bridge girder 3, or the frequency of closing lanes to traffic is reduced, or traffic closure becomes unnecessary. Also, for example, when inspecting the side (side surface) of the bridge girder 3, the hovering camera 100 is automatically flown and the hovering camera 100 is caused to capture an image of the side of the bridge girder 3. Therefore, by automatically flying the hovering camera 100 and causing the hovering camera 100 to capture an image of the back side or side of the bridge girder 3, the safety of the workers is ensured, and the bridge 1 can be inspected at low cost without affecting traffic.

[0025] In order to automatically fly the hovering camera 100 to capture an image of the underside of the bridge girder 3, it is necessary to set the flight path of the hovering camera 100 and to set information on the capture position of a still image at the underside of the bridge girder 3. In one embodiment of the present disclosure, an object is to enable efficient inspection of the bridge 1 by efficiently creating flight information to be set in the hovering camera 100 using information on the general condition of the bridge 1.

[0026] The outline of an embodiment of the present disclosure has been described above. Next, a configuration example of an inspection system according to an embodiment of the present disclosure will be described.

[0027] [1.2. System configuration example] Fig. 2 is an explanatory diagram showing a system configuration example of the inspection system 10 according to an embodiment of the present disclosure. The inspection system 10 according to an embodiment of the present disclosure shown in Fig. 2 is a system intended to efficiently inspect a structure, for example, a bridge 1. Hereinafter, a system configuration example of the inspection system 10 according to an embodiment of the present disclosure will be described with reference to Fig. 2.

[0028] As shown in FIG. 2, the inspection system 10 according to one embodiment of the present disclosure includes a hovering camera 100, a control terminal 200, an information processing device 300, a wireless relay node 400, a position estimation node 500, a base station 600, a charging station 700, and a server device 800.

[0029] The hovering camera 100 is an example of an imaging device of the present disclosure, and is an aircraft equipped with an imaging device as described above. The hovering camera 100 is an aircraft configured to fly automatically based on a specified flight path and capture still images with the imaging device at a specified imaging position. The hovering camera 100 flies, for example, with four rotors. By controlling the rotation of each rotor, the drone can fly while ascending, descending, and moving horizontally. Of course, the number of rotors is not limited to the above example.

[0030] The flight path from the flight start position to the flight end position and the imaging position set in the hovering camera 100 are set as, for example, GPS (Global Positioning System) position information. Therefore, the hovering camera 100 may incorporate a GPS receiver that receives radio waves from a GPS satellite and calculates the current position. The flight path set in the hovering camera 100 may be set with all of the latitude, longitude, and altitude set as GPS position information, or only the latitude and longitude may be set as GPS position information, and the altitude may be set as, for example, a relative height from a base station 600 described later.

[0031] The control terminal 200 is an example of a control device of the present disclosure, and is a terminal that executes control regarding the flight of the hovering camera 100. The control terminal 200 performs control regarding the flight of the hovering camera 100, for example, generation of flight information to be sent to the hovering camera 100, instruction to the hovering camera 100 to take off, instruction to return to the base station 600 described later, and operation of the hovering camera 100 when the hovering camera 100 cannot fly automatically for some reason. The process of generating flight information of the hovering camera 100 by the control terminal 200 will be described in detail later, but an example will be briefly described here.

[0032] When generating flight information for the hovering camera 100, the control terminal 200 reads information on the general condition of the bridge 1 to be inspected, for example, a general condition map of the bridge 1 to be inspected, and displays it on the screen. Points on the general condition map of the bridge 1 are associated with points on map data containing detailed GPS information in advance. This association is preferably performed for at least two pairs of points. By associating points on map data containing detailed GPS information in advance with the general condition map of the bridge 1, the flight path of the hovering camera 100 is defined as a GPS value. Then, the control terminal 200 generates the flight path of the hovering camera 100 based on the general condition map of the bridge 1. The flight path of the hovering camera 100 is displayed superimposed on the general condition map so that the user (the worker inspecting the structure) can easily understand it.

[0033] When generating flight information for the hovering camera 100, the control terminal 200 may take into consideration the structure and dimensions of the bridge 1, and the parts of the bridge 1 to be imaged by the hovering camera 100. When generating flight information for the hovering camera 100, the control terminal 200 may generate flight information that causes the hovering camera 100 to image in detail parts that are considered to have a high possibility of being damaged.

[0034] As described above, the flight route set for the hovering camera 100 may have all of the latitude, longitude, and altitude set as GPS position information, but there may be cases where altitude data does not exist in the general situation map of the bridge 1. If altitude data does not exist in the general situation map of the bridge 1, the flight route set for the hovering camera 100 may have only the latitude and longitude set as GPS position information, and the altitude may be set as a relative height from the base station 600, for example.

[0035] When setting flight information for the hovering camera 100, the control terminal 200 desirably generates flight information that keeps a constant distance from the image-capturing surface when the hovering camera 100 captures an image of the bridge 1. By generating flight information that keeps a constant distance from the image-capturing surface when the hovering camera 100 captures an image of the bridge 1, the control terminal 200 can cause the hovering camera 100 to generate an image of the same scale.

[0036] The control terminal 200 is a portable device such as a notebook computer or a tablet terminal. The control terminal 200 is a device capable of wirelessly transmitting and receiving information to and from the hovering camera 100. The control terminal 200 may perform wireless communication with the hovering camera 100 directly, but since the hovering camera 100 may fly beyond the communication range of the control terminal 200 during inspection of a structure, particularly a bridge 1, the communication may be performed via a wireless relay node 400 that is installed during inspection.

[0037] The control terminal 200 acquires images captured by an imaging device while the hovering camera 100 is flying, and displays them as necessary. The control terminal 200 may acquire and display, by streaming, moving images captured by an imaging device while the hovering camera 100 is flying. By acquiring and displaying, by streaming, moving images captured by an imaging device while the hovering camera 100 is flying, the control terminal 200 can present to the user where the hovering camera 100 is flying.

[0038] The information processing device 300 is a device that processes various information, and may be a device having a function of processing information such as a personal computer (PC) or a game machine. In this embodiment, the information processing device 300 is a device that has a function of displaying an image captured by the hovering camera 100 in particular to allow a user to check the state of the bridge 1. The information processing device 300 also has a function of calculating an absolute damage position of the bridge girder 3 from the image captured by the hovering camera 100 and generating damage data described later. The information processing device 300 may have a function of transmitting the generated damage data to the server device 800. Note that the control terminal 200 may have a function of calculating an absolute damage position of the bridge girder 3 from the image captured by the hovering camera 100 and generating damage data described later.

[0039] The information processing device 300 acquires the image captured by the hovering camera 100, for example, from the control terminal 200. The timing at which the information processing device 300 acquires the image captured by the hovering camera 100 is not limited to a specific timing, but for example, the information processing device 300 may acquire the image captured by the hovering camera 100 from the control terminal 200 at the timing at which one flight of the hovering camera 100 ends.

[0040] The wireless relay node 400 is a device that relays wireless communication between the hovering camera 100 and the control terminal 200. As described above, in the inspection of a structure, particularly a bridge 1, the hovering camera 100 may fly beyond the communication range of the control terminal 200. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 may be performed via the wireless relay node 400 that is installed when inspecting the structure. The number of wireless relay nodes 400 is not limited to one, and multiple nodes may be installed depending on the inspection range of the bridge 1. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 may be performed via multiple wireless relay nodes 400. The hovering camera 100 may switch the communication destination between the control terminal 200 and the wireless relay node 400 depending on the radio wave conditions.

[0041] The wireless relay node 400 can be installed at an appropriate location on the bridge surface (preferably on the sidewalk) when inspecting the bridge 1. The wireless relay node 400 may also be installed so as to be suspended from the parapet of the bridge girder 3. Before inspecting the bridge 1, it is desirable to confirm that the wireless relay node 400 is operating normally by a predetermined method, for example, using the control terminal 200.

[0042] The position estimation node 500 is a device for causing the hovering camera 100 to estimate its current position. As described above, the flight path of the hovering camera 100 is set as, for example, GPS position information. At this time, if there is nothing blocking radio waves from a GPS satellite, the hovering camera 100 can know its current position with extremely high accuracy. However, it is unavoidable for the hovering camera 100 to get under the bridge girder 3, which may block the GPS. If radio waves from the satellite are blocked, or if multipath occurs due to radio wave reflection by the bridge 1, the hovering camera 100 may not be able to determine its current position with high accuracy.

[0043] Therefore, in this embodiment, a position estimation node 500 is provided under the bridge girder 3 to allow the hovering camera 100 to accurately acquire the current position. As the position estimation node 500, for example, an AR (Augmented Reality) marker or a GPS signal transmitter may be used.

[0044] When an AR marker is used as the position estimation node 500, in order to have the hovering camera 100 recognize the current position, for example, the position estimation node 500 is hung from both ends of the bridge 1, and the hovering camera 100 is made to capture the position estimation node 500. Then, the hovering camera 100 that captured the position estimation node 500 is made to fly between the specified position estimation nodes 500. The hovering camera 100 can grasp the position between the position estimation nodes 500, for example, from an integrated value of a sensor (for example, an IMU (Inertial Measurement Unit) sensor) provided in the hovering camera 100 and a distance to the destination position estimation node 500 calculated from the captured image. Therefore, the hovering camera 100 can accurately obtain the current position even under the bridge girder 3 by capturing an image of the position estimation node 500.

[0045] Furthermore, when a GPS signal transmitter is used as the position estimation node 500, in order to allow the hovering camera 100 to recognize its current position, the position estimation node 500 is installed, for example, at a diagonal or at each of the four corners of the bridge 1. By receiving the GPS signal emitted from the position estimation node 500, the hovering camera 100 can accurately obtain its current position even under the bridge girder 3.

[0046] The base station 600 is a device provided for the takeoff and landing of the hovering camera 100. The base station 600 is equipped with a GPS receiver and calculates its current position by receiving radio waves from a GPS satellite. The current position calculated by the base station 600 is sent to the control terminal 200. By sending the current position calculated by the base station 600 to the control terminal 200, the control terminal 200 becomes able to display the position of the base station 600 on the overview map of the bridge 1.

[0047] The base station 600 may have a function of checking the operation of the hovering camera 100. The check of the operation of the hovering camera 100 performed by the base station 600 may include, for example, checking the communication function, the imaging function, the flight function, and calibration of various sensors. It goes without saying that the method of calibrating the sensor of the hovering camera 100 is not limited to the method using the base station 600. As a method of calibrating the sensor of the hovering camera 100, for example, a method of fixing the hovering camera 100 to a jig dedicated to calibration and rotating the hovering camera 100 in the pitch direction or roll direction to calibrate the sensor may be used.

[0048] The charging station 700 charges a secondary battery provided in the hovering camera 100. The hovering camera 100 is powered by a battery, and consumes power stored in the battery during flight and image capture. If the battery provided in the hovering camera 100 is a secondary battery, the charging station 700 can recover power consumed by the hovering camera 100 by charging the battery. The charging station 700 may charge the hovering camera 100 by connecting a cable or the like to the hovering camera 100 and supplying power to the hovering camera 100, or may charge the hovering camera 100 by supplying power to the hovering camera 100 using a non-contact power transmission method.

[0049] The server device 800 is a device that stores various data. In this embodiment, the server device 800 may store the damage data generated by the information processing device 300.

[0050] 2, the inspection system 10 according to an embodiment of the present disclosure can capture an image of the bridge 1 using the hovering camera 100 to obtain an image of the bridge 1. By capturing an image of the bridge 1 using the hovering camera 100, the inspection system 10 according to an embodiment of the present disclosure can eliminate the need to set up scaffolding on bridge piers and girders, reduce the frequency of closing some or all lanes to ensure the safety of workers, or eliminate the need for road closures, thereby enabling low-cost and efficient inspection of the bridge 1.

[0051] An example of the system configuration of the inspection system 10 according to an embodiment of the present disclosure has been described. Next, an example of the functional configuration of the hovering camera 100 and the control terminal 200 constituting the inspection system 10 according to an embodiment of the present disclosure will be described.

[0052] [1.3. Example of functional configuration] First, a functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure will be described. Fig. 3 is an explanatory diagram showing a functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure. Hereinafter, the functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure will be described with reference to Fig. 3.

[0053] As shown in FIG. 3, the hovering camera 100 according to one embodiment of the present disclosure includes an imaging device 101, rotors 104a to 104d, motors 108a to 108d, a control unit 110, a communication unit 120, a sensor unit 130, a position information acquisition unit 132, a memory unit 140, and a battery 150.

[0054] The control unit 110 controls the operation of the hovering camera 100. For example, the control unit 110 can control adjustment of the rotation speed of the rotors 104a-104d by adjusting the rotation speed of the motors 108a-108d, imaging processing by the imaging device 101, transmission and reception processing of information to and from other devices (e.g., the control terminal 200) via the communication unit 120, and storage and reading of information in the memory unit 140.

[0055] In this embodiment, the control unit 110 controls flight by adjusting the rotation speeds of the motors 108a to 108d based on flight information transmitted from the control terminal 200, and controls the execution of still image capture processing for the imaging device 101. The control unit 110 controls the motors 108a to 108d and the imaging device 101 based on flight information transmitted from the control terminal 200, thereby making it possible to provide the control terminal 200 with an image based on a request from the control terminal 200.

[0056] The imaging device 101 is composed of a lens, an imaging element such as a CCD image sensor or a CMOS image sensor, a flash, etc. The imaging device 101 provided in the hovering camera 100 captures still images or moving images under the control of the control terminal 200. The image captured by the imaging device 101 is transmitted from the communication unit 120 to the control terminal 200. In this embodiment, the imaging device 101 performs imaging processing based on information on the imaging position of the still image included in the flight information transmitted from the control terminal 200. The image obtained by the imaging processing of the imaging device 101 may be stored in the storage unit 140 or transmitted from the communication unit 120 to the control terminal 200. When the hovering camera 100 captures an image of the bottom side of the bridge 1, it is considered that the sunlight is blocked by the bridge 1 and the brightness is insufficient, so the hovering camera 100 may emit a flash when capturing an image of the bottom side of the bridge 1.

[0057] The imaging device 101 can change the imaging direction to any direction, for example, under the control of the control unit 110. For example, when the horizontal direction of the hovering camera is set to 0 degrees, The imaging device 101 can capture an image in an imaging direction represented by a range of ±90 degrees in the direction. Since the imaging device 101 can change the imaging direction, the hovering camera 100 can capture an image in a predetermined direction and provide the captured image to the control terminal 200. The control unit 110 associates the position information of the hovering camera 100 at the time when the imaging device 101 captured a still image (which may include position information obtained by positioning using GPS or positioning using the position estimation node 500. Positioning using the position estimation node 500 will be described later), aircraft information at the time of capturing an image (e.g., yaw angle, pitch angle, acceleration, angular velocity), and information on the imaging direction as metadata of the still image. As a method for storing the associated metadata, the metadata may be added to an additional information area of ​​the still image data (e.g., a specific area of ​​the Exif format), or the metadata may be recorded in a file separate from the image file, or may be stored as separate data.

[0058] The rotors 104a to 104d generate lift through rotation, thereby flying the hovering camera 100. The rotation of the rotors 104a to 104d is achieved by the rotation of the motors 108a to 108d. The motors 108a to 108d rotate the rotors 104a to 104d. The rotation of the motors 108a to 108d can be controlled by the control unit 110.

[0059] The communication unit 120 performs processing for transmitting and receiving information via wireless communication with the control terminal 200. The hovering camera 100 transmits images captured by the imaging device 101 from the communication unit 120 to the control terminal 200. The hovering camera 100 also receives flight-related instructions from the control terminal 200 via the communication unit 120.

[0060] The sensor unit 130 is a group of devices that acquires the state of the hovering camera 100, and may be composed of, for example, an acceleration sensor, a gyro sensor, an ultrasonic sensor, an air pressure sensor, an optical flow sensor, a laser range finder, and the like. The sensor unit 130 may convert the acquired state of the hovering camera 100 into a predetermined signal and provide it to the control unit 110 as necessary. The position information acquisition unit 132 acquires information on the current position of the hovering camera 100 using, for example, a GPS or a vision sensor. The position information acquisition unit 132 may provide the acquired information on the current position of the hovering camera 100 to the control unit 110 as necessary. The control unit 110 executes flight control of the hovering camera 100 based on the flight information received from the control terminal 200, using the information on the current position of the hovering camera 100 acquired by the position information acquisition unit 132.

[0061] In addition, the sensor unit 130 detects obstacles that may impede flight during flight. By detecting an obstacle with the sensor unit 130, the hovering camera 100 can provide information about the detected obstacle to the control terminal 200.

[0062] The storage unit 140 stores various information. Examples of information stored in the storage unit 140 include flight information of the hovering camera 100 transmitted from the control terminal 200, images captured by the imaging device 101, and the like.

[0063] The battery 150 stores power for operating the hovering camera 100. The battery 150 may be a primary battery that can only be discharged, or may be a secondary battery that can also be charged. When the battery 150 is a secondary battery, the battery 150 may receive a supply of power from, for example, a charging station 700 shown in FIG. 2.

[0064] The hovering camera 100 according to one embodiment of the present disclosure has a configuration as shown in FIG. 3, and can automatically fly based on a flight path included in the flight information transmitted from the control terminal 200, and can perform imaging processing based on information on the imaging position of a still image included in the flight information transmitted from the control terminal 200.

[0065] An example of the functional configuration of the hovering camera 100 according to an embodiment of the present disclosure has been described above with reference to Fig. 3. Next, an example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure will be described.

[0066] 4 is an explanatory diagram showing an example of a functional configuration of the control terminal 200 according to an embodiment of the present disclosure. Hereinafter, the example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure will be described with reference to FIG.

[0067] As shown in FIG. 4, the control terminal 200 according to an embodiment of the present disclosure includes a display unit 210, a communication unit 220, a control unit 230, and a storage unit 240.

[0068] The display unit 210 is composed of a flat panel display device such as a liquid crystal display device, an organic EL display device, etc. The display unit 210 can display, for example, images captured by the imaging device 101 and information for controlling the operation of the hovering camera 100. The display unit 210 is provided with a touch panel, and a user can directly operate the information displayed on the display unit 210 by touching the display unit 210 with a finger or the like.

[0069] The communication unit 220 transmits and receives information to and from the hovering camera 100 via wireless communication. The control terminal 200 receives images captured by the imaging device 101 from the hovering camera 100 via the communication unit 220. The control terminal 200 also transmits instructions related to the flight of the hovering camera 100 from the communication unit 220 to the hovering camera 100. Commands related to the flight of the hovering camera 100 can be generated by the control unit 230.

[0070] The control unit 230 controls the operation of the control terminal 200. For example, the control unit 230 can control the display processing of characters, figures, images, and other information on the display unit 210, and the transmission and reception processing of information between the control unit 200 and other devices (e.g., the hovering camera 100) via the communication unit 220. The control unit 230 is also configured to include a flight information generation unit 232 and a display control unit 234.

[0071] The flight information generating unit 232 generates flight information to be transmitted to the hovering camera 100. When generating the flight information, the flight information generating unit 232 uses, for example, information on the structure to be inspected that is stored in the storage unit 240 described below. After generating the flight information, the flight information generating unit 232 causes the communication unit 220 to transmit the generated flight information before the hovering camera 100 takes off.

[0072] The flight information generating process by the flight information generating unit 232 will be described in detail later, but an example of the flight information generating process by the flight information generating unit 232 will be briefly described. When generating flight information for the hovering camera 100, the flight information generating unit 232 reads a general view of the bridge 1 to be inspected. The read general view of the bridge 1 is displayed on the display unit 210 by the display control unit 234. As described above, points on the general view of the bridge 1 are previously associated with points on map data containing detailed GPS information. It is desirable that this association is performed for at least two pairs of points. By previously associating points on map data containing detailed GPS information with the general view of the bridge 1, the flight route of the hovering camera 100 is defined as a GPS value (a set of latitude and longitude).

[0073] The flight information generating unit 232 then generates a flight path for the hovering camera 100 based on this overview map of the bridge 1. When generating the flight path for the hovering camera 100, the flight information generating unit 232 uses information on the construction method, width, span length, and other structural information on the bridge 1, the flight time for the hovering camera 100, and information on the inspection method for the bridge 1. Concrete bridges are classified into reinforced concrete (RC) and prestressed concrete (PC) depending on the reinforcement method, and are classified into various types depending on the shape of the girders, for example, RCT girder bridges, PCT girder bridges, PC hollow deck bridges, RC box girder bridges, Bridges can be divided into two types: PC box girder bridges, and PC box girder bridges. Therefore, if the construction method of the bridge 1 to be inspected is known, the flight information generating unit 232 can generate a flight path suitable for the construction method of the bridge 1. Then, the flight information generating unit 232 displays the flight path of the hovering camera 100 superimposed on the overview map of the bridge 1.

[0074] The flight information generating unit 232 defines the flight path of the hovering camera 100 as GPS values ​​(a set of latitude and longitude) as described above. By the flight information generating unit 232 defining the flight path of the hovering camera 100 as GPS values, the hovering camera 100 can determine at what position during flight the imaging process should be performed, based on the GPS values.

[0075] The display control unit 234 controls the display of characters, figures, images, and other information on the display unit 210. The display of characters, figures, symbols, images, and other information displayed on the display unit 210 in the drawings referred to in the following description is controlled by the display control unit 234. For example, when the flight information generating unit 232 generates flight information to be transmitted to the hovering camera 100, the display control unit 234 executes control to display on the display unit 210 an overview diagram of the structure to be inspected (bridge 1) and the generated flight information.

[0076] The storage unit 240 stores various information. Information stored in the storage unit 240 includes, for example, information about the structure (bridge 1) to be inspected. The information about the structure to be inspected may include, for example, an overview map of the structure to be inspected (bridge 1) and a construction method of the structure to be inspected. Furthermore, if locations of the structure to be inspected that are thought to be prone to damage are known in advance, the information about the structure to be inspected may include information about the parts that are thought to be highly likely to be damaged.

[0077] The control terminal 200 may receive information about the structure (bridge 1) to be inspected, for example, from the information processing device 300, when inspecting the structure, without having to store the information in advance in the storage unit 240.

[0078] The control terminal 200 according to one embodiment of the present disclosure has a configuration as shown in FIG. 4, and thereby generates flight information to be transmitted to the hovering camera 100 based on information regarding the structure (bridge 1) to be inspected, and can acquire images captured based on the flight information by the hovering camera 100, which flies based on the flight information.

[0079] An example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure has been described above with reference to Fig. 4. Next, an example of the operation of the inspection system 10 according to an embodiment of the present disclosure will be described.

[0080] [1.4. Example of operation] Fig. 5 is a flow chart showing an example of operation of the inspection system 10 according to an embodiment of the present disclosure. Fig. 5 shows an example of operation of the inspection system 10 according to an embodiment of the present disclosure when inspecting the bridge 1 by flying the hovering camera 100 and having the hovering camera 100 capture an image of the bridge 1. It is assumed that the wireless relay node 400 and the position estimation node 500 are installed in advance at appropriate positions on the bridge 1 when inspecting the bridge 1 using the hovering camera 100. Hereinafter, an example of operation of the inspection system 10 according to an embodiment of the present disclosure will be described with reference to Fig. 5.

[0081] The control terminal 200, which generates flight information for the hovering camera 100, reads information about the bridge 1, including a general view of the bridge 1 to be inspected, and displays the general view of the bridge 1 on the display unit 210 (step S101). The reading of the information about the bridge 1 is performed, for example, by the flight information generating unit 232, and the display of the general view of the bridge 1 on the display unit 210 is performed, for example, by the display control unit 234. The control terminal 200, which is displaying a general view of the bridge 1 on the display unit 210, prompts the user to specify an area of ​​the bridge 1 to be inspected, using the general view of the bridge 1 displayed on the display unit 210 (step S102). The process of prompting the user to specify in step S102 is executed by, for example, the flight information generating unit 232.

[0082] For example, when a portion of the bridge 1 is to be inspected, the control terminal 200 allows the user to specify the area to be inspected in the overview map of the bridge 1 displayed on the display unit 210. Also, when the entire bridge 1 is to be inspected, the control terminal 200 allows the user to specify the area of ​​all parts of the bridge 1 in the overview map of the bridge 1 displayed on the display unit 210.

[0083] FIG. 6 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 6 shows an example of a screen displayed on the display unit 210 when the user is prompted to specify the area of ​​the bridge 1 to be inspected in step S102. In FIG. 6, a screen is displayed on the display unit 210 when a bridge girder is specified as the area of ​​the bridge 1 to be inspected. The control terminal 200 has, for example, a touch panel as an input unit (not shown), and can allow the user to specify the area of ​​the bridge 1 by, for example, tracing the screen or selecting a span to be inspected. Of course, the method of allowing the user to specify the area of ​​the bridge 1 to be inspected is not limited to the example. Moreover, the display of the area specified by the user is not limited to the example shown in FIG. 6.

[0084] 6 also shows a state in which a mark B1 indicating the position of the base station 600 is displayed superimposed on the general situation map of the bridge 1. As described above, the base station 600 is equipped with a GPS receiver and can calculate its current position by receiving radio waves from GPS satellites. Therefore, the control terminal 200 can display the mark B1 indicating the position of the base station 600 superimposed on the general situation map of the bridge 1 based on the information on the current position calculated by the base station 600.

[0085] After the user specifies the area of ​​the bridge 1 to be inspected, the control terminal 200 then generates flight information of the hovering camera 100 in the area to be inspected specified by the user based on information about the bridge 1 (step S103). The flight information generation process in step S103 is executed by, for example, the flight information generation unit 232.

[0086] When generating flight information of the hovering camera 100 in step S103, the control terminal 200 uses information on the structure of the bridge 1, such as the construction method, width, span length, etc., the flight time of the hovering camera 100, and the inspection method of the bridge 1. For example, when a T-girder is used as the construction method of the bridge 1, the control terminal 200 generates a flight path as flight information in which the hovering camera 100 repeatedly rises and falls on the bottom side of the bridge 1. Furthermore, when generating flight information of the hovering camera 100 in step S103, the control terminal 200 may use information on the imaged surface of the bridge 1. For example, when the user selects to image the side of the bridge 1, the control terminal 200 generates a flight path along the side of the bridge 1 as flight information, and when the user selects to image the bottom of the bridge 1, the control terminal 200 generates a flight path that goes back and forth on the bottom side of the bridge 1 as flight information.

[0087] A description will be given of an example of flight information generated by the control terminal 200. The flight information may specify, for example, a list for each position where the image capturing process is to be performed in the following format. ID: (relative coordinates of the imaging point, imaging direction, imaging speed, travel time to the next imaging point, etc.) The relative coordinates of the imaging point are specified by three points on the X-axis, Y-axis, and Z-axis. The X-axis is the latitude direction, the Y-axis is the longitude direction, and the Z-axis is the height direction. Other information may include, for example, information for controlling special imaging. Information for controlling special imaging may include, for example, information for controlling multiple imaging at the same position. The information may include information on the imaging direction, information on parameters for bracketing imaging (imaging at the same position and imaging direction with different exposures, shutter speeds, ISO sensitivities, etc.), and information on the infrared wavelength at the time of imaging. According to this format, the flight information generated by the control terminal 200 may be configured as a list of values ​​such as the following: 0: (0,0,0,0,0,2,1.0) 1:(5,0,0,0,0,2,1.0) 2:(7,0,0,0,0,2,1.0) 3:(9,0,0,0,0,2,1.0) The imaging point included in the flight information generated by the control terminal 200 may be specified by relative coordinates from the reference point, for example, the absolute coordinates of the base station 600 or the absolute coordinates of an arbitrary position such as the first imaging position. The hovering camera 100 may convert the relative coordinates from the absolute coordinates of the reference point into absolute coordinates and refer to the converted coordinates during flight. Also, the imaging point included in the flight information generated by the control terminal 200 may be specified by absolute coordinates instead of relative coordinates. Also, a predetermined value may be stored in the information for controlling special imaging included in the flight information generated by the control terminal 200. For example, the information for controlling special imaging may store values ​​such as 1: imaging in multiple imaging directions, 2: bracket imaging (change in shutter speed), 3: bracket imaging (change in ISO sensitivity), etc. The control terminal 200 may include information for controlling special imaging in the flight information for a location that is considered to be prone to damage in the bridge girder 3, which is stored in the storage unit 240, for example.

[0088] During the flight information generation process in step S103, the control terminal 200 may generate flight information that causes the hovering camera 100 to capture images of the underside of the bridge girder 3 of the bridge 1 at equal intervals. Therefore, during the flight information generation process in step S103, the control terminal 200 may generate flight information such that the imaging positions of the still images are equal intervals.

[0089] When generating flight information for the hovering camera 100 in step S103, if information on the part considered to be highly likely to be damaged is stored in advance in the storage unit 140, the control terminal 200 may read out the stored information and generate flight information that causes the hovering camera 100 to capture a detailed image of the part. When causing the hovering camera 100 to capture an image of the part considered to be highly likely to be damaged, the control terminal 200 may include information for controlling the above-mentioned special image capture in the flight information. Of course, the information on the part considered to be highly likely to be damaged does not have to be stored in advance in the storage unit 140, and in that case, the user may input information on the part considered to be highly likely to be damaged at the time of inspection.

[0090] When flying the hovering camera 100 over an area of ​​the bridge 1 to be inspected, it may be impossible to fly the hovering camera 100 over the area at once, depending on the flight time of the hovering camera 100. The flight time of the hovering camera 100 can be calculated in advance from the capacity of the battery 150, the power consumption of the motors 108a-108d for moving the rotors 104a-d, the power consumption of the imaging device 101, the control unit 110, and the communication unit 120, etc. When generating flight information, it is also possible to estimate the time required for one inspection flight of the hovering camera 100 from the planned travel time from the start position (e.g., the base station 600) to the first imaging point, the planned travel time between the imaging points, the planned travel time from the last imaging point to return to the start position, etc. Therefore, if the flight route for the area of ​​the bridge 1 to be inspected cannot be flown by the hovering camera 100 in one inspection flight, the control terminal 200 may divide the generated flight route into several parts.

[0091] Furthermore, when generating flight information of the hovering camera 100 in step S103, the control terminal 200 may generate a plurality of flight paths and display the flight paths on the display unit 210. FIG. 7 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. 7 shows an example of a state in which multiple flight routes are generated when flight information of the hovering camera 100 is generated in S103 above, and the flight routes R1 and R2 are displayed on the display unit 210. The control terminal 200 displays the multiple flight routes on the display unit 210 and allows the user to select one of the flight routes. The control terminal 200 generates flight information based on the user's selection of the flight route.

[0092] After generating the flight information of the hovering camera 100 in step S103, the control terminal 200 subsequently transmits the generated flight information to the hovering camera 100 and transmits a takeoff instruction to the hovering camera 100 (step S104). The transmission of the generated flight information and the takeoff instruction is executed by the flight information generating unit 232 via the communication unit 220, for example.

[0093] FIG. 8 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 8 shows an example of a screen displayed on the display unit 210 of the control terminal 200 when a takeoff instruction is transmitted to the hovering camera 100. The user can transmit a takeoff instruction from the control terminal 200 to the hovering camera 100 by touching a takeoff instruction button 211 displayed on the display unit 210. When transmitting a takeoff instruction from the control terminal 200 to the hovering camera 100, the flight information generated in the above step S103 may be transmitted from the control terminal 200 to the hovering camera 100 before the takeoff instruction, or the flight information generated in the above step S103 may be transmitted from the control terminal 200 to the hovering camera 100 after transmitting a takeoff instruction from the control terminal 200 to the hovering camera 100.

[0094] The hovering camera 100 receives flight information and a takeoff instruction from the control terminal 200 and takes off from the base station 600, and flies based on the flight information sent from the control terminal 200, performs imaging processing to obtain a still image (step S105). The hovering camera 100 acquires position information when performing imaging processing to obtain a still image and information on the aircraft at the time of imaging processing, and links the information to the still image. The information on the aircraft at the time of imaging processing may include, for example, yaw angle, pitch angle, acceleration, and angular velocity information. The hovering camera 100 may also transmit, to the control terminal 200, streaming video images captured by the imaging device 101 during flight. The control terminal 200 acquires and displays, by streaming, video images captured by the imaging device during flight, of the hovering camera 100, so that the control terminal 200 can present to the user where the hovering camera 100 is flying.

[0095] When performing imaging processing, it is desirable for the hovering camera 100 to maintain a constant distance from the imaging target surface (for example, the side or bottom surface of the bridge girder 3) at all imaging points. By maintaining a constant distance from the imaging target surface at all imaging points, the hovering camera 100 can obtain still images captured at the same size.

[0096] When a part considered to be highly likely to be damaged is included in the flight path of the hovering camera 100, the hovering camera 100 may capture multiple still images of that part by changing the imaging direction of the imaging device, applying infrared rays with different wavelengths, or changing the shutter speed. Also, when a part considered to be highly likely to be damaged is included in the flight path of the hovering camera 100, the hovering camera 100 may narrow the intervals between positions for performing imaging processing for that part compared to other parts.

[0097] 9 is an explanatory diagram conceptually showing the operation of the hovering camera 100 in the inspection system 10 according to one embodiment of the present disclosure. When the hovering camera 100 flies over the underside of the bridge 1 based on flight information, the hovering camera 100 stops, for example, at time t1 to capture an image of the underside of the bridge 1, flies to a position for capturing an image at time t2 after capturing the image, stops at time t2 to capture an image of the underside of the bridge 1 at a different position, and thereafter repeats the flying, stopping, and capturing images until time tn in order to capture an image of the underside of the bridge 1. When the hovering camera 100 flies and stops By repeating the imaging, an image of the bottom surface of bridge 1 can be obtained.

[0098] When flying based on flight information, the hovering camera 100 can accurately determine its current position if it can receive radio waves from GPS satellites without any interference. However, in a location where it is difficult to receive radio waves from GPS satellites, such as under a bridge 1, it becomes difficult for the hovering camera 100 to accurately determine its current position. Therefore, in this embodiment, by using a position estimation node 500, the hovering camera 100 can accurately determine its current position even in a location where it is difficult to receive radio waves from GPS satellites.

[0099] Fig. 10 is an explanatory diagram conceptually illustrating the operation of the hovering camera 100 in the inspection system 10 according to an embodiment of the present disclosure. For example, when a route is set such that the hovering camera 100 flies from Start to Goal in Fig. 10, the hovering camera 100 travels between a GPS positioning area 40 where the hovering camera 100 receives radio waves from a GPS satellite 30 without interference to determine its position, and a sensor positioning area 50 where the current position is estimated using, for example, a vision sensor.

[0100] In a GPS positioning area 40, the hovering camera 100 grasps its current position using radio waves from a GPS satellite 30. In a sensor positioning area 50, the hovering camera 100 grasps its position between position estimation nodes 500, and if the position estimation node 500 is an AR marker, it grasps its current position using an integrated value of a sensor (e.g., an IMU sensor) provided in the hovering camera 100 and a distance to the destination position estimation node 500 calculated from an image captured by the imaging device 101. If the position estimation node 500 is a GPS signal transmitter, the hovering camera 100 grasps each of them using a signal transmitted from the position estimation node 500.

[0101] By using the position estimation node 500 in this manner, the hovering camera 100 can determine its exact current position even when it moves to a location where it is difficult to receive radio waves from a GPS satellite.

[0102] When the hovering camera 100 completes the imaging process at the last imaging point, it automatically flies to the base station 600 and returns to the base station 600 (step S106). Then, the control terminal 200 acquires from the hovering camera 100 the images captured by the hovering camera 100 that has returned to the base station 600 (step S107). Note that the images captured by the hovering camera 100 may be acquired after the hovering camera 100 returns to the base station 600 in this manner, but the control terminal 200 may successively acquire still images each time the hovering camera 100 executes imaging process to obtain the still images.

[0103] In an inspection system 10 according to one embodiment of the present disclosure, the hovering camera 100 and the control terminal 200 perform operations as shown in FIG. 5, so that flight information to be transmitted to the hovering camera 100 is generated by the control terminal 200 based on information about the structure (bridge 1) to be inspected, the hovering camera 100 flies based on the flight information, images are captured based on the flight information, and the images captured by the hovering camera 100 are acquired by the control terminal 200.

[0104] It is assumed that while the hovering camera 100 is flying, the user finds a part that he or she wishes to capture in detail while viewing the video captured by the hovering camera 100. In this case, for example, the user may operate the control terminal 200 to cause the control terminal 200 to send an instruction to the hovering camera 100 to stop automatic flight and switch to manual operation.

[0105] In the above example, flight information is generated by the control terminal 200, and the hovering camera 100 automatically flies based on the generated flight information to perform image capture processing. However, it is possible that an obstacle that cannot be seen in the overview diagram of the bridge 1 exists on the flight path.

[0106] Fig. 11 is an explanatory diagram conceptually illustrating the operation of the hovering camera 100 in the inspection system 10 according to an embodiment of the present disclosure. Fig. 11 shows a state in which a tree 4 is growing under a bridge girder 3. This tree 4 is an obstacle that does not appear in the overview diagram of the bridge 1, and its presence may become apparent for the first time when the hovering camera 100 flies.

[0107] Therefore, in this embodiment, the hovering camera 100 may be test-flew once based on the flight information generated by the control terminal 200 to check whether there are any obstacles on the flight path included in the flight information.

[0108] When the hovering camera 100 is test-flew once based on the flight information generated by the control terminal 200, the moving image captured by the hovering camera 100 may be received by streaming on the control terminal 200, and the user may check whether there is an obstacle on the flight path included in the flight information while watching the moving image, or the sensor unit 130 of the hovering camera 100 may detect the obstacle. The detailed position of the obstacle can be grasped by providing a stereo camera as the imaging device 101 of the hovering camera 100, grasping the distance to the obstacle by imaging with the stereo camera, or identifying the direction of the obstacle by the orientation of the hovering camera 100. Note that, if an obstacle exists on the flight path during the test flight of the hovering camera 100, the hovering camera 100 may stop the automatic flight and transition to a hovering state and wait for an operation from the user, or may automatically return to the base station 600.

[0109] When it is found that an obstacle exists on the flight path included in the flight information, the control terminal 200 may register the location of the obstacle in the overview map of the bridge 1. The location of the obstacle may be manually input by the user, or when the hovering camera 100 detects an obstacle by the sensor unit 130, the location of the detected obstacle may be acquired from the hovering camera 100 and the location of the obstacle may be registered in the overview map of the bridge 1.

[0110] Fig. 12 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. Fig. 12 shows an example of a screen displayed on the display unit 210 when the test flight of the hovering camera 100 reveals that an obstacle exists on the flight path. If the test flight of the hovering camera 100 reveals that an obstacle exists on the flight path, the control terminal 200 displays a mark O1 indicating the location of the obstacle superimposed on the overview map of the bridge 1.

[0111] Once the location of the obstacle is known, the control terminal 200 regenerates flight information so as to create a flight path that avoids the location of the obstacle, and transmits the regenerated flight information to the hovering camera 100. The hovering camera 100 flies based on the flight information regenerated by the control terminal 200, and can fly and capture images while avoiding the obstacle (tree 4).

[0112] The method of determining the location of an obstacle by flying the hovering camera 100 is not limited to the above example. For example, the hovering camera 100 may fly along a simple route around the periphery of the flight route generated by the control terminal 200 while capturing moving images with the imaging device 101, to check whether there is an obstacle under the bridge girder 3.

[0113] [1.5. Example of damage data generation] By flying the hovering camera 100 and having the hovering camera 100 capture the image of the bridge 1, it is possible to grasp the state of places that workers cannot easily approach, such as the bottom surface of the bridge girder 3. The still image captured by the hovering camera 100 is associated with, for example, the position information of the hovering camera 100 that captured the still image (which may include positioning by GPS or positioning by using the position estimation node 500), the aircraft information at the time of capturing the image (for example, yaw angle, pitch angle, acceleration, angular velocity), and information on the capturing direction. In addition, the hovering camera 100 captures the image while maintaining a constant distance from the surface to be captured at all capturing points, so that the relative position of the location where the damage occurs in the image can be known. Therefore, if the still image captured by the hovering camera 100 includes a damaged portion of the bridge girder 3, it is possible to grasp the absolute location of the damaged portion. For example, the relative value of the damaged portion is calculated with the center of the still image as the origin, and the relative value is calculated to the position information of the hovering camera 100 at the time the image was captured, so that the position information of the damaged portion can be obtained. In addition, the following data, for example, can be recorded as the position information of the damaged portion. (1) Record the capture position information of the still image as the position of the damaged area (without recording a relative value (offset)). (2) Information on the imaging position of the still image and a relative value (offset) corresponding to the damaged portion are recorded as the position of the damaged portion. (3) The reference absolute value (for example, as described below, the capture positions of still images of the four corners where the accuracy of position information is considered to be high, or the coordinates of the position estimation node 500) and the relative value (offset) are recorded as the position of the damaged part. (4) The calculated absolute values ​​(e.g., latitude, longitude, altitude) are recorded as the location of the damaged part.

[0114] A technique is known for determining the physical size of the imaging range by using numerical values ​​such as the focal length of the lens, the size of the image sensor, and the distance to the imaging target. Therefore, when grasping the damaged part, the physical size of the imaging range of the hovering camera 100 may be estimated using distance information from the hovering camera 100 to the imaging target (for example, the back side or side of the bridge girder 3) and angle of view information of the imaging device 101. The center position of the captured image (the position where the hovering camera 100 captured the image) is set as the origin, and the physical relative position from the origin to the damaged part is estimated, and the position coordinate of the origin of the captured image is added to this relative position, thereby determining the physical position information of the damaged part. Note that the distance information and angle of view information may be obtained via a sensor possessed by the hovering camera 100 at the time of imaging and recorded in association with the image, or values ​​previously set in the hovering camera 100 or the imaging device 101 may be used. Furthermore, the position information of the damaged portion may be calculated using aircraft information at the time of imaging (for example, yaw angle, pitch angle, acceleration, angular velocity) and imaging direction information in addition to imaging position information, distance information, and angle of view information.

[0115] The detection of damaged parts from the still images captured by the hovering camera 100 may be performed by the user himself / herself by visually inspecting the damaged parts, or may be performed automatically by image processing, for example, by the information processing device 300. When automating the detection of damaged parts, for example, an image processing technique such as pattern matching processing may be used.

[0116] The data structure of the damage data is defined, for example, in the following format. (Image ID, damage ID, location information of the damaged part, coordinates of the damaged part on the image, damage type ID, degree of damage) The damage type ID refers to an ID given to each type of damage, such as cracks, peeling, water leakage, free lime, etc. In addition, the maximum width of the damage, the length of the damaged part in the image, etc. can be recorded in the damage degree field. The inspection system 10 according to this embodiment can generate damage data according to the above-mentioned format from the still image captured by the hovering camera 100, either manually input by the user or automatically processed by the information processing device 300. The damage data generated by the inspection system 10 according to this embodiment can be used in ordering to a construction company to repair the damage caused to the bridge 1.

[0117] However, the hovering camera 100 captures many still images during one inspection flight, and therefore it is a heavy burden on the user to check each still image captured by the hovering camera 100 during an inspection flight.

[0118] Therefore, the still images captured by the hovering camera 100 are joined (stitched) together to obtain a single image. By stitching the still images captured by the hovering camera 100, for example, the state of the bottom surface of one span of the bridge girder 3 can be obtained as a single image. Then, by checking the image of the bottom surface of one span of the bridge girder 3 obtained by stitching the still images captured by the hovering camera 100, the user can check whether there is any damage to the bottom surface of the bridge girder 3. The stitching process of the still images may be performed by the control terminal 200 or the information processing device 300.

[0119] 13 is an explanatory diagram showing an overview of inspecting the bottom surface of a bridge girder 3 based on still images captured by a hovering camera 100. A part of the bottom surface of the bridge girder 3 (for example, the length of one span of the bridge girder 3) is captured by the hovering camera 100, and the still images captured by the hovering camera 100 are stitched together to obtain a single image 20 of the bottom surface of the bridge girder 3. Note that reference numeral 21 indicates an image captured by the hovering camera 100 in one imaging process.

[0120] When determining the absolute location of the damaged part from an image obtained by stitching still images captured by the hovering camera 100, position information of the stitched images with relatively high accuracy of position information at the time of capture can be selected as a reference point. Position information of the hovering camera 100 at the time of capture of the still images of the four corners that are the basis of the stitched image may be used as a reference point. The still images of the four corners that are the basis of the stitched image have the least distortion, and the GPS positioning area has less error in position information, and it is considered preferable to use the position information at the time of capture of the four corners close to the GPS positioning area as a reference point. Therefore, by determining the absolute location of the damaged part from the position information of the hovering camera 100 corresponding to the still images of the four corners, it is possible to accurately determine the position of the damaged part. Note that, for example, positioning status information in the GPS positioning data (information indicating a state of 2D positioning, 3D positioning, or positioning not possible, data such as the number of received satellites, etc.) may be used as the accuracy of each position information.

[0121] 14 is an explanatory diagram showing an example of image 20 obtained by stitching still images captured by hovering camera 100. Centers G1 to G4 of still images C1 to C4 at the four corners that form the basis of image 20 correspond to the positions of the hovering camera 100 when each still image was captured. In this embodiment, the absolute positions of each damaged portion in image 20 are calculated using position information of the hovering camera 100 that corresponds to the still images C1 to C4 at the four corners.

[0122] When generating damage data from stitched images, the data configuration of the damage data is defined, for example, in the following format: That is, the image ID is deleted from the above-mentioned damage data. (Damage ID, location information of the damaged part, coordinates of the damaged part on the image, damage type ID, degree of damage) Alternatively, an image ID of the stitched image may be generated and included in the damage data. The inspection system 10 according to the present embodiment can generate damage data according to the above-mentioned format from the stitched image by manual input by the user or automatic processing by the information processing device 300.

[0123] [1.5.1. Example of functional configuration] FIG. 15 is an explanatory diagram illustrating an example of a functional configuration of an information processing device 300 according to an embodiment of the present disclosure. 15 shows an example of a functional configuration of an information processing device 300 according to an embodiment of the present disclosure, which has a function of determining an absolute damaged position of the bridge girder 3 from a still image captured by the hovering camera 100 and generating damage data. Hereinafter, a functional configuration example of the information processing device 300 according to an embodiment of the present disclosure will be described with reference to FIG. 15.

[0124] As shown in FIG. 15, an information processing device 300 according to an embodiment of the present disclosure includes a display unit 310, a communication unit 320, a control unit 330, and a storage unit 340.

[0125] The display unit 310 is composed of a flat panel display device such as a liquid crystal display device, an organic EL display device, etc. The display unit 310 can display, for example, an image captured by the imaging device 101 of the hovering camera 100, information about damage to the bridge 1 obtained from the image captured by the imaging device 101, etc.

[0126] The communication unit 320 transmits and receives information via wireless communication, for example, with the control terminal 200. The information processing device 300 receives, via the communication unit 320, an image captured by the hovering camera 100 from the control terminal 200 together with information on the absolute imaging position of the image.

[0127] The control unit 330 controls the operation of the information processing device 300. For example, the control unit 330 can control the display processing of characters, figures, images, and other information on the display unit 310, and the transmission and reception processing of information with other devices (for example, the control terminal 200) via the communication unit 320. The control unit 330 is also configured to include an imaging position information acquisition unit 332, a damage position calculation unit 334, an image synthesis unit 336, and a damage data generation unit 338.

[0128] The imaging position information acquisition unit 332 acquires information on the imaging position at the time of imaging, which is acquired by the hovering camera 100 when the hovering camera 100 images the bridge 1. The damage position calculation unit 334 detects damaged parts of the bridge 1 from the image captured by the hovering camera 100, for example, by using an image processing technique such as pattern matching processing, and calculates the absolute position of the damaged parts by using the imaging position information acquired by the imaging position information acquisition unit 332.

[0129] The image synthesis unit 336 executes image stitching processing to generate one image by connecting still images captured by the hovering camera 100. When stitching the still images captured by the hovering camera 100, the image synthesis unit 336 may use information on the imaging positions at the time of capturing each still image.

[0130] When calculating the damage position, the damage position calculation unit 334 may use information on the imaging positions of the captured images at the corners (e.g., each of the four corners) in the captured images that are the basis of the image stitched by the image synthesis unit 336. As described above, it is considered that the still images at the four corners in the captured images that are the basis of the stitched image have the least distortion, so the damage position calculation unit 334 can find a more accurate damage position by using information on the imaging positions of the captured images at the corners in the captured images that are the basis of the stitched image.

[0131] The damage data generation unit 338 generates the above-mentioned damage data using the absolute position of the damaged portion of the bridge 1 calculated by the damage position calculation unit 334. The damage data generation unit 338 may generate damage data for each still image, or may generate one piece of damage data for the images stitched by the image synthesis unit 336.

[0132] The storage unit 340 stores various information. The information stored in the storage unit 340 may include, for example, information on a still image captured by the imaging device 101 of the hovering camera 100 and the absolute imaging position of the hovering camera 100 when the still image was captured, and information on damage data generated by the damage data generation unit 338.

[0133] The information processing device 300 according to an embodiment of the present disclosure has a configuration as shown in FIG. 15, and thus it is possible to generate damage data from a still image captured by the hovering camera 100. Therefore, the information processing device 300 according to an embodiment of the present disclosure can efficiently generate an inspection result of the bridge 1, which is a structure to be inspected. As described above, the damage data may be generated by the control terminal 200, not the information processing device 300. Therefore, the configuration of the control unit 330 of the information processing device 300 shown in FIG. 15 may be included in the control terminal 200. Furthermore, the inspection result of the bridge 1, which is a structure to be inspected, can be accumulated in a public or private database and utilized. As described above, the damage data may be generated by the control terminal 200, not the information processing device 300. Therefore, the configuration of the control unit 330 of the information processing device 300 shown in FIG. 15 may be included in the control terminal 200.

[0134] An example of the functional configuration of the information processing device 300 according to an embodiment of the present disclosure has been described above with reference to Fig. 15. Next, an example of the operation of the information processing device 300 according to an embodiment of the present disclosure will be described.

[0135] [1.5.2. Example of operation] Fig. 16 is a flow chart showing an operation example of the information processing device 300 according to an embodiment of the present disclosure. Fig. 16 shows an operation example of the information processing device 300 according to an embodiment of the present disclosure when obtaining an absolute damage position of the bridge girder 3 from a still image captured by the hovering camera 100 and generating damage data. Hereinafter, an operation example of the information processing device 300 according to an embodiment of the present disclosure will be described with reference to Fig. 16.

[0136] The information processing device 300 first acquires an image associated with information on the image capturing position captured by the hovering camera 100 while flying around the edge of the bridge 1 (step S301). After acquiring the image associated with the information on the image capturing position in step S301, the information processing device 300 subsequently detects a damaged portion from the image using an image processing technique such as a pattern matching process (step S302). The detection process of the damaged portion in step S302 may be executed by the damage position calculation unit 334, for example.

[0137] When the damaged portion is detected from the image in the above step S302, the information processing device 300 subsequently calculates the absolute position of the damaged portion (step S303). The calculation process in step S303 may be executed by, for example, the damage position calculation unit 334. The information processing device 300 calculates the absolute position of the damaged portion in step S303 based on the information of the imaging position of the still image captured by the hovering camera 100. When calculating the absolute position of the damaged portion, the information processing device 300 may estimate the physical size of the imaging range of the hovering camera 100 based on the distance information from the hovering camera 100 to the imaging target (for example, the back surface or side surface of the bridge girder 3) and the field angle information of the imaging device 101. The information processing device 300 can determine the physical position information of the damaged portion by estimating the physical relative position from the center of the captured image to the damaged portion and adding the position coordinate as the origin of the captured image to this relative position. Then, the information processing device 300 generates damage data including the absolute position of the damaged portion (step S304). The damage data generating process in step S304 can be executed by the damage data generating unit 338, for example.

[0138] The information processing device 300 according to an embodiment of the present disclosure can generate damage data from a still image captured by the hovering camera 100 by performing the operation shown in Fig. 16, and therefore the information processing device 300 according to an embodiment of the present disclosure can efficiently generate an inspection result of the bridge 1, which is a structure to be inspected. As described above, the damage data may be generated by the control terminal 200, not by the information processing device 300. Therefore, the operation shown in Fig. 16 may be executed by the control terminal 200.

[0139] Fig. 17 is a flow chart showing an example of the operation of the control terminal 200 according to an embodiment of the present disclosure. Fig. 17 shows an example of a process for generating flight information by the control terminal 200 using damage data generated by the information processing device 300. Hereinafter, an example of the operation of the control terminal 200 according to an embodiment of the present disclosure will be described with reference to Fig. 17.

[0140] When the control terminal 200 displays a still image captured by the hovering camera 100 and the location of damage is specified by the user on the still image (step S311), the control terminal 200 acquires the damage position of the specified location from the damage data generated by the information processing device 300 (step S312). There is no restriction on the method of specifying the location of damage, but for example, a still image may be displayed and the user may specify the location of damage by touching the touch panel of the control terminal 200 with a finger or the like.

[0141] When the damage position at the location specified by the user is acquired from the damage data, the control terminal 200 then generates flight information for flying the hovering camera 100 to the damage position acquired from the damage data and capturing an image of the damage position (step S313). The process of step S313 is executed, for example, by the flight information generating unit 232. The flight information generated by the control terminal 200 in this step S313 is for checking the damage position in detail, and therefore may instruct the hovering camera 100 to narrow the interval between the image capturing positions or to perform the above-mentioned special image capturing at each image capturing position, as compared with the flight information generated by the control terminal 200 described in the description of FIG. 5 above. When performing special image capturing, the flight information includes information for controlling the special image capturing, as described above.

[0142] When the flight information is generated in step S313, the control terminal 200 transmits the generated flight information to the hovering camera 100, and the hovering camera 100 performs flight and imaging processing based on the flight information, as shown in steps S104 and S105 of Fig. 5. Then, when the hovering camera 100 completes the imaging processing at the last imaging point, it automatically flies to the base station 600 and returns to the base station 600, and the control terminal 200 acquires the image captured by the hovering camera 100 from the hovering camera 100, as shown in steps S106 and S107 of Fig. 5.

[0143] By executing operations such as those shown in FIG. 17, the control terminal 200 according to one embodiment of the present disclosure can use the damage data generated by the information processing device 300 to generate flight information for causing the hovering camera 100 to capture detailed images of the damaged portion of the bridge girder 3.

[0144] <2. Summary> As described above, one embodiment of the present disclosure provides a hovering camera 100 that flies automatically based on set flight information and captures images of a structure to be inspected, and an inspection system 10 that can check the damage state of the structure based on still images captured by the hovering camera 100.

[0145] The inspection system 10 according to an embodiment of the present disclosure uses information about the structure to be inspected when the control terminal 200 generates flight information to be transmitted to the hovering camera 100. By using the information about the structure to be inspected, the control terminal 200 can generate flight information for flying the hovering camera 100 to efficiently inspect the structure to be inspected.

[0146] In the above embodiment, the image captured by the hovering camera 100 is a still image, and an example of the inspection system 10 in which the still image is used to inspect the damage state of the bridge 1 is shown, but the present disclosure is not limited to this example. The hovering camera 100 may capture the bridge 1 as a moving image while flying, and the information processing device 300 may generate damage data using the moving image captured by the hovering camera 100. The hovering camera 100 periodically acquires position information when performing the imaging process of the moving image, and by linking the imaging time of the moving image with the acquisition time of the position information, the information processing device 300 can generate damage data using the moving image.

[0147] The steps in the process performed by each device in this specification do not necessarily have to be processed in chronological order according to the order described in the sequence diagram or flowchart. For example, the steps in the process performed by each device may be processed in an order different from the order described in the flowchart, or may be processed in parallel.

[0148] It is also possible to create a computer program for causing hardware such as a CPU, ROM, and RAM built into each device to perform functions equivalent to those of the configuration of each device described above. It is also possible to provide a storage medium storing the computer program. It is also possible to realize a series of processes by hardware or hardware circuits by configuring each functional block shown in the functional block diagram with hardware or hardware circuits. A part or all of each functional block shown in the functional block diagram used in the above description may be realized by a server device connected via a network such as the Internet. The configuration of each functional block shown in the functional block diagram used in the above description may be realized by a single device or may be realized by a system in which multiple devices cooperate with each other. A system in which multiple devices cooperate with each other may include, for example, a combination of multiple server devices, a combination of a server device and a terminal device, etc.

[0149] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0150] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0151] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An acquisition unit that acquires information regarding an overview of a structure; A flight information generating unit that generates flight information of an aircraft that flies around the periphery of the structure and captures the structure based on the information acquired by the acquiring unit; A control device comprising: (2) The control device described in (1), wherein when an obstacle is detected while the aircraft is flying based on the flight information, the flight information generation unit regenerates flight information to avoid the obstacle. (3) The control device described in (2), wherein when an obstacle is detected while the aircraft is flying based on the flight information, the flight information generating unit presents information regarding the general condition of the structure including information regarding the position of the obstacle. (4) The control device according to any one of (1) to (3), wherein the flight information generating unit generates information as the flight information including information on the flight path of the aircraft and the position where the aircraft performs the imaging process. (5) The control device described in (4), wherein the flight information generation unit generates information as the flight information including information on the imaging direction at the position where the aircraft performs the imaging process. (6) The control device described in (4) or (5), wherein the flight information generation unit generates information as the flight information including information for controlling special imaging at a position where the aircraft performs imaging processing. (7) The control device according to any one of (1) to (6), wherein the structure is a bridge. (8) The control device described in (7), wherein the flight information generation unit generates flight information for capturing an image of the bottom surface of the bridge. (9) The control device described in (7), wherein the flight information generation unit generates flight information that captures an image of the side of the bridge. (10) The control device according to any one of (7) to (9), wherein the flight information generating unit generates flight information based on a construction method of the bridge. (11) The control device according to any one of (7) to (10), wherein the flight information generating unit generates information including information for controlling special imaging based on information about the bridge. (12) The control device according to (11), wherein the information about the bridge is information about locations on the bridge that are prone to damage. (13) A control unit that controls the drone to fly around the periphery of the structure based on flight information generated based on information about the general condition of the structure; an imaging unit that performs imaging processing of the structure while flying based on the flight information; An imaging device comprising: (14) Obtaining information regarding the general condition of the structure; generating flight information for an aircraft that flies around the periphery of the structure and captures an image of the structure based on the acquired information; A control method comprising: (15) Controlling the aircraft to fly around the periphery of the structure based on flight information generated based on information about the general condition of the structure; Executing an imaging process of the structure while flying based on the flight information; An imaging method comprising: (16) On the computer, Obtaining information regarding the general condition of the structure; generating flight information for an aircraft that flies around the periphery of the structure and captures an image of the structure based on the acquired information; A computer program that executes the following: (17) On the computer, Controlling the aircraft to fly around the periphery of the structure based on flight information generated based on information about the general condition of the structure; Executing an imaging process of the structure while flying based on the flight information; A computer program that executes the following: [Explanation of symbols]

[0152] 10 Inspection System 100 Hovering Camera 101 Imaging device 104a~104d rotor 108a~108d Motor 110 Control section 120 Communications Department 130 Sensor section 132 Location information acquisition unit 140 Storage section 150 Battery 200 Control terminal 300 Information processing device 400 Wireless Relay Nodes 500 Location Estimation Nodes 600 Base Station 700 Charging Station

Claims

1. Creating flight information for a flying imaging device including setting information for a camera to be photographed; Based on data from an IMU (Inertial Measurement Unit) sensor and image data captured by the camera, the position of the flying imaging device is determined; Controlling the flight of the flying imaging device based on the flight information and the position of the flying imaging device; An information processing method that performs control to stop flight and transition to a hovering state based on information about detected obstacles that may interfere with flight.

2. creating the flight information including a flight route based on a designation of an inspection target received from a user to whom map data including the inspection target is presented; The information processing method according to claim 1 , further comprising: controlling presentation of the flight route on the map data based on the flight information including the flight route created based on the user's designation.

3. Creating the flight information based on the designation of the inspection area received from the user; The information processing method according to claim 2 , further comprising the step of: controlling presentation of the flight route on the map data based on flight information created based on the user's area designation.

4. The information processing method according to any one of claims 1 to 3, wherein, when the flying imaging device detects an obstacle while flying based on the flight information, flight information for avoiding the obstacle is regenerated.

5. The information processing method according to claim 4 , wherein when the flying imaging device detects an obstacle while flying based on the flight information, information regarding the position of the obstacle is presented together with information regarding an object imaged by the flying imaging device.

6. The information processing method according to any one of claims 1 to 5, wherein the flight information includes information on a flight path of the flying imaging device and information on a position where the flying imaging device performs imaging processing.

7. The information processing method according to claim 6 , further comprising generating, as the flight information, information including information on an imaging direction at a position where the flying imaging device performs imaging processing.

8. The information processing method according to claim 6 , further comprising generating, as the flight information, information including information for controlling special imaging at a position where the flying imaging device executes imaging processing.

9. The information processing method according to claim 5 , wherein the object is a structure.

10. The information processing method according to claim 9 , wherein the object is a bridge.

11. The information processing method according to claim 10 , further comprising generating flight information for capturing an image of the bottom surface of the bridge.

12. The information processing method according to claim 10 or 11, further comprising generating flight information for capturing an image of a side surface of the bridge.

13. The information processing method according to any one of claims 10 to 12, further comprising generating flight information based on a construction method of the bridge.

14. The information processing method according to any one of claims 10 to 13, further comprising generating information including information for controlling special imaging based on the information regarding the bridge.

15. The information processing method according to claim 14 , wherein the information relating to the bridge is information relating to locations on the bridge where damage is likely to occur.

16. A flight information generating unit that generates flight information of a flight imaging device including setting information of a camera to be photographed; An information processing device comprising: a control unit that determines a position of the flying imaging device based on data from an IMU (Inertial Measurement Unit) sensor and image data captured by the camera, controls the flight of the flying imaging device based on the flight information and the position of the flying imaging device, and controls the device to stop the flight and transition to a hovering state based on information about a detected obstacle that may interfere with the flight.

17. Creating flight information for a flying imaging device including setting information for a camera to be photographed; Determine the position of the flying imaging device based on data from an IMU (Inertial Measurement Unit) sensor and image data captured by the camera, control the flight of the flying imaging device based on the flight information and the position of the flying imaging device, and control the device to stop the flight and transition to a hovering state based on information about a detected obstacle that may interfere with the flight. A computer program that causes a computer to execute the following:

Images