Control method, control device and program

Abstract

To provide a control method, a control device and a program that can intuitively instruct a moving object to move using an image, and can detect a damaged portion of a structure to be inspected using an image captured by the moving object.SOLUTION: A control method includes: generating movement information for moving a moving object in response to an operation to designate an area for an operation image generated from a captured image; and detecting a damaged portion of a structure from an image including the structure in at least a portion of the image, which is captured by the moving object which moves based on the generated movement information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a control method, a control device, and a program. [Background technology]

[0002] There is disclosed a technology for a photographic method in which a camera is attached to a wirelessly controlled flying object and photographs are taken with the camera (see, for example, Patent Document 1, etc.). By attaching a camera to the flying object, it becomes possible to take photographs from the sky or from places where a tripod cannot be set up. Furthermore, by attaching a camera to the flying object to take photographs, various advantages are obtained, such as lower costs than when using a real airplane or helicopter, safer photography, the ability to take photographs at low altitudes or in narrow places, and the ability to approach a target and take photographs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-27448 A Summary of the Invention [Problem to be solved by the invention]

[0004] A dedicated controller is usually required to operate a moving object such as an aircraft or a robot equipped with such a camera. If a user could easily specify movement instructions for the moving object using an image captured by the moving object, even a user who is not familiar with operating a moving object would be able to easily move it to a desired position.

[0005] Therefore, this disclosure proposes a new and improved control method, control device, and program that can intuitively instruct a moving object to move using images, and can detect damaged parts of a structure to be inspected using images captured by the moving object. [Means for solving the problem]

[0006] According to the present disclosure, a control method is provided that includes generating movement information for moving a moving body in response to an operation of specifying an area for an operation image generated from an captured image, and detecting a damaged portion of the structure from an image captured by the moving body that moves based on the generated movement information and that includes the structure in at least a portion of the image.

[0007] According to the present disclosure, there is also provided a control device including: a movement information generation unit that generates movement information for moving a moving body in response to an operation of specifying an area for an operation image generated from an captured image; and a damaged portion detection unit that detects damaged portions of a structure from an image captured by the moving body that moves based on the generated movement information and that includes the structure in at least a portion of the image.

[0008] In addition, according to the present disclosure, a program is provided that causes a computer to generate movement information for moving a moving body in response to an operation of specifying an area for an operation image generated from an captured image, and detect damaged portions of the structure from an image captured by the moving body that has moved based on the generated movement information and that includes the structure in at least a portion of the image. Effect of the Invention

[0009] As described above, according to the present disclosure, there are provided new and improved control methods, control devices, and programs that are capable of intuitively instructing a moving object to move using images, and of detecting damaged portions of a structure to be inspected using images captured by the moving object.

[0010] It should be noted that the above effects are not necessarily limiting, and any of the effects shown in this specification, or other effects that can be understood from this specification, may be achieved in addition to or instead of the above effects. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an embodiment of the present disclosure. [Diagram 2] 1 is an explanatory diagram illustrating an example of a system configuration of an inspection system 10 according to an embodiment of the present disclosure. [Diagram 3] 1 is an explanatory diagram illustrating an example of a functional configuration of a hovering camera 100 according to an embodiment of the present disclosure. [Figure 4] 2 is an explanatory diagram illustrating an example of a functional configuration of a control terminal 200 according to an embodiment of the present disclosure. FIG. [Diagram 5] 4 is a flowchart showing an example of the operation of the inspection system 10 according to an embodiment of the present disclosure. [Figure 6] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 7] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 8] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 9] 1 is an explanatory diagram conceptually showing how the underside of a bridge 1 is imaged by a hovering camera 100. FIG. [Figure 10] 1 is an explanatory diagram conceptually showing the operation of a hovering camera 100 in an inspection system 10 according to an embodiment of the present disclosure. [Figure 11] 1 is an explanatory diagram conceptually showing the operation of a hovering camera 100 in an inspection system 10 according to an embodiment of the present disclosure. [Figure 12] 2 is an explanatory diagram showing an example of a screen displayed on a display unit 210 of a control terminal 200. FIG. [Figure 13] FIG. 13 is an explanatory diagram showing an overview of inspecting the bottom surface of a bridge girder 3. [Figure 14] 1 is an explanatory diagram showing an example of an image 20 obtained by stitching still images captured by a hovering camera 100. FIG. [Figure 15] 2 is an explanatory diagram illustrating an example of a functional configuration of an information processing device 300 according to an embodiment of the present disclosure. [Figure 16]4 is a flowchart showing an example of operation of the information processing device 300 according to an embodiment of the present disclosure. [Figure 17] 4 is a flowchart showing an example of the operation of the control terminal 200 according to an embodiment of the present disclosure. [Figure 18] FIG. 2 is an explanatory diagram showing a state in which the hovering camera 100 is caused to capture an image in the direction of the ground. [Figure 19] FIG. 13 is an explanatory diagram showing a state in which a user is allowed to specify a flight path for the hovering camera 100 using a composite image. [Figure 20] FIG. 1 is an explanatory diagram showing a state in which the hovering camera 100 is caused to capture an image in the upward direction (the underside of a bridge). [Figure 21] FIG. 13 is an explanatory diagram showing a state in which a user is allowed to specify a flight path for the hovering camera 100 using a composite image. [Figure 22] FIG. 13 is an explanatory diagram showing a state in which a composite image is generated and displayed on the control terminal 200. [Figure 23] FIG. 11 is an explanatory diagram for explaining a process of generating a flight path by the control terminal 200 based on a user's input for a composite image. [Figure 24] FIG. 11 is an explanatory diagram showing an example of a composite image. [Diagram 25] FIG. 11 is an explanatory diagram showing an example of a composite image. [Figure 26] 4 is a flowchart showing an example of the operation of the control terminal 200 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0013] The explanation will be given in the following order. 1. One embodiment of the present disclosure 1.1. Overview 1.2. System configuration example 1.3. Example of functional configuration 1.4. Example of operation 1.5. Example of Damage Data Generation 1.5.1 Functional configuration example 1.5.2 Example of operation 1.6. Example of flight instructions using synthetic images 2. Summary

[0014] 1. One embodiment of the present disclosure [1.1. Overview] Before describing an embodiment of the present disclosure in detail, an overview of the embodiment of the present disclosure will be first described.

[0015] In the maintenance and management of structures such as roads, bridges, tunnels, buildings, etc., human inspection of the structure's condition is essential. Normally, visual inspection of such structures is done by workers approaching the structure and visually checking for damage such as corrosion or cracks, or loosening of connecting members such as bolts, and also by conducting a tapping test to check for the presence or absence of such abnormalities.

[0016] In order to maintain bridges, especially concrete bridges, it has become necessary to erect scaffolding on the undersides of bridge piers and girders for workers to carry out visual and hammering inspections of the girders and piers, and to close some or all of the lanes to ensure the safety of the workers and to position work vehicles. These factors can cause problems not only in the cost required for inspections, but also in the cost required to arrange road guides for road closures, and further in the traffic congestion on detours that may occur due to road closures.

[0017] In addition, there are bridges where it is not easy or even possible to set up scaffolding because they are built over rivers or the sea. In light of these circumstances, there is a need for technology that enables safe, low-cost inspection of structures without affecting traffic.

[0018] In light of the above circumstances, the present inventors have considered technology that allows for safe, low-cost inspection of structures without affecting traffic, and have devised a technology that uses an aircraft equipped with an imaging device (in the following description, an aircraft equipped with an imaging device is also referred to as a "hovering camera") to allow for safe, low-cost inspection.

[0019] Fig. 1 is an explanatory diagram for explaining an outline of an embodiment of the present disclosure. Fig. 1 shows a schematic diagram of a bridge 1 constructed of concrete, for example. When inspecting the bridge 1 constructed of concrete, as described above, it has been necessary to set up scaffolding on the back surface of the pier 2 or bridge girder 3 so that workers can visually inspect whether there is any damage such as cracks or corrosion, and it has been necessary to close some or all of the lanes to ensure the safety of workers or to place work vehicles.

[0020] In one embodiment of the present disclosure, a hovering camera 100 is used when inspecting a bridge 1. The hovering camera 100 is an aircraft equipped with an imaging device, configured to fly automatically according to preset flight information (which in this embodiment includes information on a flight path and an imaging position of a still image). Note that the information on the imaging position of a still image may include, for example, a position where an imaging process is performed, an imaging direction, a travel time to a position where the next imaging process is performed, and the like.

[0021] For example, when inspecting the back side (bottom surface) of the bridge girder 3, the hovering camera 100 is automatically flown and the hovering camera 100 is caused to capture an image of the back side of the bridge girder 3. By causing the hovering camera 100 to capture an image of the back side of the bridge girder 3, it becomes unnecessary to set up scaffolding on the back side of the pier 2 or the bridge girder 3 for inspecting the bridge girder 3, or the frequency of closing lanes to traffic is reduced, or traffic closure becomes unnecessary. Also, for example, when inspecting the side (side surface) of the bridge girder 3, the hovering camera 100 is automatically flown and the hovering camera 100 is caused to capture an image of the side of the bridge girder 3. Therefore, by automatically flying the hovering camera 100 and causing the hovering camera 100 to capture an image of the back side or side of the bridge girder 3, the safety of the workers is ensured, and the bridge 1 can be inspected at low cost without affecting traffic.

[0022] In order to automatically fly the hovering camera 100 to capture an image of the underside of the bridge girder 3, it is necessary to set the flight path of the hovering camera 100 and to set information on the capture position of a still image at the underside of the bridge girder 3. In one embodiment of the present disclosure, an object is to enable efficient inspection of the bridge 1 by efficiently creating flight information to be set in the hovering camera 100 using information on the general condition of the bridge 1.

[0023] The outline of an embodiment of the present disclosure has been described above. Next, a configuration example of an inspection system according to an embodiment of the present disclosure will be described.

[0024] [1.2. System configuration example] Fig. 2 is an explanatory diagram showing a system configuration example of the inspection system 10 according to an embodiment of the present disclosure. The inspection system 10 according to an embodiment of the present disclosure shown in Fig. 2 is a system intended to efficiently inspect a structure, for example, a bridge 1. Hereinafter, a system configuration example of the inspection system 10 according to an embodiment of the present disclosure will be described with reference to Fig. 2.

[0025] As shown in FIG. 2, the inspection system 10 according to one embodiment of the present disclosure includes a hovering camera 100, a control terminal 200, an information processing device 300, a wireless relay node 400, a position estimation node 500, a base station 600, a charging station 700, and a server device 800.

[0026] The hovering camera 100 is an example of the imaging device of the present disclosure, and is an aircraft equipped with an imaging device as described above. The hovering camera 100 is an aircraft configured to automatically fly based on a specified flight path and capture still images with the imaging device at a specified imaging position. The hovering camera 100 can fly, for example, with four rotors, and can fly while ascending, descending, and moving horizontally by controlling the rotation of each rotor. Of course, the number of rotors is not limited to the example.

[0027] The flight path from the flight start position to the flight end position and the imaging position set in the hovering camera 100 are set as, for example, GPS (Global Positioning System) position information. Therefore, the hovering camera 100 may incorporate a GPS receiver that receives radio waves from a GPS satellite and calculates the current position. The flight path set in the hovering camera 100 may be set with all of the latitude, longitude, and altitude set as GPS position information, or only the latitude and longitude may be set as GPS position information, and the altitude may be set as, for example, a relative height from a base station 600 described later.

[0028] The control terminal 200 is an example of a control device of the present disclosure, and is a terminal that executes control regarding the flight of the hovering camera 100. The control terminal 200 performs control regarding the flight of the hovering camera 100, for example, generation of flight information to be sent to the hovering camera 100, instruction to the hovering camera 100 to take off, instruction to return to the base station 600 described later, and operation of the hovering camera 100 when the hovering camera 100 cannot fly automatically for some reason. The process of generating flight information of the hovering camera 100 by the control terminal 200 will be described in detail later, but an example will be briefly described here.

[0029] When generating flight information for the hovering camera 100, the control terminal 200 reads information on the general condition of the bridge 1 to be inspected, for example, a general condition map of the bridge 1 to be inspected, and displays it on the screen. Points on the general condition map of the bridge 1 are associated with points on map data containing detailed GPS information in advance. This association is preferably performed for at least two pairs of points. By associating points on map data containing detailed GPS information in advance with the general condition map of the bridge 1, the flight path of the hovering camera 100 is defined as a GPS value. Then, the control terminal 200 generates the flight path of the hovering camera 100 based on the general condition map of the bridge 1. The flight path of the hovering camera 100 is displayed superimposed on the general condition map so that the user (the worker inspecting the structure) can easily understand it.

[0030] When generating flight information for the hovering camera 100, the control terminal 200 may take into consideration the structure and dimensions of the bridge 1, and the parts of the bridge 1 to be imaged by the hovering camera 100. When generating flight information for the hovering camera 100, the control terminal 200 may generate flight information that causes the hovering camera 100 to image in detail parts that are considered to have a high possibility of being damaged.

[0031] As described above, the flight route set for the hovering camera 100 may have all of the latitude, longitude, and altitude set as GPS position information, but there may be cases where altitude data does not exist in the general situation map of the bridge 1. If altitude data does not exist in the general situation map of the bridge 1, the flight route set for the hovering camera 100 may have only the latitude and longitude set as GPS position information, and the altitude may be set as a relative height from the base station 600, for example.

[0032] When setting flight information for the hovering camera 100, the control terminal 200 desirably generates flight information that keeps a constant distance from the image-capturing surface when the hovering camera 100 captures an image of the bridge 1. By generating flight information that keeps a constant distance from the image-capturing surface when the hovering camera 100 captures an image of the bridge 1, the control terminal 200 can cause the hovering camera 100 to generate an image of the same scale.

[0033] The control terminal 200 is a portable device such as a notebook computer or tablet terminal, and transmits and receives information wirelessly to and from the hovering camera 100. The control terminal 200 may perform wireless communication with the hovering camera 100 directly, but since the hovering camera 100 may fly beyond the communication range of the control terminal 200 during inspection of a structure, particularly a bridge 1, the communication may be performed via a wireless relay node 400 that is installed during inspection.

[0034] The control terminal 200 acquires images captured by an imaging device while the hovering camera 100 is flying, and displays them as necessary. The control terminal 200 may acquire and display, by streaming, moving images captured by an imaging device while the hovering camera 100 is flying. By acquiring and displaying, by streaming, moving images captured by an imaging device while the hovering camera 100 is flying, the control terminal 200 can present to the user where the hovering camera 100 is flying.

[0035] The information processing device 300 is a device that processes various information, and may be a device having a function of processing information such as a personal computer (PC) or a game machine. In this embodiment, the information processing device 300 is a device that has a function of displaying an image captured by the hovering camera 100 in particular to allow a user to check the state of the bridge 1. The information processing device 300 also has a function of calculating an absolute damage position of the bridge girder 3 from the image captured by the hovering camera 100 and generating damage data described later. The information processing device 300 may have a function of transmitting the generated damage data to the server device 800. Note that the control terminal 200 may have a function of calculating an absolute damage position of the bridge girder 3 from the image captured by the hovering camera 100 and generating damage data described later.

[0036] The information processing device 300 acquires the image captured by the hovering camera 100, for example, from the control terminal 200. The timing at which the information processing device 300 acquires the image captured by the hovering camera 100 is not limited to a specific timing, but for example, the information processing device 300 may acquire the image captured by the hovering camera 100 from the control terminal 200 at the timing at which one flight of the hovering camera 100 ends.

[0037] The wireless relay node 400 is a device that relays wireless communication between the hovering camera 100 and the control terminal 200. As described above, in the inspection of a structure, particularly a bridge 1, the hovering camera 100 may fly beyond the communication range of the control terminal 200. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 may be performed via the wireless relay node 400 that is installed when inspecting the structure. The number of wireless relay nodes 400 is not limited to one, and multiple nodes may be installed depending on the inspection range of the bridge 1. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 may be performed via multiple wireless relay nodes 400. The hovering camera 100 may switch the communication destination between the control terminal 200 and the wireless relay node 400 depending on the radio wave conditions.

[0038] The wireless relay node 400 can be installed at an appropriate location on the bridge surface (preferably on the sidewalk) when inspecting the bridge 1. The wireless relay node 400 may also be installed so as to be suspended from the parapet of the bridge girder 3. Before inspecting the bridge 1, it is desirable to confirm that the wireless relay node 400 is operating normally by a predetermined method, for example, using the control terminal 200.

[0039] The position estimation node 500 is a device that allows the hovering camera 100 to estimate the current position. As described above, the flight path of the hovering camera 100 is set as, for example, GPS position information. In this case, if there is nothing blocking radio waves from GPS satellites, the hovering camera 100 can know the current position with extremely high accuracy. However, it is inevitable that the hovering camera 100 will get under the bridge girder 3. If the bridge girder 3 blocks radio waves from the GPS satellites or if multipath occurs due to radio wave reflection by the bridge 1, the hovering camera 100 may not be able to know the current position with high accuracy.

[0040] Therefore, in this embodiment, a position estimation node 500 is provided under the bridge girder 3 to allow the hovering camera 100 to accurately acquire the current position. As the position estimation node 500, for example, an AR (Augmented Reality) marker or a GPS signal transmitter may be used.

[0041] When an AR marker is used as the position estimation node 500, in order to have the hovering camera 100 recognize the current position, for example, the position estimation node 500 is hung from both ends of the bridge 1, and the hovering camera 100 is made to capture the position estimation node 500. Then, the hovering camera 100 that captured the position estimation node 500 is made to fly between the specified position estimation nodes 500. The hovering camera 100 can grasp the position between the position estimation nodes 500, for example, from an integrated value of a sensor (for example, an IMU (Inertial Measurement Unit) sensor) provided in the hovering camera 100 and a distance to the destination position estimation node 500 calculated from the captured image. Therefore, the hovering camera 100 can accurately obtain the current position even under the bridge girder 3 by capturing an image of the position estimation node 500.

[0042] Furthermore, when a GPS signal transmitter is used as the position estimation node 500, in order to allow the hovering camera 100 to recognize its current position, the position estimation node 500 is installed, for example, at a diagonal or at each of the four corners of the bridge 1. By receiving the GPS signal emitted from the position estimation node 500, the hovering camera 100 can accurately obtain its current position even under the bridge girder 3.

[0043] The base station 600 is a device provided for the takeoff and landing of the hovering camera 100. The base station 600 is equipped with a GPS receiver and calculates its current position by receiving radio waves from a GPS satellite. The current position calculated by the base station 600 is sent to the control terminal 200. By sending the current position calculated by the base station 600 to the control terminal 200, the control terminal 200 becomes able to display the position of the base station 600 on the overview map of the bridge 1.

[0044] The base station 600 may have a function of checking the operation of the hovering camera 100. The check of the operation of the hovering camera 100 performed by the base station 600 may include, for example, checking the communication function, the imaging function, the flight function, and calibration of various sensors. It goes without saying that the method of calibrating the sensor of the hovering camera 100 is not limited to the method using the base station 600. As a method of calibrating the sensor of the hovering camera 100, for example, a method of fixing the hovering camera 100 to a jig dedicated to calibration and rotating the hovering camera 100 in the pitch direction or roll direction to calibrate the sensor may be used.

[0045] The charging station 700 charges a secondary battery provided in the hovering camera 100. The hovering camera 100 is powered by a battery, and consumes power stored in the battery during flight and image capture. If the battery provided in the hovering camera 100 is a secondary battery, the charging station 700 can recover power consumed by the hovering camera 100 by charging the battery. The charging station 700 may charge the hovering camera 100 by connecting a cable or the like to the hovering camera 100 and supplying power to the hovering camera 100, or may charge the hovering camera 100 by supplying power to the hovering camera 100 using a non-contact power transmission method.

[0046] The server device 800 is a device that stores various data. In this embodiment, the server device 800 may store the damage data generated by the information processing device 300.

[0047] 2, the inspection system 10 according to an embodiment of the present disclosure can capture an image of the bridge 1 using the hovering camera 100 to obtain an image of the bridge 1. By capturing an image of the bridge 1 using the hovering camera 100, the inspection system 10 according to an embodiment of the present disclosure can eliminate the need to set up scaffolding on bridge piers and girders, reduce the frequency of closing some or all lanes to ensure the safety of workers, or eliminate the need for road closures, thereby enabling low-cost and efficient inspection of the bridge 1.

[0048] An example of the system configuration of the inspection system 10 according to an embodiment of the present disclosure has been described. Next, an example of the functional configuration of the hovering camera 100 and the control terminal 200 constituting the inspection system 10 according to an embodiment of the present disclosure will be described.

[0049] [1.3. Example of functional configuration] First, a functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure will be described. Fig. 3 is an explanatory diagram showing a functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure. Hereinafter, the functional configuration example of the hovering camera 100 according to an embodiment of the present disclosure will be described with reference to Fig. 3.

[0050] As shown in FIG. 3, the hovering camera 100 according to one embodiment of the present disclosure includes an imaging device 101, rotors 104a to 104d, motors 108a to 108d, a control unit 110, a communication unit 120, a sensor unit 130, a position information acquisition unit 132, a memory unit 140, and a battery 150.

[0051] The control unit 110 controls the operation of the hovering camera 100. For example, the control unit 110 can control adjustment of the rotation speed of the rotors 104a-104d by adjusting the rotation speed of the motors 108a-108d, imaging processing by the imaging device 101, transmission and reception processing of information to and from other devices (e.g., the control terminal 200) via the communication unit 120, and storage and reading of information in the memory unit 140.

[0052] In this embodiment, the control unit 110 controls flight by adjusting the rotation speeds of the motors 108a to 108d based on flight information transmitted from the control terminal 200, and controls the execution of still image capture processing for the imaging device 101. The control unit 110 controls the motors 108a to 108d and the imaging device 101 based on flight information transmitted from the control terminal 200, thereby making it possible to provide the control terminal 200 with an image based on a request from the control terminal 200.

[0053] The imaging device 101 is composed of a lens, an imaging element such as a CCD image sensor or a CMOS image sensor, a flash, etc. The imaging device 101 provided in the hovering camera 100 captures still images or moving images under the control of the control terminal 200. The image captured by the imaging device 101 is transmitted from the communication unit 120 to the control terminal 200. In this embodiment, the imaging device 101 performs imaging processing based on information on the imaging position of the still image included in the flight information transmitted from the control terminal 200. The image obtained by the imaging processing of the imaging device 101 may be stored in the storage unit 140 or transmitted from the communication unit 120 to the control terminal 200. When the hovering camera 100 captures an image of the bottom side of the bridge 1, it is considered that the sunlight is blocked by the bridge 1 and the brightness is insufficient, so the hovering camera 100 may emit a flash when capturing an image of the bottom side of the bridge 1.

[0054] The imaging device 101 can change the imaging direction to any direction, for example, under the control of the control unit 110. For example, when the horizontal direction of the hovering camera is set to 0 degrees, imaging can be performed in an imaging direction expressed in a range of ±90 degrees in the vertical direction. Since the imaging device 101 can change the imaging direction, the hovering camera 100 can capture an image in a predetermined direction and provide the captured image to the control terminal 200. The control unit 110 associates the position information of the hovering camera 100 at the time when the imaging device 101 captured a still image (which may include position information obtained by positioning using GPS or positioning using the position estimation node 500. Positioning using the position estimation node 500 will be described later), aircraft information at the time of imaging (for example, yaw angle, pitch angle, acceleration, angular velocity), and imaging direction information as metadata of the still image. As a method for storing the associated metadata, the metadata may be added to an additional information area of ​​the still image data (for example, a specific area of ​​the Exif format), or the metadata may be recorded in a file separate from the image file, or may be separate data.

[0055] The rotors 104a to 104d generate lift through rotation, thereby flying the hovering camera 100. The rotation of the rotors 104a to 104d is achieved by the rotation of the motors 108a to 108d. The motors 108a to 108d rotate the rotors 104a to 104d. The rotation of the motors 108a to 108d can be controlled by the control unit 110.

[0056] The communication unit 120 performs processing for transmitting and receiving information via wireless communication with the control terminal 200. The hovering camera 100 transmits images captured by the imaging device 101 from the communication unit 120 to the control terminal 200. The hovering camera 100 also receives flight-related instructions from the control terminal 200 via the communication unit 120.

[0057] The sensor unit 130 is a group of devices that acquires the state of the hovering camera 100, and may be composed of, for example, an acceleration sensor, a gyro sensor, an ultrasonic sensor, an air pressure sensor, an optical flow sensor, a laser range finder, and the like. The sensor unit 130 may convert the acquired state of the hovering camera 100 into a predetermined signal and provide it to the control unit 110 as necessary. The position information acquisition unit 132 acquires information on the current position of the hovering camera 100 using, for example, a GPS or a vision sensor. The position information acquisition unit 132 may provide the acquired information on the current position of the hovering camera 100 to the control unit 110 as necessary. The control unit 110 executes flight control of the hovering camera 100 based on the flight information received from the control terminal 200, using the information on the current position of the hovering camera 100 acquired by the position information acquisition unit 132.

[0058] In addition, the sensor unit 130 detects obstacles that may impede flight during flight. By detecting an obstacle with the sensor unit 130, the hovering camera 100 can provide information about the detected obstacle to the control terminal 200.

[0059] The storage unit 140 stores various information. Examples of the information stored in the storage unit 140 include flight information of the hovering camera 100 transmitted from the control terminal 200, images captured by the imaging device 101, and the like.

[0060] The battery 150 stores power for operating the hovering camera 100. The battery 150 may be a primary battery that can only be discharged, or may be a secondary battery that can also be charged. When the battery 150 is a secondary battery, the battery 150 may receive a supply of power from, for example, a charging station 700 shown in FIG. 2.

[0061] The hovering camera 100 according to one embodiment of the present disclosure has a configuration as shown in FIG. 3, and can automatically fly based on a flight path included in the flight information transmitted from the control terminal 200, and can perform imaging processing based on information on the imaging position of a still image included in the flight information transmitted from the control terminal 200.

[0062] An example of the functional configuration of the hovering camera 100 according to an embodiment of the present disclosure has been described above with reference to Fig. 3. Next, an example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure will be described.

[0063] 4 is an explanatory diagram showing an example of a functional configuration of the control terminal 200 according to an embodiment of the present disclosure. Hereinafter, the example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure will be described with reference to FIG.

[0064] As shown in FIG. 4, the control terminal 200 according to an embodiment of the present disclosure includes a display unit 210, a communication unit 220, a control unit 230, and a storage unit 240.

[0065] The display unit 210 is composed of a flat panel display device such as a liquid crystal display device, an organic EL display device, etc. The display unit 210 can display, for example, images captured by the imaging device 101 and information for controlling the operation of the hovering camera 100. The display unit 210 is provided with a touch panel, and a user can directly operate the information displayed on the display unit 210 by touching the display unit 210 with a finger or the like.

[0066] The communication unit 220 transmits and receives information to and from the hovering camera 100 via wireless communication. The control terminal 200 receives images captured by the imaging device 101 from the hovering camera 100 via the communication unit 220. The control terminal 200 also transmits instructions related to the flight of the hovering camera 100 from the communication unit 220 to the hovering camera 100. Commands related to the flight of the hovering camera 100 can be generated by the control unit 230.

[0067] The control unit 230 controls the operation of the control terminal 200. For example, the control unit 230 can control the display processing of characters, figures, images, and other information on the display unit 210, and the transmission and reception processing of information between the control unit 200 and other devices (e.g., the hovering camera 100) via the communication unit 220. The control unit 230 is also configured to include a flight information generation unit 232 and a display control unit 234.

[0068] The flight information generating unit 232 generates flight information to be transmitted to the hovering camera 100. When generating the flight information, the flight information generating unit 232 uses, for example, information on the structure to be inspected that is stored in the storage unit 240 described below. After generating the flight information, the flight information generating unit 232 causes the communication unit 220 to transmit the generated flight information before the hovering camera 100 takes off.

[0069] The flight information generating process by the flight information generating unit 232 will be described in detail later, but an example of the flight information generating process by the flight information generating unit 232 will be briefly described. When generating flight information for the hovering camera 100, the flight information generating unit 232 reads a general view of the bridge 1 to be inspected. The read general view of the bridge 1 is displayed on the display unit 210 by the display control unit 234. As described above, points on the general view of the bridge 1 are previously associated with points on map data containing detailed GPS information. It is desirable that this association is performed for at least two pairs of points. By previously associating points on map data containing detailed GPS information with the general view of the bridge 1, the flight route of the hovering camera 100 is defined as a GPS value (a set of latitude and longitude).

[0070] The flight information generating unit 232 generates a flight path for the hovering camera 100 based on the general situation map of the bridge 1. When generating the flight path for the hovering camera 100, the flight information generating unit 232 uses information on the structure of the bridge 1, such as the construction method, width, and span length, the flight time of the hovering camera 100, and the inspection method of the bridge 1. Concrete bridges are classified into reinforced concrete (RC) and prestressed concrete (PC) according to the reinforcement method, and are classified into RCT girder bridges, PCT girder bridges, PC hollow deck bridges, RC box girder bridges, PC box girder bridges, etc. according to the shape of the girders. Therefore, if the construction method of the bridge 1 to be inspected is known, the flight information generating unit 232 can generate a flight path suitable for the construction method of the bridge 1. The flight information generating unit 232 displays the flight path of the hovering camera 100 superimposed on the general situation map of the bridge 1.

[0071] The flight information generating unit 232 defines the flight path of the hovering camera 100 as GPS values ​​(a set of latitude and longitude) as described above. By the flight information generating unit 232 defining the flight path of the hovering camera 100 as GPS values, the hovering camera 100 can determine at what position during flight the imaging process should be performed, based on the GPS values.

[0072] The display control unit 234 controls the display of characters, figures, images, and other information on the display unit 210. The display of characters, figures, symbols, images, and other information displayed on the display unit 210 in the drawings referred to in the following description is controlled by the display control unit 234. For example, when the flight information generating unit 232 generates flight information to be transmitted to the hovering camera 100, the display control unit 234 executes control to display on the display unit 210 an overview diagram of the structure to be inspected (bridge 1) and the generated flight information.

[0073] The storage unit 240 stores various information. Information stored in the storage unit 240 includes, for example, information about the structure (bridge 1) to be inspected. The information about the structure to be inspected may include, for example, an overview map of the structure to be inspected (bridge 1) and a construction method of the structure to be inspected. Furthermore, if locations of the structure to be inspected that are thought to be prone to damage are known in advance, the information about the structure to be inspected may include information about the parts that are thought to be highly likely to be damaged.

[0074] The control terminal 200 may receive information about the structure (bridge 1) to be inspected, for example, from the information processing device 300, when inspecting the structure, without having to store the information in advance in the storage unit 240.

[0075] The control terminal 200 according to one embodiment of the present disclosure has a configuration as shown in FIG. 4, and thereby generates flight information to be transmitted to the hovering camera 100 based on information regarding the structure (bridge 1) to be inspected, and can acquire images captured based on the flight information by the hovering camera 100, which flies based on the flight information.

[0076] An example of the functional configuration of the control terminal 200 according to an embodiment of the present disclosure has been described above with reference to Fig. 4. Next, an example of the operation of the inspection system 10 according to an embodiment of the present disclosure will be described.

[0077] [1.4. Example of operation] Fig. 5 is a flow chart showing an example of operation of the inspection system 10 according to an embodiment of the present disclosure. Fig. 5 shows an example of operation of the inspection system 10 according to an embodiment of the present disclosure when inspecting the bridge 1 by flying the hovering camera 100 and having the hovering camera 100 capture an image of the bridge 1. It is assumed that the wireless relay node 400 and the position estimation node 500 are installed in advance at appropriate positions on the bridge 1 when inspecting the bridge 1 using the hovering camera 100. Hereinafter, an example of operation of the inspection system 10 according to an embodiment of the present disclosure will be described with reference to Fig. 5.

[0078] The control terminal 200, which generates flight information for the hovering camera 100, reads information about the bridge 1, including a general view of the bridge 1 to be inspected, and displays the general view of the bridge 1 on the display unit 210 (step S101). The information about the bridge 1 is read by, for example, the flight information generating unit 232, and the display of the general view of the bridge 1 on the display unit 210 is performed by, for example, the display control unit 234. The control terminal 200, which is displaying the general view of the bridge 1 on the display unit 210, prompts the user to specify an area of ​​the bridge 1 to be inspected using the general view of the bridge 1 displayed on the display unit 210 (step S102). The process of prompting the user to specify in step S102 is performed by, for example, the flight information generating unit 232.

[0079] For example, when a portion of the bridge 1 is to be inspected, the control terminal 200 allows the user to specify the area to be inspected in the overview map of the bridge 1 displayed on the display unit 210. Also, when the entire bridge 1 is to be inspected, the control terminal 200 allows the user to specify the area of ​​all parts of the bridge 1 in the overview map of the bridge 1 displayed on the display unit 210.

[0080] FIG. 6 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 6 shows an example of a screen displayed on the display unit 210 when the user is prompted to specify the area of ​​the bridge 1 to be inspected in step S102. In FIG. 6, a screen is displayed on the display unit 210 when a bridge girder is specified as the area of ​​the bridge 1 to be inspected. The control terminal 200 has, for example, a touch panel as an input unit (not shown), and can allow the user to specify the area of ​​the bridge 1 by, for example, tracing the screen or selecting a span to be inspected. Of course, the method of allowing the user to specify the area of ​​the bridge 1 to be inspected is not limited to the example. Moreover, the display of the area specified by the user is not limited to the example shown in FIG. 6.

[0081] 6 also shows a state in which a mark B1 indicating the position of the base station 600 is displayed superimposed on the general situation map of the bridge 1. As described above, the base station 600 is equipped with a GPS receiver and can calculate its current position by receiving radio waves from GPS satellites. Therefore, the control terminal 200 can display the mark B1 indicating the position of the base station 600 superimposed on the general situation map of the bridge 1 based on the information on the current position calculated by the base station 600.

[0082] After the user specifies the area of ​​the bridge 1 to be inspected, the control terminal 200 then generates flight information of the hovering camera 100 in the area to be inspected specified by the user based on information about the bridge 1 (step S103). The flight information generation process in step S103 is executed by, for example, the flight information generation unit 232.

[0083] When generating flight information of the hovering camera 100 in step S103, the control terminal 200 uses information on the structure of the bridge 1, such as the construction method, width, span length, etc., the flight time of the hovering camera 100, and the inspection method of the bridge 1. For example, when a T-girder is used as the construction method of the bridge 1, the control terminal 200 generates a flight path as flight information in which the hovering camera 100 repeatedly rises and falls on the bottom side of the bridge 1. Furthermore, when generating flight information of the hovering camera 100 in step S103, the control terminal 200 may use information on the imaged surface of the bridge 1. For example, when the user selects to image the side of the bridge 1, the control terminal 200 generates a flight path along the side of the bridge 1 as flight information, and when the user selects to image the bottom of the bridge 1, the control terminal 200 generates a flight path that goes back and forth on the bottom side of the bridge 1 as flight information.

[0084] A description will be given of an example of flight information generated by the control terminal 200. The flight information may specify, for example, a list for each position where the image capturing process is to be performed in the following format. ID: (relative coordinates of the imaging point, imaging direction, imaging speed, travel time to the next imaging point, etc.) The relative coordinates of the imaging point are specified by three points on the X-axis, Y-axis, and Z-axis. The X-axis is the latitude direction, the Y-axis is the longitude direction, and the Z-axis is the height direction. Other information may include, for example, information for controlling special imaging. Information for controlling special imaging may include, for example, information for imaging in multiple imaging directions at the same position, information on parameters for bracket imaging (imaging at the same position and imaging direction with different exposures, shutter speeds, ISO sensitivity, etc.), and information on the wavelength of infrared light at the time of imaging. According to this format, the flight information generated by the control terminal 200 may be composed of a list of values ​​such as the following: 0: (0,0,0,0,0,2,1.0) 1:(5,0,0,0,0,2,1.0) 2:(7,0,0,0,0,2,1.0) 3:(9,0,0,0,0,2,1.0) The imaging point included in the flight information generated by the control terminal 200 may be specified by relative coordinates from the reference point, for example, the absolute coordinates of the base station 600 or the absolute coordinates of an arbitrary position such as the first imaging position. The hovering camera 100 may convert the relative coordinates from the absolute coordinates of the reference point into absolute coordinates and refer to the converted coordinates during flight. Also, the imaging point included in the flight information generated by the control terminal 200 may be specified by absolute coordinates instead of relative coordinates. Also, a predetermined value may be stored in the information for controlling special imaging included in the flight information generated by the control terminal 200. For example, the information for controlling special imaging may store values ​​such as 1: imaging in multiple imaging directions, 2: bracket imaging (change in shutter speed), 3: bracket imaging (change in ISO sensitivity), etc. The control terminal 200 may include information for controlling special imaging in the flight information for a location that is considered to be prone to damage in the bridge girder 3, which is stored in the storage unit 240, for example.

[0085] During the flight information generation process in step S103, the control terminal 200 may generate flight information that causes the hovering camera 100 to capture images of the underside of the bridge girder 3 of the bridge 1 at equal intervals. Therefore, during the flight information generation process in step S103, the control terminal 200 may generate flight information such that the imaging positions of the still images are equal intervals.

[0086] When generating flight information for the hovering camera 100 in step S103, if information on the part considered to be highly likely to be damaged is stored in advance in the storage unit 140, the control terminal 200 may read out the stored information and generate flight information that causes the hovering camera 100 to capture a detailed image of the part. When causing the hovering camera 100 to capture an image of the part considered to be highly likely to be damaged, the control terminal 200 may include information for controlling the above-mentioned special image capture in the flight information. Of course, the information on the part considered to be highly likely to be damaged does not have to be stored in advance in the storage unit 140, and in that case, the user may input information on the part considered to be highly likely to be damaged at the time of inspection.

[0087] When flying the hovering camera 100 over an area of ​​the bridge 1 to be inspected, it may be impossible to fly the hovering camera 100 over the area at once, depending on the flight time of the hovering camera 100. The flight time of the hovering camera 100 can be calculated in advance from the capacity of the battery 150, the power consumption of the motors 108a-108d for moving the rotors 104a-d, the power consumption of the imaging device 101, the control unit 110, and the communication unit 120, etc. When generating flight information, it is also possible to estimate the time required for one inspection flight of the hovering camera 100 from the planned travel time from the start position (e.g., the base station 600) to the first imaging point, the planned travel time between the imaging points, the planned travel time from the last imaging point to return to the start position, etc. Therefore, if the flight route for the area of ​​the bridge 1 to be inspected cannot be flown by the hovering camera 100 in one inspection flight, the control terminal 200 may divide the generated flight route into several parts.

[0088] Furthermore, when the control terminal 200 generates flight information of the hovering camera 100 in the above step S103, the control terminal 200 may generate a plurality of flight paths and display the flight paths on the display unit 210. FIG. 7 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 7 shows an example of a state in which a plurality of flight paths are generated when flight information of the hovering camera 100 is generated in the above step S103, and the flight paths R1 and R2 are displayed on the display unit 210. The control terminal 200 displays a plurality of flight paths on the display unit 210 and allows the user to select one of the flight paths. The control terminal 200 generates flight information based on the user's selection of the flight path.

[0089] After generating the flight information of the hovering camera 100 in step S103, the control terminal 200 subsequently transmits the generated flight information to the hovering camera 100 and transmits a takeoff instruction to the hovering camera 100 (step S104). The transmission of the generated flight information and the takeoff instruction is executed by the flight information generating unit 232 via the communication unit 220, for example.

[0090] FIG. 8 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. FIG. 8 shows an example of a screen displayed on the display unit 210 of the control terminal 200 when a takeoff instruction is transmitted to the hovering camera 100. The user can transmit a takeoff instruction from the control terminal 200 to the hovering camera 100 by touching a takeoff instruction button 211 displayed on the display unit 210. When transmitting a takeoff instruction from the control terminal 200 to the hovering camera 100, the flight information generated in the above step S103 may be transmitted from the control terminal 200 to the hovering camera 100 before the takeoff instruction, or the flight information generated in the above step S103 may be transmitted from the control terminal 200 to the hovering camera 100 after transmitting a takeoff instruction from the control terminal 200 to the hovering camera 100.

[0091] The hovering camera 100 receives flight information and a takeoff instruction from the control terminal 200 and takes off from the base station 600, and flies based on the flight information sent from the control terminal 200, performs imaging processing to obtain a still image (step S105). The hovering camera 100 acquires position information when performing imaging processing to obtain a still image and information on the aircraft at the time of imaging processing, and links the information to the still image. The information on the aircraft at the time of imaging processing may include, for example, yaw angle, pitch angle, acceleration, and angular velocity information. The hovering camera 100 may also transmit, to the control terminal 200, streaming video images captured by the imaging device 101 during flight. The control terminal 200 acquires and displays, by streaming, video images captured by the imaging device during flight, of the hovering camera 100, so that the control terminal 200 can present to the user where the hovering camera 100 is flying.

[0092] When performing imaging processing, it is desirable for the hovering camera 100 to maintain a constant distance from the imaging target surface (for example, the side or bottom surface of the bridge girder 3) at all imaging points. By maintaining a constant distance from the imaging target surface at all imaging points, the hovering camera 100 can obtain still images captured at the same size.

[0093] When a part considered to be highly likely to be damaged is included in the flight path of the hovering camera 100, the hovering camera 100 may capture multiple still images of that part by changing the imaging direction of the imaging device, applying infrared rays with different wavelengths, or changing the shutter speed. Also, when a part considered to be highly likely to be damaged is included in the flight path of the hovering camera 100, the hovering camera 100 may narrow the intervals between positions for performing imaging processing for that part compared to other parts.

[0094] 9 is an explanatory diagram conceptually illustrating the operation of the hovering camera 100 in the inspection system 10 according to an embodiment of the present disclosure. When the hovering camera 100 flies over the bottom side of the bridge 1 based on flight information, the hovering camera 100 stops, for example, at time t1 to capture an image of the bottom side of the bridge 1, flies to a position for capturing an image at time t2 after capturing the image, stops at time t2 to capture an image of the bottom side of the bridge 1 at another position, and thereafter repeats flying, stopping, and capturing an image until time tn to capture an image of the bottom side of the bridge 1. An image of the bottom side of the bridge 1 is obtained by the hovering camera 100 repeatedly flying, stopping, and capturing an image.

[0095] When flying based on flight information, the hovering camera 100 can accurately determine its current position if it can receive radio waves from GPS satellites without any interference. However, in a location where it is difficult to receive radio waves from GPS satellites, such as under a bridge 1, it becomes difficult for the hovering camera 100 to accurately determine its current position. Therefore, in this embodiment, by using a position estimation node 500, the hovering camera 100 can accurately determine its current position even in a location where it is difficult to receive radio waves from GPS satellites.

[0096] Fig. 10 is an explanatory diagram conceptually illustrating the operation of the hovering camera 100 in the inspection system 10 according to an embodiment of the present disclosure. For example, when a route is set such that the hovering camera 100 flies from Start to Goal in Fig. 10, the hovering camera 100 travels between a GPS positioning area 40 where the hovering camera 100 receives radio waves from a GPS satellite 30 without interference to determine its position, and a sensor positioning area 50 where the current position is estimated using, for example, a vision sensor.

[0097] In a GPS positioning area 40, the hovering camera 100 grasps its current position using radio waves from a GPS satellite 30. In a sensor positioning area 50, the hovering camera 100 grasps its position between position estimation nodes 500, and if the position estimation node 500 is an AR marker, it grasps its current position using an integrated value of a sensor (e.g., an IMU sensor) provided in the hovering camera 100 and a distance to the destination position estimation node 500 calculated from an image captured by the imaging device 101. If the position estimation node 500 is a GPS signal transmitter, the hovering camera 100 grasps each of them using a signal transmitted from the position estimation node 500.

[0098] By using the position estimation node 500 in this manner, the hovering camera 100 can determine its exact current position even when it moves to a location where it is difficult to receive radio waves from a GPS satellite.

[0099] When the hovering camera 100 completes the imaging process at the last imaging point, it automatically flies to the base station 600 and returns to the base station 600 (step S106). Then, the control terminal 200 acquires from the hovering camera 100 the images captured by the hovering camera 100 that has returned to the base station 600 (step S107). Note that the images captured by the hovering camera 100 may be acquired after the hovering camera 100 returns to the base station 600 in this manner, but the control terminal 200 may successively acquire still images each time the hovering camera 100 executes imaging process to obtain the still images.

[0100] In an inspection system 10 according to one embodiment of the present disclosure, the hovering camera 100 and the control terminal 200 perform operations as shown in FIG. 5, so that flight information to be transmitted to the hovering camera 100 is generated by the control terminal 200 based on information about the structure (bridge 1) to be inspected, the hovering camera 100 flies based on the flight information, images are captured based on the flight information, and the images captured by the hovering camera 100 are acquired by the control terminal 200.

[0101] It is assumed that while the hovering camera 100 is flying, the user finds a part that he or she wishes to capture in detail while viewing the video captured by the hovering camera 100. In this case, for example, the user may operate the control terminal 200 to cause the control terminal 200 to send an instruction to the hovering camera 100 to stop automatic flight and switch to manual operation.

[0102] In the above example, flight information is generated by the control terminal 200, and the hovering camera 100 automatically flies based on the generated flight information to perform image capture processing. However, it is possible that an obstacle that cannot be seen in the overview diagram of the bridge 1 exists on the flight path.

[0103] Fig. 11 is an explanatory diagram conceptually illustrating the operation of the hovering camera 100 in the inspection system 10 according to an embodiment of the present disclosure. Fig. 11 shows a state in which a tree 4 is growing under a bridge girder 3. This tree 4 is an obstacle that does not appear in the overview diagram of the bridge 1, and its presence may become apparent for the first time when the hovering camera 100 flies.

[0104] Therefore, in this embodiment, the hovering camera 100 may be test-flew once based on the flight information generated by the control terminal 200 to check whether there are any obstacles on the flight path included in the flight information.

[0105] When the hovering camera 100 is test-flew once based on the flight information generated by the control terminal 200, the moving image captured by the hovering camera 100 may be received by streaming on the control terminal 200, and the user may check whether there is an obstacle on the flight path included in the flight information while watching the moving image, or the sensor unit 130 of the hovering camera 100 may detect the obstacle. The detailed position of the obstacle can be grasped by providing a stereo camera as the imaging device 101 of the hovering camera 100, grasping the distance to the obstacle by imaging with the stereo camera, or identifying the direction of the obstacle by the orientation of the hovering camera 100. Note that, if an obstacle exists on the flight path during the test flight of the hovering camera 100, the hovering camera 100 may stop the automatic flight and transition to a hovering state and wait for an operation from the user, or may automatically return to the base station 600.

[0106] When it is found that an obstacle exists on the flight path included in the flight information, the control terminal 200 may register the location of the obstacle in the overview map of the bridge 1. The location of the obstacle may be manually input by the user, or when the hovering camera 100 detects an obstacle by the sensor unit 130, the location of the detected obstacle may be acquired from the hovering camera 100 and the location of the obstacle may be registered in the overview map of the bridge 1.

[0107] Fig. 12 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. Fig. 12 shows an example of a screen displayed on the display unit 210 when the test flight of the hovering camera 100 reveals that an obstacle exists on the flight path. If the test flight of the hovering camera 100 reveals that an obstacle exists on the flight path, the control terminal 200 displays a mark O1 indicating the location of the obstacle superimposed on the overview map of the bridge 1.

[0108] Once the location of the obstacle is known, the control terminal 200 regenerates flight information so as to create a flight path that avoids the location of the obstacle, and transmits the regenerated flight information to the hovering camera 100. The hovering camera 100 flies based on the flight information regenerated by the control terminal 200, and can fly and capture images while avoiding the obstacle (tree 4).

[0109] The method of determining the location of an obstacle by flying the hovering camera 100 is not limited to the above example. For example, the hovering camera 100 may fly along a simple route around the periphery of the flight route generated by the control terminal 200 while capturing moving images with the imaging device 101, to check whether there is an obstacle under the bridge girder 3.

[0110] [1.5. Example of damage data generation] By flying the hovering camera 100 and having the hovering camera 100 capture an image of the bridge 1, it becomes possible to grasp the state of a place that workers cannot easily approach, such as the bottom surface of the bridge girder 3. A still image captured by the hovering camera 100 is associated with, for example, position information of the hovering camera 100 that captured the still image (which may include position information obtained by GPS positioning or positioning using the position estimation node 500), aircraft information at the time of capturing the image (e.g., yaw angle, pitch angle, acceleration, angular velocity), and information on the capturing direction. In addition, the hovering camera 100 captures images while maintaining a constant distance from the surface to be captured at all capturing points, so that the relative position of the location where damage has occurred in the image can be known. Therefore, if a damaged portion of the bridge girder 3 is included in the still image captured by the hovering camera 100, it becomes possible to grasp the absolute location of the damaged portion. For example, the position information of the damaged part can be obtained by calculating the relative value of the damaged location with the center of the still image as the origin, and then calculating the relative value to the position information of the hovering camera 100 when the image was captured. Note that, for example, the following data can be recorded as the position information of the damaged part. (1) Record the capture position information of the still image as the position of the damaged area (without recording a relative value (offset)). (2) Information on the imaging position of the still image and a relative value (offset) corresponding to the damaged part are recorded as the position of the damaged part. (3) The reference absolute value (for example, as described below, the capture positions of still images of the four corners where the accuracy of position information is considered to be high, or the coordinates of the position estimation node 500) and the relative value (offset) are recorded as the position of the damaged part. (4) The calculated absolute values ​​(e.g., latitude, longitude, altitude) are recorded as the location of the damaged part.

[0111] A technique is known for determining the physical size of the imaging range by using numerical values ​​such as the focal length of the lens, the size of the image sensor, and the distance to the imaging target. Therefore, when grasping the damaged part, the physical size of the imaging range of the hovering camera 100 may be estimated using distance information from the hovering camera 100 to the imaging target (for example, the back side or side of the bridge girder 3) and angle of view information of the imaging device 101. The center position of the captured image (the position where the hovering camera 100 captured the image) is set as the origin, and the physical relative position from the origin to the damaged part is estimated, and the position coordinate of the origin of the captured image is added to this relative position, thereby determining the physical position information of the damaged part. Note that the distance information and angle of view information may be obtained via a sensor possessed by the hovering camera 100 at the time of imaging and recorded in association with the image, or values ​​previously set in the hovering camera 100 or the imaging device 101 may be used. Furthermore, the position information of the damaged portion may be calculated using aircraft information at the time of imaging (for example, yaw angle, pitch angle, acceleration, angular velocity) and imaging direction information in addition to imaging position information, distance information, and angle of view information.

[0112] The detection of damaged parts from the still images captured by the hovering camera 100 may be performed by the user himself / herself by visually inspecting the damaged parts, or may be performed automatically by image processing, for example, by the information processing device 300. When automating the detection of damaged parts, for example, an image processing technique such as pattern matching processing may be used.

[0113] The data structure of the damage data is defined, for example, in the following format. (Image ID, damage ID, location information of the damaged part, coordinates of the damaged part on the image, damage type ID, degree of damage) The damage type ID refers to an ID given to each type of damage, such as cracks, peeling, water leakage, free lime, etc. In addition, the maximum width of the damage, the length of the damaged part in the image, etc. can be recorded in the damage degree field. The inspection system 10 according to this embodiment can generate damage data according to the above-mentioned format from the still image captured by the hovering camera 100, either manually input by the user or automatically processed by the information processing device 300. The damage data generated by the inspection system 10 according to this embodiment can be used in ordering to a construction company to repair the damage caused to the bridge 1.

[0114] However, the hovering camera 100 captures many still images during one inspection flight, and therefore it is a heavy burden on the user to check each still image captured by the hovering camera 100 during an inspection flight.

[0115] Therefore, the still images captured by the hovering camera 100 are joined (stitched) together to obtain a single image. By stitching the still images captured by the hovering camera 100, for example, the state of the bottom surface of one span of the bridge girder 3 can be obtained as a single image. Then, by checking the image of the bottom surface of one span of the bridge girder 3 obtained by stitching the still images captured by the hovering camera 100, the user can check whether there is any damage to the bottom surface of the bridge girder 3. The stitching process of the still images may be performed by the control terminal 200 or the information processing device 300.

[0116] 13 is an explanatory diagram showing an overview of inspecting the bottom surface of a bridge girder 3 based on still images captured by a hovering camera 100. A part of the bottom surface of the bridge girder 3 (for example, the length of one span of the bridge girder 3) is captured by the hovering camera 100, and the still images captured by the hovering camera 100 are stitched together to obtain a single image 20 of the bottom surface of the bridge girder 3. Note that reference numeral 21 indicates an image captured by the hovering camera 100 in one imaging process.

[0117] When determining the absolute location of the damaged part from an image obtained by stitching still images captured by the hovering camera 100, position information of the stitched images with relatively high accuracy of position information at the time of capture can be selected as a reference point. Position information of the hovering camera 100 at the time of capture of the still images of the four corners that are the basis of the stitched image may be used as a reference point. The still images of the four corners that are the basis of the stitched image have the least distortion, and the GPS positioning area has less error in position information, and it is considered preferable to use the position information at the time of capture of the four corners close to the GPS positioning area as a reference point. Therefore, by determining the absolute location of the damaged part from the position information of the hovering camera 100 corresponding to the still images of the four corners, it is possible to accurately determine the position of the damaged part. Note that, for example, positioning status information in the GPS positioning data (information indicating a state of 2D positioning, 3D positioning, or positioning not possible, data such as the number of received satellites, etc.) may be used as the accuracy of each position information.

[0118] 14 is an explanatory diagram showing an example of image 20 obtained by stitching still images captured by hovering camera 100. Centers G1 to G4 of still images C1 to C4 at the four corners that form the basis of image 20 correspond to the positions of the hovering camera 100 when each still image was captured. In this embodiment, the absolute positions of each damaged portion in image 20 are calculated using position information of the hovering camera 100 that corresponds to the still images C1 to C4 at the four corners.

[0119] When generating damage data from stitched images, the data configuration of the damage data is defined, for example, in the following format: That is, the image ID is deleted from the above-mentioned damage data. (Damage ID, location information of the damaged part, coordinates of the damaged part on the image, damage type ID, degree of damage) Alternatively, an image ID of the stitched image may be generated and included in the damage data. The inspection system 10 according to the present embodiment can generate damage data according to the above-mentioned format from the stitched image by manual input by the user or automatic processing by the information processing device 300.

[0120] [1.5.1. Example of functional configuration] Fig. 15 is an explanatory diagram showing a functional configuration example of an information processing device 300 according to an embodiment of the present disclosure. Fig. 15 shows a functional configuration example of the information processing device 300 according to an embodiment of the present disclosure, which has a function of obtaining an absolute damaged position of the bridge girder 3 from a still image captured by the hovering camera 100 and generating damage data. Hereinafter, a functional configuration example of the information processing device 300 according to an embodiment of the present disclosure will be described with reference to Fig. 15.

[0121] As shown in FIG. 15, an information processing device 300 according to an embodiment of the present disclosure includes a display unit 310, a communication unit 320, a control unit 330, and a storage unit 340.

[0122] The display unit 310 is composed of a flat panel display device such as a liquid crystal display device, an organic EL display device, etc. The display unit 310 can display, for example, an image captured by the imaging device 101 of the hovering camera 100, information about damage to the bridge 1 obtained from the image captured by the imaging device 101, etc.

[0123] The communication unit 320 transmits and receives information via wireless communication, for example, with the control terminal 200. The information processing device 300 receives, via the communication unit 320, an image captured by the hovering camera 100 from the control terminal 200 together with information on the absolute imaging position of the image.

[0124] The control unit 330 controls the operation of the information processing device 300. For example, the control unit 330 can control the display processing of characters, figures, images, and other information on the display unit 310, and the transmission and reception processing of information with other devices (for example, the control terminal 200) via the communication unit 320. The control unit 330 is also configured to include an imaging position information acquisition unit 332, a damage position calculation unit 334, an image synthesis unit 336, and a damage data generation unit 338.

[0125] The imaging position information acquisition unit 332 acquires information on the imaging position at the time of imaging, which is acquired by the hovering camera 100 when the hovering camera 100 images the bridge 1. The damage position calculation unit 334 detects damaged parts of the bridge 1 from the image captured by the hovering camera 100, for example, by using an image processing technique such as pattern matching processing, and calculates the absolute position of the damaged parts by using the imaging position information acquired by the imaging position information acquisition unit 332.

[0126] The image synthesis unit 336 executes image stitching processing to generate one image by connecting still images captured by the hovering camera 100. When stitching the still images captured by the hovering camera 100, the image synthesis unit 336 may use information on the imaging positions at the time of capturing each still image.

[0127] When calculating the damage position, the damage position calculation unit 334 may use information on the imaging positions of the captured images at the corners (e.g., each of the four corners) in the captured images that are the basis of the image stitched by the image synthesis unit 336. As described above, it is considered that the still images at the four corners in the captured images that are the basis of the stitched image have the least distortion, so the damage position calculation unit 334 can find a more accurate damage position by using information on the imaging positions of the captured images at the corners in the captured images that are the basis of the stitched image.

[0128] The damage data generation unit 338 generates the above-mentioned damage data using the absolute position of the damaged portion of the bridge 1 calculated by the damage position calculation unit 334. The damage data generation unit 338 may generate damage data for each still image, or may generate one piece of damage data for the images stitched by the image synthesis unit 336.

[0129] The storage unit 340 stores various information. The information stored in the storage unit 340 may include, for example, information on a still image captured by the imaging device 101 of the hovering camera 100 and the absolute imaging position of the hovering camera 100 when the still image was captured, and information on damage data generated by the damage data generation unit 338.

[0130] The information processing device 300 according to an embodiment of the present disclosure has a configuration as shown in FIG. 15, and thus it is possible to generate damage data from a still image captured by the hovering camera 100. Therefore, the information processing device 300 according to an embodiment of the present disclosure can efficiently generate an inspection result of the bridge 1, which is a structure to be inspected. As described above, the damage data may be generated by the control terminal 200, not the information processing device 300. Therefore, the configuration of the control unit 330 of the information processing device 300 shown in FIG. 15 may be included in the control terminal 200. Furthermore, the inspection result of the bridge 1, which is a structure to be inspected, can be accumulated in a public or private database and utilized. As described above, the damage data may be generated by the control terminal 200, not the information processing device 300. Therefore, the configuration of the control unit 330 of the information processing device 300 shown in FIG. 15 may be included in the control terminal 200.

[0131] An example of the functional configuration of the information processing device 300 according to an embodiment of the present disclosure has been described above with reference to Fig. 15. Next, an example of the operation of the information processing device 300 according to an embodiment of the present disclosure will be described.

[0132] [1.5.2. Example of operation] Fig. 16 is a flow chart showing an operation example of the information processing device 300 according to an embodiment of the present disclosure. Fig. 16 shows an operation example of the information processing device 300 according to an embodiment of the present disclosure when obtaining an absolute damage position of the bridge girder 3 from a still image captured by the hovering camera 100 and generating damage data. Hereinafter, an operation example of the information processing device 300 according to an embodiment of the present disclosure will be described with reference to Fig. 16.

[0133] The information processing device 300 first acquires an image associated with information on the image capturing position captured by the hovering camera 100 while flying around the edge of the bridge 1 (step S301). After acquiring the image associated with the information on the image capturing position in step S301, the information processing device 300 subsequently detects a damaged portion from the image using an image processing technique such as a pattern matching process (step S302). The detection process of the damaged portion in step S302 may be executed by the damage position calculation unit 334, for example.

[0134] When the damaged portion is detected from the image in the above step S302, the information processing device 300 subsequently calculates the absolute position of the damaged portion (step S303). The calculation process in step S303 may be executed by, for example, the damage position calculation unit 334. The information processing device 300 calculates the absolute position of the damaged portion in step S303 based on the information of the imaging position of the still image captured by the hovering camera 100. When calculating the absolute position of the damaged portion, the information processing device 300 may estimate the physical size of the imaging range of the hovering camera 100 based on the distance information from the hovering camera 100 to the imaging target (for example, the back surface or side surface of the bridge girder 3) and the field angle information of the imaging device 101. The information processing device 300 can determine the physical position information of the damaged portion by estimating the physical relative position from the center of the captured image to the damaged portion and adding the position coordinate as the origin of the captured image to this relative position. Then, the information processing device 300 generates damage data including the absolute position of the damaged portion (step S304). The damage data generating process in step S304 can be executed by the damage data generating unit 338, for example.

[0135] The information processing device 300 according to an embodiment of the present disclosure can generate damage data from a still image captured by the hovering camera 100 by performing the operation shown in Fig. 16, and therefore the information processing device 300 according to an embodiment of the present disclosure can efficiently generate an inspection result of the bridge 1, which is a structure to be inspected. As described above, the damage data may be generated by the control terminal 200, not by the information processing device 300. Therefore, the operation shown in Fig. 16 may be executed by the control terminal 200.

[0136] Fig. 17 is a flow chart showing an example of the operation of the control terminal 200 according to an embodiment of the present disclosure. Fig. 17 shows an example of a process for generating flight information by the control terminal 200 using damage data generated by the information processing device 300. Hereinafter, an example of the operation of the control terminal 200 according to an embodiment of the present disclosure will be described with reference to Fig. 17.

[0137] When the control terminal 200 displays a still image captured by the hovering camera 100 and the location of damage is specified by the user on the still image (step S311), the control terminal 200 acquires the damage position of the specified location from the damage data generated by the information processing device 300 (step S312). There is no restriction on the method of specifying the location of damage, but for example, a still image may be displayed and the user may specify the location of damage by touching the touch panel of the control terminal 200 with a finger or the like.

[0138] When the damage position at the location specified by the user is acquired from the damage data, the control terminal 200 then generates flight information for flying the hovering camera 100 to the damage position acquired from the damage data and capturing an image of the damage position (step S313). The process of step S313 is executed, for example, by the flight information generating unit 232. The flight information generated by the control terminal 200 in this step S313 is for checking the damage position in detail, and therefore may instruct the hovering camera 100 to narrow the interval between image capturing positions or to perform special image capturing at each image capturing position, as compared to the flight information generated by the control terminal 200 described in the description of FIG. 5 above.

[0139] When the flight information is generated in step S313, the control terminal 200 transmits the generated flight information to the hovering camera 100, and the hovering camera 100 performs flight and imaging processing based on the flight information, as shown in steps S104 and S105 of Fig. 5. Then, when the hovering camera 100 completes the imaging processing at the last imaging point, it automatically flies to the base station 600 and returns to the base station 600, and the control terminal 200 acquires the image captured by the hovering camera 100 from the hovering camera 100, as shown in steps S106 and S107 of Fig. 5.

[0140] By executing operations such as those shown in FIG. 17, the control terminal 200 according to one embodiment of the present disclosure can use the damage data generated by the information processing device 300 to generate flight information for causing the hovering camera 100 to capture detailed images of the damaged portion of the bridge girder 3.

[0141] [1.6. Example of flight instructions using synthetic images] When the hovering camera 100 captures moving images or still images at a predetermined interval and transmits them to the control terminal 200, the control terminal 200 can perform a process (real-time stitching process) of synthesizing the moving images captured by the hovering camera 100 for each predetermined frame or the still images captured at a predetermined interval in real time. The control terminal 200 can allow the user to specify the target position of the hovering camera 100 using the synthesized image. The control terminal 200 then transmits flight information for flying to the target position specified by the user to the hovering camera 100.

[0142] The control terminal 200 synthesizes the images in this way, displays the composite image on the display unit 210, and generates flight information for flying to a target position specified for the composite image, allowing the user to generate a flight path for the hovering camera 100 more intuitively. In addition, the control terminal 200 generates a composite image from a moving image or still image captured by the hovering camera 100 and displays the composite image on the display unit 210, so that even in places where it is difficult to obtain an aerial photograph, such as the back side of a bridge, a flight path can be specified using detailed image information and current conditions that cannot be obtained from a map or an aerial photograph, and access to a database of external map information, etc. is not required. In addition, the control terminal 200 generates a composite image from a moving image or still image captured by the hovering camera 100, making it possible to specify a flight path for the hovering camera 100 indoors.

[0143] Fig. 18 is an explanatory diagram showing a state in which the hovering camera 100 is made to capture an image in the direction of the ground while flying the hovering camera 100. Fig. 19 is an explanatory diagram showing a state in which images captured by the hovering camera 100 are synthesized and the user is allowed to specify the flight path of the hovering camera 100 using the synthesized image.

[0144] 19, a current position mark 252 indicating the current position of the hovering camera 100 is superimposed on a composite image 251 obtained as a result of capturing an image from the sky. When generating the composite image 251, the control terminal 200 uses, for example, feature point matching as described below.

[0145] 19 , for example, when the user specifies a target position 253 of the hovering camera 100 on the composite image 251, the control terminal 200 generates flight information for flying the hovering camera 100 from a current position mark 252 to the target position 253 along a flight path 254 and transmits the flight information to the hovering camera 100. The hovering camera 100 flies based on the flight information transmitted from the control terminal 200.

[0146] Fig. 20 is an explanatory diagram showing a state in which the hovering camera 100 is caused to capture an image in the upward direction (the underside of a bridge) while flying the hovering camera 100. Fig. 21 is an explanatory diagram showing a state in which images captured by the hovering camera 100 are synthesized and the user is allowed to specify the flight path of the hovering camera 100 using the synthesized image.

[0147] 21, similarly to Fig. 19, a current position mark 262 indicating the current position of the hovering camera 100 is superimposed on a composite image 261 obtained as a result of capturing an image of the underside of the bridge. When generating the composite image 261, the control terminal 200 uses, for example, feature point matching as described below.

[0148] 21, for example, when the user specifies a target position 263 of the hovering camera 100 on the composite image 261, the control terminal 200 generates flight information for flying the hovering camera 100 from a current position mark 262 to the target position 263 along a flight path 264 and transmits the flight information to the hovering camera 100. The hovering camera 100 flies based on the flight information transmitted from the control terminal 200.

[0149] First, an example of generation of a composite image by the control terminal 200 is shown. Fig. 22 is an explanatory diagram showing a state in which a composite image is generated by the control terminal 200 and displayed on the display unit 210. Note that the process described below may be executed by, for example, the display control unit 234 of the control terminal 200. Therefore, the display control unit 234 can function as an example of an image processing unit of the present disclosure.

[0150] 22 shows a state in which a composite image is generated and displayed by the control terminal 200 using, for example, an image group 271 captured at times t-5 to t-1 and an image 272 captured at time t. When the image 272 captured at time t is transmitted to the control terminal 200, the control terminal 200 compares the feature points of the image 272 captured at time t with the feature points of an image captured at time t-1 from the image group 271. Note that the control terminal 200 may also compare the feature points of the image 272 captured at time t with the feature points of a composite image that has already been composited using the image group 271.

[0151] Next, the control terminal 200 finds rotation and translation parameters (or affine transformation parameters) that minimize the position error of the feature points. Next, the control terminal 200 uses the new image 272 as a reference and synthesizes (α-blending) the synthetic image generated by the previous stitching process by transformation using the rotation and translation parameters (or affine transformation parameters). Through this synthesis, the control terminal 200 can generate a new synthetic image in which the image captured at time t is located at the center of the synthetic image. Then, the control terminal 200 displays the synthetic image so that the center of the synthetic image, i.e., the center 273 of the image captured at time t, is the position of the hovering camera 100.

[0152] Next, a process of generating a flight path by the control terminal 200 based on a user's input to the composite image thus generated will be described.

[0153] 23 is an explanatory diagram for explaining the process of generating a flight path of the control terminal 200 based on a user's input for a composite image. When a user specifies a target position 276 for a composite image 275 generated by the control terminal 200, the direction from the current position of the hovering camera 100 to the target position is known. Since the composite image is generated based on an image captured by the hovering camera 100, the direction to the target position represents the direction from the current hovering camera 100.

[0154] Note that the target position 276 may be a location that has not yet been imaged by the hovering camera 100, specified by the user.

[0155] For example, assume that the imaging device of the hovering camera 100 is provided facing downward so that the top of the captured image is the front of the hovering camera 100 and the left direction is the left direction of the hovering camera 100. Therefore, when the user specifies a target position 276 for the composite image 275 shown in Fig. 23, the target position 276 is in the direction to the left rear as viewed from the current position of the hovering camera 100 (i.e., the center 273).

[0156] Therefore, the control terminal 200 determines a flight path 277 that causes the hovering camera 100 to fly in a direction rearward and to the left as viewed from the current position of the hovering camera 100, generates flight information for horizontal flight along the flight path 277, and transmits the information to the hovering camera 100. The determination of the flight path 277 and the generation of the flight information may be performed by, for example, the flight information generation unit 232. The hovering camera 100 flies horizontally rearward and to the left based on the flight information transmitted from the control terminal 200.

[0157] Then, as the hovering camera 100 starts flying horizontally, the control terminal 200 can obtain a new captured image by the hovering camera 100. The control terminal 200 updates the composite image using a new captured image obtained by the movement of the hovering camera 100. When updating the composite image, the control terminal 200 not only updates the orientation and position of the image, but also updates the target position. By updating the orientation and position of the image and the target position, the target position moves together with the position specified by the user. FIG. 24 is an explanatory diagram showing an example of a composite image after updating using a new captured image obtained by the movement of the hovering camera 100 to the target position 276. In FIG. 24, the position of the center 273 is always the same, and the target position 276 is approaching the center 273, as compared with FIG. 23. That is, in FIG. 24, it can be seen that the target position 276 has moved from that shown in FIG. 23.

[0158] The control terminal 200 generates flight information by applying feedback control so that the target position 276 approaches the center 273 within a predetermined distance by capturing new images and updating the composite image using the hovering camera 100, and transmits the generated flight information to the hovering camera 100.

[0159] In the above example, a method of moving the hovering camera 100 to a target position without rotating the hovering camera 100 has been described, but the present disclosure is not limited to such an example. When a target position is specified, the control terminal 200 may first generate flight information for rotating the hovering camera 100 toward the target position, and then generate flight information for moving the hovering camera 100 toward the target position after rotating the hovering camera 100. Even when rotating the hovering camera 100, the control terminal 200 performs feature point matching as described above, so that the composite image can be rotated in accordance with the rotation of the hovering camera 100.

[0160] The direction of movement in the image differs depending on how the imaging device is attached to the hovering camera 100. Therefore, for example, if the imaging device 101 is attached to the hovering camera 100 facing upward, the direction of movement will be reversed from when it is attached facing downward.

[0161] Of course, it is also possible to install the imaging device 101 laterally on the hovering camera 100. When the imaging device 101 is installed laterally on the hovering camera 100, the movement direction of the hovering camera 100 is a combination of up and down and rotation, or up and down and left and right. FIG. 25 is an explanatory diagram showing an example of a composite image when the imaging device 101 is installed laterally on the hovering camera 100. FIG. 25 shows a composite image 281 generated from images captured by the hovering camera 100. Also, the part surrounded by the reference symbol 282 shows an image captured by the hovering camera 100 from the current position.

[0162] Based on the above description, an operation example of the control terminal 200 according to an embodiment of the present disclosure will be described.

[0163] 26 is a flow chart showing an example of the operation of the control terminal 200 according to an embodiment of the present disclosure. The example shown in FIG. 26 shows an example of the operation of the control terminal 200 when generating a composite image from an image captured by the hovering camera 100, generating flight information based on the composite image, and sending the flight information to the hovering camera 100.

[0164] The control terminal 200 first performs an initialization process in the display control unit 234 (step S401). This initialization process is a process in which the captured image first sent from the hovering camera 100 is used as an input image, and the input image is used as the first composite image.

[0165] The control terminal 200 then uses the image captured by the hovering camera 100 as an input image and extracts feature points from the input image and the composite image in the display control unit 234 (step S402). After extracting the feature points, the control terminal 200 then matches the extracted feature points and calculates the amount of movement and rotation between the feature points in the display control unit 234 (step S403).

[0166] The control terminal 200 then converts the composite image in the display control unit 234 in accordance with the amount of movement and rotation between the feature points calculated in step S403 (step S404). If a target position has already been specified during this conversion, the control terminal 200 also converts the target position in the display control unit 234.

[0167] The control terminal 200 then composites the input image with the previous composite image to generate a new composite image, and the display control unit 234 performs display processing of the composite image (step S405).

[0168] The control terminal 200 then determines with the display control unit 234 whether or not there has been a user input to the display unit 210 on which the composite image is displayed (step S406). The user input to the display unit 210 on which the composite image is displayed is, for example, a user input to a touch panel provided on the display unit 210. If there has been a user input to a touch panel provided on the display unit 210, the control terminal 200 detects, for example, the display control unit 234, the coordinate position on the image input by the user. If there has been a user input to the composite image (step S406, Yes), the control terminal 200 performs a process of registering the user's touch position as a target position with the display control unit 234 (step S407). If there has been no user input to the composite image (step S406, No), the control terminal 200 skips the process of step S407.

[0169] The control terminal 200 then determines whether or not the target position has been registered using the display control unit 234 (step S408). If the target position has been registered (step S408, Yes), the control terminal 200 then determines the direction from the current position to the target position using the display control unit 234 (step S409).

[0170] After determining the direction from the current position to the target position, the control terminal 200 then converts the direction on the image into the movement direction of the body of the hovering camera 100 in accordance with the attachment of the imaging device 101 (step S410). Then, the control terminal 200 generates a command for moving in the converted movement direction as flight information in the flight information generating unit 232, and sends it to the hovering camera 100 (step S411).

[0171] After transmitting the flight information to the hovering camera 100, the control terminal 200 returns to the feature point extraction process of step S402 above.

[0172] On the other hand, if it is determined in step S408 that the target position has not been registered (step S408, No), the control terminal 200 skips steps S409 to S411 and returns to the feature point extraction process in step S402.

[0173] By executing the above-mentioned processing, the control terminal 200 according to one embodiment of the present disclosure can generate a composite image by combining images captured by the hovering camera 100 at any time, and can also generate flight information for flying to a specified position for the composite image and send it to the hovering camera 100.

[0174] In the above-described example, the control terminal 200 generates flight information for controlling the flight of the hovering camera 100 in response to touch processing on the composite image, but the present disclosure is not limited to such an example.

[0175] For example, the control terminal 200 may generate flight information for controlling the flight of the hovering camera 100 in response to a pinch-in operation to reduce the composite image, a pinch-out operation to enlarge the composite image, or a rotation operation to rotate the composite image.

[0176] That is, when a pinch-in operation is performed on the composite image, the control terminal 200 may generate flight information to move the hovering camera 100 away from the imaging target. When a pinch-in operation is performed on the composite image, the control terminal 200 may generate flight information to move the hovering camera 100 closer to the imaging target. When a rotation operation is performed on the composite image, the control terminal 200 may generate flight information to rotate the imaging device 101 of the hovering camera 100 while facing the imaging target.

[0177] In addition, in the above-described example, the control terminal 200 performs synthesis processing on the images captured by the hovering camera 100 and sends flight instructions from the control terminal 200 to the hovering camera 100. However, it goes without saying that the technology of this embodiment can be applied to the control of any moving object, such as a robot equipped with an imaging device.

[0178] In the above example, the control terminal 200 performs synthesis processing on the images captured by the hovering camera 100, and sends flight instructions from the control terminal 200 to the hovering camera 100. In this case, an image captured by the hovering camera 100 is transmitted from the hovering camera 100 to the control terminal 200 every time an image is captured, and flight information generated by the control terminal 200, i.e., information that allows the hovering camera 100 to directly interpret the direction of movement, is transmitted from the control terminal 200 to the hovering camera 100 every time the flight information is generated and updated.

[0179] However, the present disclosure is not limited to such an example. For example, the hovering camera 100 may extract parameters by image processing of the image captured by the hovering camera 100 and generate commands to control the flight of the hovering camera 100, and the control terminal 200 may only receive a synthesis process for the image captured by the hovering camera 100 and an input of a target position from the user. In this case, the image captured by the hovering camera 100 is transmitted from the hovering camera 100 to the control terminal 200 every time the image is captured, and information on the target position designated by the user is transmitted from the control terminal 200 to the hovering camera 100 every time the user designates the target position.

[0180] The hovering camera 100 extracts parameters by image processing of images captured by the hovering camera 100 and generates commands to control the flight of the hovering camera 100, so that feedback control is completed only within the hovering camera 100. This makes it possible to reduce the amount of communication required for exchanging information between the control terminal 200 and the hovering camera 100, and to fly the hovering camera 100 safely even if communication between the control terminal 200 and the hovering camera 100 is cut off.

[0181] In the above example, the composite image is generated so that the image most recently captured by the hovering camera 100 is located at the center of the screen, but the present disclosure is not limited to this example. That is, the display position of the composite image may be fixed, and the image captured by the hovering camera 100 may be composited into the composite image. When the display position of the composite image is fixed, the target position is fixed, and the current position of the hovering camera 100 moves with the update of the composite image. When the display position of the composite image is fixed, when the composite image reaches the edge of the screen due to the update of the composite image, the entire composite image may be scrolled so that the most recently captured image fits into the screen, and the current position and the target position may be updated.

[0182] The imaging device 101 may be attached to the hovering camera 100 so that the imaging direction is fixed, or may be attached to the hovering camera 100 so that the imaging direction is changed by, for example, a motor or the like. When the imaging device 101 is attached to the hovering camera 100 so that the imaging direction is changeable, the imaging direction of the imaging device 101 may be detected by, for example, the control unit 110, and a process of generating a composite image or a process of specifying the moving direction of the hovering camera 100 may be performed according to the imaging direction detected by the control unit 110.

[0183] <2. Summary> As described above, according to one embodiment of the present disclosure, there is provided a hovering camera 100 that flies automatically based on set flight information and captures images of a structure to be inspected, and an inspection system 10 that can check the damage state of the structure based on still images captured by the hovering camera 100.

[0184] The inspection system 10 according to an embodiment of the present disclosure uses information about the structure to be inspected when the control terminal 200 generates flight information to be transmitted to the hovering camera 100. By using the information about the structure to be inspected, the control terminal 200 can generate flight information for flying the hovering camera 100 to efficiently inspect the structure to be inspected.

[0185] Further, according to an embodiment of the present disclosure, a control terminal 200 is provided that can generate a composite image from an image captured by the hovering camera 100 and generate flight information for moving the hovering camera 100 by inputting the composite image. The control terminal 200 according to an embodiment of the present disclosure generates a composite image from an image captured by the hovering camera 100 and generates flight information for moving the hovering camera 100 by inputting the composite image, thereby enabling the user to intuitively operate the hovering camera 100. Therefore, the control terminal 200 according to an embodiment of the present disclosure enables the user to operate the hovering camera 100 more easily without forcing the user to perform complicated operations.

[0186] In the above embodiment, the image captured by the hovering camera 100 is a still image, and an example of the inspection system 10 in which the still image is used to inspect the damage state of the bridge 1 is shown, but the present disclosure is not limited to this example. The hovering camera 100 may capture the bridge 1 as a moving image while flying, and the information processing device 300 may generate damage data using the moving image captured by the hovering camera 100. The hovering camera 100 periodically acquires position information when performing the imaging process of the moving image, and by linking the imaging time of the moving image with the acquisition time of the position information, the information processing device 300 can generate damage data using the moving image.

[0187] The steps in the process performed by each device in this specification do not necessarily have to be processed in chronological order according to the order described in the sequence diagram or flowchart. For example, the steps in the process performed by each device may be processed in an order different from the order described in the flowchart, or may be processed in parallel.

[0188] It is also possible to create a computer program for causing hardware such as a CPU, ROM, and RAM built into each device to perform functions equivalent to those of the configuration of each device described above. It is also possible to provide a storage medium storing the computer program. It is also possible to realize a series of processes by hardware or hardware circuits by configuring each functional block shown in the functional block diagram with hardware or hardware circuits. A part or all of each functional block shown in the functional block diagram used in the above description may be realized by a server device connected via a network such as the Internet. The configuration of each functional block shown in the functional block diagram used in the above description may be realized by a single device or may be realized by a system in which multiple devices cooperate with each other. A system in which multiple devices cooperate with each other may include, for example, a combination of multiple server devices, a combination of a server device and a terminal device, etc.

[0189] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0190] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0191] Note that the following configurations also fall within the technical scope of the present disclosure. (1) generating an operation image from an image captured by a moving object having an imaging device; generating movement information for moving the moving object in response to an operation on the generated image for operation; A control method comprising: (2) The generating of the movement information generates movement information for moving the moving object to a location based on a designated position designated for the operation image. The control method described in (1) above. (3) The generating of the movement information includes determining a movement direction of the moving body based on an installation state of the imaging device in the moving body when generating the movement information. The control method described in (2) above. (4) The generating of the movement information includes generating movement information for changing the orientation of the moving body so as to face a front toward a location based on the specified position and then moving the moving body to the location. The control method according to (2) or (3). (5) The generating of the movement information includes generating movement information for moving the moving object so as to approach an imaging target of the imaging device based on an enlargement process for the operation image. The control method according to any one of (1) to (4). (6) The generating of the movement information includes generating movement information for moving the moving object so as to move away from an imaging target of the imaging device based on a reduction process for the operation image. The control method according to any one of (1) to (5) above. (7) The generating of the movement information includes generating movement information for moving the moving body so that the imaging device rotates while facing the imaging target, based on a rotation process for the operation image. The control method according to any one of (1) to (6). (8) The generating of the image for operation generates the image for operation so that the center of the image most recently captured by the moving object is positioned at the center of the screen. The control method according to any one of (1) to (7) above. (9) The control method according to any one of (1) to (8), wherein the moving object is a flying device. (10) an image processing unit that generates an image for operation from an image captured by a moving object having an imaging device; a movement information generating unit that generates movement information for moving the moving object in response to an operation on the operation image generated by the image processing unit; A control device comprising: [Explanation of symbols]

[0192] 10 Inspection System 100 Hovering Camera 101 Imaging device 104a~104d rotor 108a~108d Motor 110 Control section 120 Communications Department 130 Sensor section 132 Location information acquisition unit 140 Storage section 150 Battery 200 Control terminal 300 Information processing device 400 Wireless Relay Nodes 500 Location Estimation Nodes 600 Base Station 700 Charging Station

Claims

1. generating movement information for moving the moving object in response to an operation of designating an area on an operation image generated from a captured image; and detecting a damaged portion of the structure from an image captured by the moving body that has moved based on the generated movement information and that includes the structure in at least a part of the image. Control methods.

2. The operation image is generated from an image captured with a constant distance from the structure. The control method according to claim 1 .

3. The operation image is a composite image generated based on a plurality of images captured by the moving object. The control method according to claim 1 .

4. The operation of specifying the area includes selecting a span between supports of a structure to be inspected by the moving object. The control method according to claim 1 .

5. The operation of designating the area includes tracing on the display screen. The control method according to claim 1 .

6. The generated movement information includes a movement path that repeatedly rises and falls relative to the imaging target surface, or travels back and forth across the imaging target surface. The control method according to claim 1 .

7. The generated movement information includes information on the positions at which the imaging target is imaged at equal intervals. The control method according to claim 1 .

8. The generated movement information includes information that the distance to the imaging target surface is constant. The control method according to claim 1 .

9. The detection of the damaged portion is performed by image processing technology. The control method according to claim 1 .

10. Calculating damage position information by detecting the damaged portion. The control method according to claim 1 .

11. The damage position information is calculated by calculating a relative value to the position information of the moving body when the image is captured. The control method according to claim 10.

12. The damage position information includes any one of the imaging position of the image, the imaging position of the image and a relative value of the damaged part, the imaging position and relative value of the four corners that are the basis of the operation image at the time of imaging, or an absolute value consisting of latitude, longitude, and altitude. The control method according to claim 10.

13. Transmitting the calculated damage location information to a server device; The control method according to claim 10.

14. a movement information generating unit that generates movement information for moving the moving object in response to an operation of designating an area for an operation image generated from a captured image; a damaged portion detection unit that detects a damaged portion of a structure from an image captured by the moving body that has moved based on the generated movement information and that includes the structure in at least a portion of the image; Equipped with Control device.

15. generating movement information for moving the moving object in response to an operation of designating an area on an operation image generated from a captured image; detecting a damaged portion of the structure from an image captured by the moving body that has moved based on the generated movement information and that includes the structure in at least a portion of the image; to cause a computer to execute program.

Images