Control method, control device, and program

Abstract

To provide a control method, a control device and a program that can intuitively instruct a moving object to move using an image, and can detect a damaged portion of a structure to be inspected using an image captured by the moving object.SOLUTION: A control method includes: generating movement information for moving a moving object in response to an operation to designate an area for an operation image generated from a captured image; and detecting a damaged portion of a structure from an image including the structure in at least a portion of the image, which is captured by the moving object which moves based on the generated movement information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a control method, a control apparatus, and a program.

Background Art

[0002] A technique relating to a photography method in which a camera is mounted on a wirelessly controllable aircraft to capture images with the camera is disclosed (see, for example, Patent Document 1). Mounting a camera on an aircraft makes it possible to capture photographs from the sky or from locations where a tripod cannot be set up. In addition, capturing images with a camera mounted on an aircraft provides various advantages compared to using a real airplane or helicopter, such as lower cost, safer imaging, enabling imaging even at low altitudes and in narrow spaces, and enabling imaging by approaching a target.

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Summary of Invention

Problem to be Solved by the Invention

[0004] Operation of a moving body such as an aircraft or a robot equipped with such a camera normally requires a dedicated controller. Here, if a movement instruction for the moving body can be easily specified by a user using an image captured by the moving body, it is considered that even a user unaccustomed to operating the moving body can easily move the moving body to a target position.

[0005] Accordingly, the present disclosure proposes a new and improved control method, control apparatus, and program that allow a user to intuitively instruct a moving body to move using an image, and also enable detection of a damaged portion of an inspection target structure using an image captured by the moving body.

Means for Solving the Problem

[0006] According to this disclosure, a control method is provided which includes generating movement information for moving a moving body in response to an operation that specifies a region on an operation image generated from an captured image, and detecting a damaged portion of a structure from an image captured by the moving body after it has moved based on the generated movement information, the image which includes the structure in at least a portion of the image.

[0007] Furthermore, according to this disclosure, a control device is provided, comprising: a movement information generation unit that generates movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image; and a damage detection unit that detects a damaged portion of a structure from an image captured by the moving object that has moved based on the generated movement information, the image which includes a structure in at least a part of the image.

[0008] Furthermore, this disclosure provides a program that causes a computer to perform the following actions: generate movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image; and detect damaged portions of a structure from an image captured by the moving object after it has moved, which includes the structure in at least a portion of the image. [Effects of the Invention]

[0009] As described above, this disclosure provides a novel and improved control method, control device, and program that enable intuitive movement instructions for a moving object using images, and the detection of damaged parts of a structure to be inspected using images captured by the moving object.

[0010] Furthermore, the effects described above are not necessarily limited, and any of the effects described herein, or any other effects that can be inferred from this specification, may be achieved in conjunction with or in lieu of the effects described above. [Brief explanation of the drawing]

[0011] [Figure 1] This is an explanatory diagram illustrating an overview of one embodiment of the present disclosure. [Figure 2] This is an explanatory diagram showing an example of the system configuration of an inspection system 10 according to one embodiment of the present disclosure. [Figure 3] This is an explanatory diagram showing an example of the functional configuration of a hovering camera 100 according to one embodiment of the present disclosure. [Figure 4] This is an explanatory diagram showing an example of the functional configuration of a control terminal 200 according to one embodiment of the present disclosure. [Figure 5] This flowchart shows an example of the operation of an inspection system 10 according to one embodiment of this disclosure. [Figure 6] This is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. [Figure 7] This is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. [Figure 8] This is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. [Figure 9] This is a conceptual diagram illustrating how the underside of bridge 1 is imaged by the hovering camera 100. [Figure 10] This is a conceptual diagram illustrating the operation of a hovering camera 100 in an inspection system 10 according to one embodiment of the present disclosure. [Figure 11] This is a conceptual diagram illustrating the operation of a hovering camera 100 in an inspection system 10 according to one embodiment of the present disclosure. [Figure 12] This is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. [Figure 13] This is an explanatory diagram showing an overview of how to inspect the bottom surface of bridge girder 3. [Figure 14] This is an explanatory diagram showing an example of an image 20 obtained by stitching together still images captured by the hovering camera 100. [Figure 15] This is an explanatory diagram showing an example of the functional configuration of an information processing device 300 according to one embodiment of the present disclosure. [Figure 16]It is a flowchart showing an operation example of the information processing apparatus 300 according to an embodiment of the present disclosure. [Figure 17] It is a flowchart showing an operation example of the control terminal 200 according to an embodiment of the present disclosure. [Figure 18] It is an explanatory diagram showing a state where the hovering camera 100 is capturing an image in the direction of the ground. [Figure 19] It is an explanatory diagram showing a state where a user is caused to specify the flight path of the hovering camera 100 using a composite image. [Figure 20] It is an explanatory diagram showing a state where the hovering camera 100 is capturing an image in an upward direction (the back surface of a bridge). [Figure 21] It is an explanatory diagram showing a state where a user is caused to specify the flight path of the hovering camera 100 using a composite image. [Figure 22] It is an explanatory diagram showing a state where the control terminal 200 generates and displays a composite image. [Figure 23] It is an explanatory diagram for explaining the flight path generation processing of the control terminal 200 based on a user's input with respect to a composite image. [Figure 24] It is an explanatory diagram showing an example of a composite image. [Figure 25] It is an explanatory diagram showing an example of a composite image. [Figure 26] It is a flowchart showing an operation example of the control terminal 200 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0013] The description will be given in the following order. 1. An embodiment of the present disclosure 1.1. Overview 1.2. System configuration example 1.3. Example of Functional Configuration 1.4. Example of Operation 1.5. Example of generating damage data 1.5.1 Example Functional Configuration 1.5.2 Example of Operation 1.6. Example of flight instructions using composite images 2. Summary

[0014] <1. One Embodiment of the Disclosure> [1.1. Overview] Before describing in detail one embodiment of this disclosure, we will first provide an overview of one embodiment of this disclosure.

[0015] Human inspection of the condition of structures such as roads, bridges, tunnels, and buildings is essential for their maintenance and management. Typically, such visual inspections involve workers approaching the structure and visually checking for damage such as corrosion or cracks, or loosening of connecting members such as bolts, or using tapping tests to confirm the presence or absence of these abnormalities.

[0016] Maintaining bridges, especially concrete bridges, often requires workers to perform visual and tapping inspections of bridge girders and piers. This may necessitate scaffolding being erected on the underside of piers and girders, and the closure of some or all lanes to ensure worker safety and position work vehicles. These factors can lead to problems not only with the inspection costs themselves, but also with the costs of arranging traffic controllers due to road closures, and the potential for traffic congestion on detours caused by these closures.

[0017] Furthermore, there are bridges that are difficult to inspect, or even impossible to inspect, due to reasons such as being constructed over rivers or the sea. Therefore, considering these circumstances, there is a need for technology that enables safe, traffic-free, and low-cost inspection of structures.

[0018] Therefore, in light of the above circumstances, the Disclosing Parties considered technologies that would enable safe, traffic-free, and low-cost inspection of structures. As described below, the Disclosing Parties have devised a technology that enables safe, traffic-free, and low-cost inspection using an aircraft equipped with an imaging device (in the following description, an aircraft equipped with an imaging device will also be referred to as a "hovering camera").

[0019] Figure 1 is an explanatory diagram illustrating an overview of one embodiment of the present disclosure. Figure 1 schematically shows, for example, a bridge 1 constructed of concrete. When inspecting a concrete bridge 1, as described above, conventionally it was necessary to erect scaffolding on the underside of the bridge piers 2 and bridge girders 3 in order for workers to visually inspect for damage such as cracks and corrosion, or to close off some or all of the lanes to ensure the safety of the workers and to position work vehicles.

[0020] In one embodiment of this disclosure, a hovering camera 100 is used when inspecting a bridge 1. The hovering camera 100 is an aircraft equipped with an imaging device that is configured to fly automatically according to pre-set flight information (in this embodiment, this includes information on the flight path and the position where still images are captured). The information on the position where still images are captured may include, for example, the position where the imaging process is performed, the direction of imaging, and the time to travel to the position where the next imaging process is performed.

[0021] For example, when inspecting the underside (bottom) of bridge girder 3, the hovering camera 100 is automatically flown to capture images of the underside of bridge girder 3. By having the hovering camera 100 capture images of the underside of bridge girder 3, it becomes unnecessary to erect scaffolding on the bridge pier 2 or the underside of bridge girder 3 for inspection of bridge girder 3, and the frequency of road closures is reduced, or road closures become unnecessary altogether. Similarly, when inspecting the side (lateral) of bridge girder 3, the hovering camera 100 is automatically flown to capture images of the side of bridge girder 3. Therefore, by automatically flying the hovering camera 100 and having it capture images of the underside or side of bridge girder 3, it becomes possible to inspect bridge 1 at low cost while ensuring the safety of workers and without affecting traffic.

[0022] In order to have the hovering camera 100 fly automatically and take images of the underside of the bridge girder 3, it is necessary to set the flight path of the hovering camera 100 and the information of the position where still images will be captured at the underside of the bridge girder 3. In one embodiment of this disclosure, the objective is to enable efficient inspection of the bridge 1 by efficiently creating the flight information to be set for the hovering camera 100 using information about the general condition of the bridge 1.

[0023] The above describes an overview of one embodiment of the present disclosure. Next, an example of the configuration of an inspection system according to one embodiment of the present disclosure will be described.

[0024] [1.2. System Configuration Example] Figure 2 is an explanatory diagram showing an example of the system configuration of an inspection system 10 according to one embodiment of the present disclosure. The inspection system 10 according to one embodiment of the present disclosure shown in Figure 2 is a system aimed at efficiently inspecting structures, such as a bridge 1. The system configuration example of the inspection system 10 according to one embodiment of the present disclosure will be described below using Figure 2.

[0025] As shown in Figure 2, an inspection system 10 according to one embodiment of the present disclosure comprises a hovering camera 100, a control terminal 200, an information processing device 300, a wireless relay node 400, a position estimation node 500, a base station 600, a charging station 700, and a server device 800.

[0026] The hovering camera 100 is an example of the imaging device of this disclosure and is an aircraft equipped with an imaging device as described above. The hovering camera 100 is an aircraft configured to fly automatically based on a designated flight path and to capture still images with the imaging device at designated imaging positions. The hovering camera 100 can fly with, for example, four rotors, and by controlling the rotation of each rotor, it can fly while ascending, descending, and moving horizontally. Of course, the number of rotors is not limited to this example.

[0027] The flight path and imaging positions from the start to the end of the flight, as set for the hovering camera 100, are set, for example, as GPS (Global Positioning System) position information. Therefore, the hovering camera 100 may incorporate a GPS receiver that receives radio waves from GPS satellites and calculates its current position. The flight path set for the hovering camera 100 may have all of its latitude, longitude, and altitude set as GPS position information, or only the latitude and longitude may be set as GPS position information, while the altitude may be set, for example, as the relative height from the base station 600 described later.

[0028] The control terminal 200 is an example of the control device of this disclosure and is a terminal that performs flight control of the hovering camera 100. The control terminal 200 performs flight control of the hovering camera 100, such as generating flight information to send to the hovering camera 100, issuing takeoff instructions to the hovering camera 100, issuing return instructions to the base station 600 described later, and controlling the hovering camera 100 if it becomes unable to fly automatically for any reason. The process of generating flight information for the hovering camera 100 by the control terminal 200 will be described in detail later, but here is a brief example.

[0029] When generating flight information for the hovering camera 100, the control terminal 200 reads information about the general condition of the bridge 1 to be inspected, such as a general diagram of the bridge 1, and displays it on the screen. Points on this general diagram of the bridge 1 are pre-associated with points on map data that contain detailed GPS information. It is desirable that this association is performed with at least two sets of points. By pre-associating points on map data that contain detailed GPS information with the general diagram of the bridge 1, the flight path of the hovering camera 100 is defined as GPS values. The control terminal 200 then generates the flight path of the hovering camera 100 based on this general diagram of the bridge 1. The flight path of the hovering camera 100 is displayed overlaid on the general diagram in a way that is easy for the user (construction worker) to understand.

[0030] When generating flight information for the hovering camera 100, the control terminal 200 may take into consideration the structure and dimensions of the bridge 1, as well as the parts of the bridge 1 to be imaged by the hovering camera 100. When generating flight information for the hovering camera 100, the control terminal 200 may also generate flight information that causes the hovering camera 100 to image in detail any parts that are considered to be highly likely to be damaged.

[0031] As described above, the flight path set for the hovering camera 100 may have all of its latitude, longitude, and altitude set as GPS position information. However, there may be cases where altitude data is not available in the overview map of bridge 1. If altitude data is not available in the overview map of bridge 1, the flight path set for the hovering camera 100 may have only its latitude and longitude set as GPS position information, and the altitude may be set as, for example, the relative height from base station 600.

[0032] When the control terminal 200 sets the flight information to be set for the hovering camera 100, it is desirable to generate flight information such that the distance between the hovering camera 100 and the surface to be imaged remains constant when the hovering camera 100 images the bridge 1. By generating flight information such that the distance between the hovering camera 100 and the surface to be imaged remains constant when the hovering camera 100 images the bridge 1, the control terminal 200 can cause the hovering camera 100 to generate an image at the same scale.

[0033] The control terminal 200 is a portable device such as a notebook computer or tablet, and transmits and receives information wirelessly with the hovering camera 100. The control terminal 200 may communicate wirelessly with the hovering camera 100 directly, but in the case of inspection of structures, especially bridges 1, the hovering camera 100 may fly beyond the communication range of the control terminal 200, so communication may also be conducted via a wireless relay node 400 installed during the inspection.

[0034] The control terminal 200 acquires images captured by the imaging device while the hovering camera 100 is in flight and displays them as needed. The control terminal 200 may also acquire and display video footage captured by the imaging device while the hovering camera 100 is in flight via streaming. By acquiring and displaying video footage captured by the imaging device while the hovering camera 100 is in flight via streaming, the control terminal 200 can show the user the current position of the hovering camera 100.

[0035] The information processing device 300 is a device that processes various types of information, and may, for example, be a device that has the function of processing information from personal computers (PCs) or game consoles. In this embodiment, the information processing device 300 is a device that has the function of displaying images captured by the hovering camera 100 to allow the user to confirm the condition of the bridge 1. The information processing device 300 also has the function of calculating the absolute location of damage to the bridge girder 3 from the images captured by the hovering camera 100 and generating the damage data described later. The information processing device 300 may also have the function of transmitting the generated damage data to the server device 800. Note that the control terminal 200 may have the function of calculating the absolute location of damage to the bridge girder 3 from the images captured by the hovering camera 100 and generating the damage data described later.

[0036] The information processing device 300 acquires images captured by the hovering camera 100, for example, from the control terminal 200. The information processing device 300 does not have to acquire the images captured by the hovering camera 100 at a specific timing, but for example, the information processing device 300 may acquire the images captured by the hovering camera 100 from the control terminal 200 at the time when the hovering camera 100 has completed one flight.

[0037] The wireless relay node 400 is a device that relays wireless communication between the hovering camera 100 and the control terminal 200. As described above, during the inspection of structures, particularly bridge 1, the hovering camera 100 may fly beyond the communication range of the control terminal 200. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 may be conducted via the wireless relay node 400 installed during the inspection of the structure. The number of wireless relay nodes 400 is not limited to one; multiple may be installed depending on the inspection range of bridge 1. Therefore, wireless communication between the hovering camera 100 and the control terminal 200 may be conducted via multiple wireless relay nodes 400. The hovering camera 100 can switch its communication destination between the control terminal 200 and the wireless relay node 400 depending on the radio wave conditions.

[0038] The wireless relay node 400 can be installed in an appropriate location on the bridge surface (preferably on the sidewalk) when inspecting the bridge 1. Alternatively, the wireless relay node 400 may be installed so as to be suspended from the railing of the bridge girder 3. Before inspecting the bridge 1, it is desirable to confirm that the wireless relay node 400 is functioning correctly using a predetermined method, for example, by using the control terminal 200.

[0039] The position estimation node 500 is a device that causes the hovering camera 100 to estimate its current position. As described above, the flight path of the hovering camera 100 is set, for example, as GPS position information. In this case, if there is nothing obstructing the radio waves from GPS satellites, the hovering camera 100 can determine its current position with extremely high accuracy. However, it is unavoidable that the hovering camera 100 will go under the bridge girder 3, and if the radio waves from GPS satellites are blocked by the bridge girder 3, or if multipath occurs due to reflection of radio waves by the bridge 1, the hovering camera 100 may not be able to determine its current position with high accuracy.

[0040] Therefore, in this embodiment, a position estimation node 500 is provided under the bridge girder 3 to allow the hovering camera 100 to accurately acquire its current position. As the position estimation node 500, for example, an AR (Augmented Reality) marker may be used, or a GPS signal transmitter may be used.

[0041] When AR markers are used as position estimation nodes 500, in order for the hovering camera 100 to recognize its current position, for example, position estimation nodes 500 are suspended from both ends of the bridge 1, and the hovering camera 100 is made to image the position estimation nodes 500. Then, the hovering camera 100 that has imaged the position estimation nodes 500 is made to fly between the specified position estimation nodes 500. The hovering camera 100 can determine the position between the position estimation nodes 500 by combining, for example, the integrated value from a sensor (for example, an IMU (Inertial Measurement Unit) sensor) installed in the hovering camera 100 and the distance to the destination position estimation node 500 calculated from the captured image. Therefore, by image the position estimation nodes 500, the hovering camera 100 can accurately acquire its current position even under the bridge girder 3.

[0042] Furthermore, if a GPS signal transmitter is used as the position estimation node 500, in order for the hovering camera 100 to recognize its current position, the position estimation nodes 500 are installed, for example, at the diagonals or four corners of the bridge 1. By receiving the GPS signal emitted from the position estimation node 500, the hovering camera 100 can accurately acquire its current position even under the bridge girder 3.

[0043] The base station 600 is a device installed for the takeoff and landing of the hovering camera 100. The base station 600 is equipped with a GPS receiver and calculates its current position by receiving radio waves from GPS satellites. The current position calculated by the base station 600 is sent to the control terminal 200. Once the current position calculated by the base station 600 is sent to the control terminal 200, the control terminal 200 is able to display the position of the base station 600 on the overview diagram of the bridge 1.

[0044] The base station 600 may also have a function to verify the operation of the hovering camera 100. The verification of the hovering camera 100 performed by the base station 600 may include, for example, verification of communication functions, verification of imaging functions, verification of flight functions, and calibration of various sensors. Needless to say, the method of calibrating the sensors of the hovering camera 100 is not limited to the method using the base station 600. For example, a method of calibrating the sensors of the hovering camera 100 may be to fix the hovering camera 100 to a dedicated calibration jig and calibrate the sensors by rotating the hovering camera 100 in the pitch direction or roll direction.

[0045] The charging station 700 charges the secondary battery installed in the hovering camera 100. The hovering camera 100 is powered by a battery and consumes power stored in the battery during flight and imaging. If the battery installed in the hovering camera 100 is a secondary battery, the charging station 700 can restore the power consumed by the hovering camera 100 by charging that battery. The charging station 700 may charge the hovering camera 100 by connecting a cable or the like to the hovering camera 100 and supplying power to the hovering camera 100, or it may charge the hovering camera 100 by supplying power to the hovering camera 100 using a contactless power transmission method.

[0046] The server device 800 is a device for storing various types of data. In this embodiment, the server device 800 may also store the damaged data generated by the information processing device 300.

[0047] An inspection system 10 according to one embodiment of the present disclosure has the configuration shown in Figure 2, which allows the hovering camera 100 to image the bridge 1 and acquire an image of the bridge 1. By having the hovering camera 100 image the bridge 1, the inspection system 10 according to one embodiment of the present disclosure eliminates the need to erect scaffolding on bridge piers and girders, reduces the frequency of closing some or all lanes to ensure the safety of workers, or eliminates the need for road closures altogether, thereby enabling low-cost and efficient inspection of the bridge 1.

[0048] An example of the system configuration of the inspection system 10 according to one embodiment of this disclosure has been described. Next, an example of the functional configuration of the hovering camera 100 and the control terminal 200 that constitute the inspection system 10 according to one embodiment of this disclosure will be described.

[0049] [1.3. Example of Functional Configuration] First, an example of the functional configuration of a hovering camera 100 according to one embodiment of this disclosure will be described. Figure 3 is an explanatory diagram showing an example of the functional configuration of a hovering camera 100 according to one embodiment of this disclosure. Hereinafter, an example of the functional configuration of a hovering camera 100 according to one embodiment of this disclosure will be described using Figure 3.

[0050] As shown in Figure 3, a hovering camera 100 according to one embodiment of the present disclosure comprises an imaging device 101, rotors 104a to 104d, motors 108a to 108d, a control unit 110, a communication unit 120, a sensor unit 130, a position information acquisition unit 132, a storage unit 140, and a battery 150.

[0051] The control unit 110 controls the operation of the hovering camera 100. For example, the control unit 110 can control the adjustment of the rotational speed of the rotors 104a to 104d by adjusting the rotational speed of the motors 108a to 108d, the imaging process by the imaging device 101, the transmission and reception of information with other devices (e.g., the control terminal 200) via the communication unit 120, and the storage and retrieval of information from the storage unit 140.

[0052] In this embodiment, the control unit 110 controls the flight by adjusting the rotational speed of the motors 108a to 108d based on the flight information transmitted from the control terminal 200, and controls the execution of still image acquisition processing on the imaging device 101. By controlling the motors 108a to 108d and the imaging device 101 based on the flight information transmitted from the control terminal 200, the control unit 110 can provide images to the control terminal 200 according to the requests of the control terminal 200.

[0053] The imaging device 101 consists of an image sensor such as a lens, a CCD image sensor, or a CMOS image sensor, a flash, etc. The imaging device 101 provided in the hovering camera 100 performs still image or moving image capture under control from the control terminal 200. The images captured by the imaging device 101 are transmitted from the communication unit 120 to the control terminal 200. In this embodiment, the imaging device 101 also performs imaging processing based on information about the capture position of still images included in the flight information transmitted from the control terminal 200. The images obtained by the imaging processing of the imaging device 101 can be stored in the storage unit 140 or transmitted from the communication unit 120 to the control terminal 200. When the hovering camera 100 is imaging the underside of the bridge 1, it is possible that sunlight will be blocked by the bridge 1 and the brightness will be insufficient, so it is advisable for the hovering camera 100 to emit a flash when imaging the underside of the bridge 1.

[0054] The imaging device 101 can change the imaging direction to any direction, for example, by control from the control unit 110. For example, when the horizontal direction of the hovering camera is set to 0 degrees, it can capture images in an imaging direction represented by a range of ±90 degrees in the vertical direction. Because the imaging device 101 can change the imaging direction, the hovering camera 100 can capture an image in a predetermined direction and provide the captured image to the control terminal 200. The control unit 110 then associates the position information of the hovering camera 100 at the time the imaging device 101 captured the still image (this may include position information obtained by GPS positioning or positioning using the position estimation node 500; positioning using the position estimation node 500 will be described later), aircraft information at the time of imaging (e.g., yaw angle, pitch angle, acceleration, angular velocity), and imaging direction information as metadata for the still image. As for how to store the associated metadata, the metadata may be added to the additional information area of ​​the still image data (e.g., a specific area of ​​the Exif format), or the metadata may be recorded as separate data, such as in a separate file from the image file.

[0055] Rotors 104a to 104d generate lift through rotation, thereby causing the hovering camera 100 to fly. The rotation of rotors 104a to 104d is driven by the rotation of motors 108a to 108d. Motors 108a to 108d rotate rotors 104a to 104d. The rotation of motors 108a to 108d can be controlled by the control unit 110.

[0056] The communication unit 120 performs wireless communication to send and receive information with the control terminal 200. The hovering camera 100 transmits images captured by the imaging device 101 from the communication unit 120 to the control terminal 200. The hovering camera 100 also receives flight instructions from the control terminal 200 via the communication unit 120.

[0057] The sensor unit 130 is a group of devices that acquire the status of the hovering camera 100, and may consist of, for example, an acceleration sensor, a gyro sensor, an ultrasonic sensor, a barometric pressure sensor, an optical flow sensor, a laser rangefinder, etc. The sensor unit 130 can convert the acquired status of the hovering camera 100 into a predetermined signal and provide it to the control unit 110 as needed. The position information acquisition unit 132 acquires information on the current position of the hovering camera 100 using, for example, a GPS or a vision sensor. The position information acquisition unit 132 can provide the acquired information on the current position of the hovering camera 100 to the control unit 110 as needed. The control unit 110 uses the information on the current position of the hovering camera 100 acquired by the position information acquisition unit 132 to perform flight control of the hovering camera 100 based on flight information received from the control terminal 200.

[0058] The sensor unit 130 also detects obstacles that may interfere with flight during flight. When the sensor unit 130 detects an obstacle, the hovering camera 100 can provide information about the detected obstacle to the control terminal 200.

[0059] The memory unit 140 stores various types of information. Examples of information stored by the memory unit 140 include flight information from the hovering camera 100 transmitted from the control terminal 200, and images captured by the imaging device 101.

[0060] Battery 150 stores power to operate the hovering camera 100. Battery 150 may be a primary battery that can only be discharged, or a secondary battery that can also be recharged. If battery 150 is a secondary battery, it may receive power from, for example, the charging station 700 shown in Figure 2.

[0061] A hovering camera 100 according to one embodiment of the present disclosure has the configuration shown in Figure 3, and can automatically fly based on the flight path included in the flight information transmitted from the control terminal 200, and can perform imaging processing based on the information of the still image capture position included in the flight information transmitted from the control terminal 200.

[0062] The above describes an example of the functional configuration of a hovering camera 100 according to one embodiment of the present disclosure using Figure 3. Next, an example of the functional configuration of a control terminal 200 according to one embodiment of the present disclosure will be described.

[0063] Figure 4 is an explanatory diagram showing an example of the functional configuration of a control terminal 200 according to one embodiment of the present disclosure. Hereinafter, an example of the functional configuration of a control terminal 200 according to one embodiment of the present disclosure will be described using Figure 4.

[0064] As shown in Figure 4, a control terminal 200 according to one embodiment of the present disclosure includes a display unit 210, a communication unit 220, a control unit 230, and a storage unit 240.

[0065] The display unit 210 consists of a flat-panel display device such as a liquid crystal display device or an organic EL display device. The display unit 210 can display, for example, images captured by the imaging device 101 or information for controlling the operation of the hovering camera 100. The display unit 210 is equipped with a touch panel, and the user can directly operate the information displayed on the display unit 210 by touching the display unit 210 with their finger or the like.

[0066] The communication unit 220 transmits and receives information wirelessly with the hovering camera 100. The control terminal 200 receives images captured by the imaging device 101 from the hovering camera 100 via the communication unit 220. The control terminal 200 also transmits flight instructions for the hovering camera 100 from the communication unit 220 to the hovering camera 100. Flight instructions for the hovering camera 100 can be generated by the control unit 230.

[0067] The control unit 230 controls the operation of the control terminal 200. For example, the control unit 230 can control the display processing of characters, graphics, images, and other information on the display unit 210, and the transmission and reception of information with other devices (e.g., the hovering camera 100) via the communication unit 220. The control unit 230 also includes a flight information generation unit 232 and a display control unit 234.

[0068] The flight information generation unit 232 generates flight information to be transmitted to the hovering camera 100. When generating flight information, the flight information generation unit 232 uses, for example, information about the structure to be inspected, which is stored in the memory unit 240 described later. Once the flight information generation unit 232 has generated the flight information, it has the communication unit 220 transmit the generated flight information before the hovering camera 100 takes off.

[0069] The process of generating flight information by the flight information generation unit 232 will be described in detail later, but here is a brief explanation of an example of the flight information generation process by the flight information generation unit 232. When generating flight information for the hovering camera 100, the flight information generation unit 232 reads a general overview map of the bridge 1 to be inspected. The read general overview map of the bridge 1 is displayed on the display unit 210 by the display control unit 234. As mentioned above, points on this general overview map of the bridge 1 are pre-associated with points on map data containing detailed GPS information. It is desirable that this association is performed with at least two sets of points. Because points on map data containing detailed GPS information are pre-associated with the general overview map of the bridge 1, the flight path of the hovering camera 100 is defined as GPS values ​​(a pair of latitude and longitude).

[0070] The flight information generation unit 232 then generates a flight path for the hovering camera 100 based on the overview diagram of the bridge 1. When generating the flight path for the hovering camera 100, the flight information generation unit 232 uses information such as the construction method of the bridge 1, structural information such as width and span length, the flight time of the hovering camera 100, and the inspection method of the bridge 1. Concrete bridges are divided into reinforced concrete (RC) and PC (prestressed concrete) depending on the reinforcement method, and into, for example, RCT girder bridges, PCT girder bridges, PC hollow slab bridges, RC box girder bridges, PC box girder bridges, etc. depending on the shape of the girder. Therefore, if the construction method of the bridge 1 to be inspected is known, the flight information generation unit 232 can generate a flight path suitable for the construction method of the bridge 1. The flight information generation unit 232 then displays the flight path of the hovering camera 100 superimposed on the overview diagram of the bridge 1.

[0071] The flight information generation unit 232 defines the flight path of the hovering camera 100 as GPS values ​​(a pair of latitude and longitude) as described above. By defining the flight path of the hovering camera 100 as GPS values, the hovering camera 100 can determine, based on the GPS values, at what position during flight the imaging process should be performed.

[0072] The display control unit 234 controls the display of characters, figures, images, and other information on the display unit 210. In the diagrams referred to in the following description, the characters, figures, symbols, images, and other information displayed on the display unit 210 are controlled by the display control unit 234. For example, when the flight information generation unit 232 generates flight information to be transmitted to the hovering camera 100, the display control unit 234 executes control to display a general diagram of the structure to be inspected (bridge 1) and the generated flight information on the display unit 210.

[0073] The memory unit 240 stores various types of information. Information stored in the memory unit 240 may include, for example, information about the structure to be inspected (bridge 1). This information may include, for example, a general diagram of the structure (bridge 1) and its construction method. Furthermore, if areas of the structure prone to damage are known in advance, the information about the structure may include information about parts that are likely to be damaged.

[0074] Furthermore, the control terminal 200 does not need to pre-store information about the structure to be inspected (bridge 1) in the storage unit 240; it may receive this information from, for example, the information processing device 300 when the structure is being inspected.

[0075] A control terminal 200 according to one embodiment of this disclosure has the configuration shown in Figure 4, which generates flight information to be transmitted to the hovering camera 100 based on information about the structure to be inspected (bridge 1), and the hovering camera 100, which flies based on the flight information, can acquire images captured based on the flight information.

[0076] The above describes an example of the functional configuration of a control terminal 200 according to one embodiment of the present disclosure using Figure 4. Next, an example of the operation of an inspection system 10 according to one embodiment of the present disclosure will be described.

[0077] [1.4. Example of Operation] Figure 5 is a flowchart showing an example of operation of an inspection system 10 according to one embodiment of the present disclosure. Figure 5 shows an example of operation of the inspection system 10 according to one embodiment of the present disclosure when inspecting a bridge 1 by flying a hovering camera 100 and having the hovering camera 100 image the bridge 1. It is assumed that when inspecting the bridge 1 using the hovering camera 100, a wireless relay node 400 and a position estimation node 500 are pre-installed at appropriate locations on the bridge 1. The operation example of the inspection system 10 according to one embodiment of the present disclosure will be explained below using Figure 5.

[0078] The control terminal 200, which generates flight information for the hovering camera 100, reads information about the bridge 1, including a general overview diagram of the bridge 1 to be inspected, and displays the general overview diagram of the bridge 1 on the display unit 210 (step S101). The reading of information about the bridge 1 is performed, for example, by the flight information generation unit 232, and the display of the general overview diagram of the bridge 1 on the display unit 210 is performed, for example, by the display control unit 234. The control terminal 200, which is displaying the general overview diagram of the bridge 1 on the display unit 210, prompts the user to specify the area of ​​the bridge 1 to be inspected using the general overview diagram of the bridge 1 displayed on the display unit 210 (step S102). The process of prompting the user to specify in step S102 is performed, for example, by the flight information generation unit 232.

[0079] For example, if a portion of bridge 1 is to be inspected, the control terminal 200 prompts the user to specify the area to be inspected in the overview diagram of bridge 1 displayed on the display unit 210. Alternatively, if the entire bridge 1 is to be inspected, the control terminal 200 prompts the user to specify the area of ​​the entire bridge 1 in the overview diagram of bridge 1 displayed on the display unit 210.

[0080] Figure 6 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. Figure 6 shows an example of a screen displayed on the display unit 210 when the user is asked to specify the area of ​​the bridge 1 to be inspected in step S102. Figure 6 shows the screen displayed on the display unit 210 when a bridge girder is specified as the area of ​​the bridge 1 to be inspected. The control terminal 200 has an input unit (not shown), such as a touch panel, and the user can specify the area of ​​the bridge 1 by, for example, having the user trace the screen or select the span to be inspected. Of course, the method of having the user specify the area of ​​the bridge 1 to be inspected is not limited to this example. Also, the display of the area specified by the user is not limited to the example shown in Figure 6.

[0081] Figure 6 also shows how the mark B1 indicating the position of base station 600 is superimposed on the overview diagram of bridge 1. As described above, base station 600 is equipped with a GPS receiver and can calculate its current position by receiving radio waves from GPS satellites. Therefore, the control terminal 200 can superimpose the mark B1 indicating the position of base station 600 onto the overview diagram of bridge 1 based on the current position information calculated by base station 600.

[0082] After the user specifies the area of ​​bridge 1 to be inspected, the control terminal 200 then generates flight information for the hovering camera 100 in the area to be inspected, based on the information about bridge 1 (step S103). The flight information generation process in step S103 is performed, for example, by the flight information generation unit 232.

[0083] When the control terminal 200 generates flight information for the hovering camera 100 in step S103, it uses information such as the construction method of the bridge 1, structural information such as width and span length, the flight time of the hovering camera 100, and the inspection method of the bridge 1. For example, if a T-girder is used as the construction method of the bridge 1, the control terminal 200 generates flight information for a flight path in which the hovering camera 100 repeatedly rises and falls on the bottom side of the bridge 1. The control terminal 200 may also use information about the surface of the bridge 1 to be imaged when generating flight information for the hovering camera 100 in step S103. For example, if the user selects to image the side of the bridge 1, the control terminal 200 generates flight information for a flight path along the side of the bridge 1, and if the user selects to image the bottom of the bridge 1, the control terminal 200 generates flight information for a flight path that goes back and forth on the bottom side of the bridge 1.

[0084] An example of flight information generated by the control terminal 200 is described below. The flight information may be specified, for example, as a list for each location where imaging processing is performed, in the following format. ID: (Relative coordinates of the imaging point, imaging direction, velocity during imaging, travel time to the next imaging point, etc.) The relative coordinates of the imaging point are specified using three points: the X, Y, and Z axes. The X axis represents latitude, the Y axis represents longitude, and the Z axis represents height. Other information may include, for example, information for controlling special imaging techniques. This information may include, for example, information for imaging in multiple directions at the same location, information on parameters for bracket imaging (imaging at the same location and imaging direction with different exposures, shutter speeds, ISO sensitivities, etc.), and information on the infrared wavelength during imaging. Following this format, the flight information generated by the control terminal 200 may consist of a list of values ​​like the following. 0:(0,0,0,0,0,2,1.0) 1:(5,0,0,0,0,2,1.0) 2:(7,0,0,0,0,2,1.0) 3:(9,0,0,0,0,2,1.0) The imaging points included in the flight information generated by the control terminal 200 may be specified using relative coordinates from an arbitrary location, such as the absolute coordinates of the base station 600 or the first imaging position, as the reference point. The hovering camera 100 may convert the relative coordinates from the absolute coordinates of the reference point to absolute coordinates and refer to the converted coordinates during flight. Alternatively, the imaging points included in the flight information generated by the control terminal 200 may be specified using absolute coordinates instead of relative coordinates. Furthermore, predetermined values ​​may be stored in the information controlling special imaging included in the flight information generated by the control terminal 200. For example, values ​​such as 1: imaging in multiple imaging directions, 2: bracket imaging (change in shutter speed), 3: bracket imaging (change in ISO sensitivity), etc. may be stored in the information controlling special imaging. For example, the control terminal 200 may include information controlling special imaging in the flight information for locations on the bridge girder 3 that are considered prone to damage, which are stored in the memory unit 240.

[0085] When the control terminal 200 performs the flight information generation process in step S103, it may generate flight information such as causing the hovering camera 100 to image the underside of the bridge girder 3 of the bridge 1 at equal intervals. Therefore, when the control terminal 200 performs the flight information generation process in step S103, it may generate flight information such that the positions where still images are captured are at equal intervals.

[0086] When the control terminal 200 generates flight information for the hovering camera 100 in step S103, if information about parts that are likely to be damaged is already stored in the storage unit 140, it may read that stored information and generate flight information that causes the hovering camera 100 to take detailed images of those parts. When causing the hovering camera 100 to take images of parts that are likely to be damaged, the control terminal 200 may include information to control the special imaging described above in the flight information. Of course, information about parts that are likely to be damaged does not have to be stored in the storage unit 140 in advance; in that case, the user may input information about parts that are likely to be damaged during inspection.

[0087] When flying the hovering camera 100 over the area of ​​bridge 1 to be inspected, it is possible that the hovering camera 100 may not be able to fly over the entire area in one go, depending on its flight time. The flight time of the hovering camera 100 can be determined in advance from the capacity of the battery 150, the power consumption of the motors 108a to 108d that move the rotors 104a to d, the power consumption of the imaging device 101, the control unit 110, the communication unit 120, etc. When generating flight information, it is also possible to estimate the time required for a single inspection flight of the hovering camera 100 from the planned travel time from the starting position (e.g., base station 600) to the first imaging point, the planned travel time between imaging points, and the planned travel time from the last imaging point back to the starting position. Therefore, if the hovering camera 100 cannot fly the entire flight path over the area of ​​bridge 1 to be inspected in a single inspection flight, the control terminal 200 may divide the generated flight path into several parts.

[0088] Furthermore, when the control terminal 200 generates flight information for the hovering camera 100 in step S103, it may generate multiple flight paths and display those flight paths on the display unit 210. Figure 7 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. Figure 7 shows an example of a state in which multiple flight paths are generated when the flight information for the hovering camera 100 is generated in S103, and the flight paths R1 and R2 are displayed on the display unit 210. The control terminal 200 displays multiple flight paths on the display unit 210 and allows the user to select one flight path. The control terminal 200 generates flight information based on the user's selection of a flight path.

[0089] After generating flight information for the hovering camera 100 in step S103, the control terminal 200 then transmits the generated flight information to the hovering camera 100 and sends a takeoff instruction to the hovering camera 100 (step S104). The transmission of the generated flight information and the takeoff instruction is performed, for example, by the flight information generation unit 232 through the communication unit 220.

[0090] Figure 8 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. Figure 8 shows an example of a screen displayed on the display unit 210 of the control terminal 200 when a takeoff instruction is sent to the hovering camera 100. The user can send a takeoff instruction from the control terminal 200 to the hovering camera 100 by touching the takeoff instruction button 211 displayed on the display unit 210. When sending a takeoff instruction from the control terminal 200 to the hovering camera 100, the flight information generated in step S103 above may be sent from the control terminal 200 to the hovering camera 100 before the takeoff instruction is sent, or the flight information generated in step S103 above may be sent from the control terminal 200 to the hovering camera 100 after the takeoff instruction has been sent from the control terminal 200 to the hovering camera 100.

[0091] The hovering camera 100, having received flight information and takeoff instructions from the control terminal 200 and taken off from the base station 600, flies based on the flight information sent from the control terminal 200 and performs imaging processing to obtain a still image (step S105). The hovering camera 100 acquires position information and aircraft information at the time of imaging processing to obtain a still image and associates it with the still image. The aircraft information at the time of imaging processing may include, for example, yaw angle, pitch angle, acceleration, and angular velocity. The hovering camera 100 may also stream video images being captured by the imaging device 101 during flight to the control terminal 200. By acquiring and displaying the video images being captured by the imaging device while the hovering camera 100 is in flight via streaming, the control terminal 200 can show the user the position where the hovering camera 100 is flying.

[0092] When the hovering camera 100 performs imaging processing, it is desirable to maintain a constant distance between the camera and the surface to be imaged (for example, the side or bottom surface of the bridge girder 3) at all imaging points. By maintaining a constant distance between the camera and the surface to be imaged at all imaging points, the hovering camera 100 can obtain still images of the same size.

[0093] If a portion that is likely to be damaged is included in the flight path of the hovering camera 100, the hovering camera 100 may capture multiple still images of that portion by changing the imaging direction of the imaging device, irradiating it with infrared light of a different wavelength, or changing the shutter speed. Also, if a portion that is likely to be damaged is included in the flight path of the hovering camera 100, the hovering camera 100 may make the spacing of the imaging processing positions for that portion narrower than for other parts.

[0094] Figure 9 is a conceptual diagram illustrating the operation of a hovering camera 100 in an inspection system 10 according to one embodiment of the present disclosure. When the hovering camera 100 flies along the underside of the bridge 1 based on flight information, the hovering camera 100 stops, for example at time t1, to image the underside of the bridge 1, flies to a position to take images at time t2, stops at time t2, and images the underside of the bridge 1 at a different position. Thereafter, the flight, stopping, and imaging are repeated until time tn to image the underside of the bridge 1. By repeatedly flying, stopping, and imaging, the hovering camera 100 obtains an image of the underside of the bridge 1.

[0095] The hovering camera 100 can accurately determine its current position if it can receive radio waves from GPS satellites without interference while flying based on flight information. However, in locations where it is difficult to receive radio waves from GPS satellites, such as under a bridge 1, it becomes difficult for the hovering camera 100 to accurately determine its current position. Therefore, in this embodiment, a position estimation node 500 is used to enable the hovering camera 100 to accurately determine its current position even in locations where it is difficult to receive radio waves from GPS satellites.

[0096] Figure 10 is a conceptual diagram illustrating the operation of a hovering camera 100 in an inspection system 10 according to one embodiment of the present disclosure. For example, if a path is set such that the hovering camera 100 flies from Start to Goal in Figure 10, the hovering camera 100 will move between a GPS positioning area 40, where it receives radio waves from GPS satellites 30 without interference to determine its position, and a sensor positioning area 50, where it estimates its current position using, for example, a vision sensor.

[0097] In the GPS positioning area 40, the hovering camera 100 determines its current position using radio waves from GPS satellites 30. In the sensor positioning area 50, the hovering camera 100 determines its position between position estimation nodes 500. If a position estimation node 500 is an AR marker, it determines its current position using the integrated value from a sensor (e.g., an IMU sensor) on the hovering camera 100 and the distance to the destination position estimation node 500 calculated from the image captured by the imaging device 101. If a position estimation node 500 is a GPS signal transmitter, the hovering camera 100 determines its position using the signal transmitted from the position estimation node 500.

[0098] By using the position estimation node 500 in this way, the hovering camera 100 can accurately determine its current position even when it moves to a location where it is difficult to receive radio waves from GPS satellites.

[0099] Once the hovering camera 100 has completed the imaging process at the last imaging point, it automatically flies back to the base station 600 and returns to the base station 600 (step S106). The control terminal 200 then acquires the images captured by the hovering camera 100 after it has returned to the base station 600 (step S107). While the images captured by the hovering camera 100 may be acquired after the hovering camera 100 has returned to the base station 600, the control terminal 200 may also acquire the still images sequentially each time the hovering camera 100 performs imaging processing and obtains a still image.

[0100] In one embodiment of the present disclosure, the inspection system 10, through the operation of the hovering camera 100 and the control terminal 200 as shown in Figure 5, generates flight information to be transmitted to the hovering camera 100 based on information about the structure to be inspected (bridge 1) at the control terminal 200, the hovering camera 100 flies based on the flight information and captures images based on the flight information, and the control terminal 200 can acquire the images captured by the hovering camera 100.

[0101] Furthermore, while the hovering camera 100 is flying, the user may look at the video images captured by the hovering camera 100 and find a part that they want to capture in detail. In that case, for example, the user may operate the control terminal 200 to send a command to the hovering camera 100 to stop automatic flight and switch to manual operation.

[0102] The example described above shows a process in which the control terminal 200 generates flight information, and the hovering camera 100 automatically flies and performs imaging processing based on the generated flight information. However, it is possible that obstacles not visible in the overview diagram of bridge 1 may exist along the flight path.

[0103] Figure 11 is a conceptual diagram illustrating the operation of a hovering camera 100 in an inspection system 10 according to one embodiment of the present disclosure. Figure 11 shows a tree 4 growing beneath a bridge girder 3. This tree 4 is an obstacle that does not appear in the overview diagram of the bridge 1, and its existence may only become apparent when the hovering camera 100 is in flight.

[0104] Therefore, in this embodiment, the hovering camera 100 may be given a test flight based on the flight information generated by the control terminal 200 to check whether there are any obstacles on the flight path included in the flight information.

[0105] When performing a test flight of the hovering camera 100 based on flight information generated by the control terminal 200, the control terminal 200 may stream video images captured by the hovering camera 100 to the user to check whether there are any obstacles on the flight path included in the flight information, or the sensor unit 130 of the hovering camera 100 may be used to detect obstacles. The detailed location of obstacles can be determined by equipping the hovering camera 100 with a stereo camera as its imaging device 101, determining the distance to the obstacle by imaging with the stereo camera, or by identifying the direction of the obstacle by the orientation of the hovering camera 100. If an obstacle is present on the flight path during a test flight of the hovering camera 100, the hovering camera 100 may stop automatic flight and enter a hovering state, waiting for user input, or it may automatically return to the base station 600.

[0106] If it is determined that an obstacle exists on the flight path included in the flight information, the control terminal 200 may register the location of the obstacle in the overview diagram of bridge 1. The location of the obstacle may be manually entered by the user, or if the hovering camera 100 detects an obstacle with the sensor unit 130, the location of the detected obstacle may be obtained from the hovering camera 100 and registered in the overview diagram of bridge 1.

[0107] Figure 12 is an explanatory diagram showing an example of a screen displayed on the display unit 210 of the control terminal 200. Figure 12 shows an example of a screen displayed on the display unit 210 when an obstacle is found to be present on the flight path during a test flight of the hovering camera 100. If an obstacle is found to be present on the flight path during a test flight of the hovering camera 100, the control terminal 200 displays a mark O1 indicating the location of the obstacle overlaid on the overview diagram of the bridge 1.

[0108] Once the location of an obstacle is known, the control terminal 200 regenerates flight information to create a flight path that avoids the obstacle and transmits the regenerated flight information to the hovering camera 100. The hovering camera 100 flies based on the flight information regenerated by the control terminal 200, thereby avoiding the obstacle (tree 4) and performing imaging processing.

[0109] The method of using the hovering camera 100 to determine the location of obstacles is not limited to the example described above. For example, the hovering camera 100 could be flown along a simple path around the outer perimeter of a flight path generated by the control terminal 200, while the imaging device 101 captures moving images, in order to check whether there are any obstacles under the bridge girders 3.

[0110] [1.5. Example of generating damage data] By flying the hovering camera 100 and having it image the bridge 1, it becomes possible to understand the condition of places that are difficult for workers to approach, such as the bottom surface of the bridge girder 3. The still image captured by the hovering camera 100 is associated with information such as the position information of the hovering camera 100 that captured the still image (which may include position information obtained by GPS or positioning using a position estimation node 500), aircraft information at the time of imaging (e.g., yaw angle, pitch angle, acceleration, angular velocity), and imaging direction. Furthermore, by having the hovering camera 100 maintain a constant distance from the surface being imaged at all imaging points, the relative position of the damaged area in the image can be determined. Therefore, if the still image captured by the hovering camera 100 includes a damaged part of the bridge girder 3, it becomes possible to determine the absolute location of that damaged part. For example, by calculating the relative value of the damaged area with the center of the still image as the origin, and then calculating that relative value using the position information of the hovering camera 100 at the time the image was captured, the position information of the damaged area can be determined. The following data, for example, can be recorded as the position information of the damaged area. (1) The location information of the still image is recorded as the location of the damaged area (relative values ​​(offsets) are not recorded). (2) The information of the acquisition position of the still image and the relative value (offset) corresponding to the damaged area are recorded as the position of the damaged area. (3) The absolute values ​​(for example, the capture locations of still images at the four corners where the accuracy of the positional information is considered to be high, as described later, or the coordinates of position estimation node 500) and relative values ​​(offsets) are recorded as the location of the damaged part. (4) Record the calculated absolute values ​​(e.g., latitude, longitude, altitude) as the location of the damaged area.

[0111] A technique is known for determining the physical size of the imaging range by using numerical values ​​such as the focal length of the lens, the size of the image sensor, and the distance to the object being imaged. Therefore, when identifying damaged areas, the physical size of the imaging range of the hovering camera 100 may be estimated using distance information from the hovering camera 100 to the object being imaged (for example, the back or side of the bridge girder 3) and the field of view information of the imaging device 101. The center position of the captured image (the position where the hovering camera 100 took the image) is used as the origin, and the physical relative position from that origin to the damaged area is estimated. By adding the position coordinates of the origin of the captured image to this relative position, the physical position information of the damaged area is determined. Note that this distance information and field of view information may be acquired via the sensors of the hovering camera 100 during imaging and recorded in association with the image, or values ​​pre-set in the hovering camera 100 or imaging device 101 may be used. In addition to imaging position information, distance information, and field of view information, the position information of the damaged area may also be calculated using aircraft information at the time of imaging (e.g., yaw angle, pitch angle, acceleration, angular velocity) and imaging direction information.

[0112] The detection of damaged areas from still images captured by the hovering camera 100 may be performed by the user visually, or it may be performed automatically by an information processing device 300, for example, through image processing. When automating the detection of damaged areas, image processing techniques such as pattern matching may be used.

[0113] The data structure of the damage data is defined, for example, in the following format: (Image ID, Damage ID, Location of damaged area, Coordinates of damaged area on the image, Damage type ID, Damage extent) The damage type ID refers to an ID assigned to each type of damage, such as cracks, spalling, water leakage, and free lime. The damage degree field may record the maximum width of the damage and the length of the damaged portion in the image. From the still images captured by the hovering camera 100, the inspection system 10 according to this embodiment can generate damage data in the above-described format, either through manual input by the user or automatic processing by the information processing device 300. The damage data generated by the inspection system 10 according to this embodiment can then be used in the ordering process for contractors to repair the damage to the bridge 1.

[0114] However, the hovering camera 100 captures numerous still images during a single inspection flight. Therefore, it would be a significant burden for the user to review each still image captured by the hovering camera 100 during the inspection flight.

[0115] The still images captured by the hovering camera 100 are then stitched together to obtain a single image. By stitching the still images captured by the hovering camera 100, for example, the condition of the bottom surface of one span of the bridge girder 3 can be obtained as a single image. By checking the image of the bottom surface of one span of the bridge girder 3 obtained by stitching the still images captured by the hovering camera 100, the user can check whether there is any damage to the bottom surface of the bridge girder 3. Note that the still image stitching process may be performed on the control terminal 200 or on the information processing device 300.

[0116] Figure 13 is an explanatory diagram illustrating the general procedure for inspecting the bottom surface of a bridge girder 3 based on still images captured by a hovering camera 100. A portion of the bottom surface of the bridge girder 3 (for example, the length of one span of the bridge girder 3) is captured by the hovering camera 100, and by stitching together the still images captured by the hovering camera 100, a single image 20 of the bottom surface of the bridge girder 3 is obtained. Reference numeral 21 indicates the image captured in a single imaging process of the hovering camera 100.

[0117] When determining the absolute location of a damaged area from an image obtained by stitching together still images captured by the hovering camera 100, a reference point can be selected from the stitched images that has relatively high accuracy in positional information at the time of capture. Alternatively, the positional information of the hovering camera 100 at the time of capture of the still images at the four corners that form the basis of the stitched image may be used as the reference point. The still images at the four corners that form the basis of the stitched image have the least distortion, and the positional information error is less in the GPS positioning area. Therefore, it is preferable to use the positional information at the time of capture of the four corners that are in or close to the GPS positioning area as the reference point. By determining the absolute location of the damaged area from the positional information of the hovering camera 100 corresponding to the still images at the four corners, it is possible to determine the location of the damaged area with high accuracy. As for the accuracy of each positional information, for example, positioning status information in GPS positioning data (information indicating 2D positioning, 3D positioning, or positioning impossible, or data such as the number of received satellites) may be used.

[0118] Figure 14 is an explanatory diagram showing an example of an image 20 obtained by stitching together still images captured by the hovering camera 100. The centers G1 to G4 of the four corner still images C1 to C4 that form the basis of image 20 correspond to the position of the hovering camera 100 when each still image was captured. In this embodiment, the absolute position of each damaged area in image 20 is calculated using the position information of the hovering camera 100 corresponding to these four corner still images C1 to C4.

[0119] When generating damage data from stitched images, the data structure of the damage data is defined, for example, in the following format. That is, the image ID is removed from the damage data described above. (Damage ID, location information of the damaged area, coordinates of the damaged area on the image, damage type ID, extent of damage) Alternatively, an image ID may be generated for the stitched image, and this image ID may be included in the damage data. From the stitched image, the inspection system 10 according to this embodiment can generate damage data in the format described above, either through manual input by the user or automatic processing by the information processing device 300.

[0120] [1.5.1. Example of Functional Configuration] Figure 15 is an explanatory diagram showing an example of the functional configuration of an information processing device 300 according to one embodiment of the present disclosure. Figure 15 shows an example of the functional configuration of an information processing device 300 according to one embodiment of the present disclosure, which has the function of determining the absolute location of damage to the bridge girder 3 from still images captured by a hovering camera 100 and generating damage data. The example of the functional configuration of an information processing device 300 according to one embodiment of the present disclosure will be described below using Figure 15.

[0121] As shown in Figure 15, an information processing device 300 according to one embodiment of the present disclosure includes a display unit 310, a communication unit 320, a control unit 330, and a storage unit 340.

[0122] The display unit 310 consists of a flat-panel display device such as a liquid crystal display device or an organic EL display device. The display unit 310 can display, for example, images captured by the imaging device 101 of the hovering camera 100, or information regarding damage to the bridge 1 obtained from the images captured by the imaging device 101.

[0123] The communication unit 320 transmits and receives information wirelessly, for example, with the control terminal 200. The information processing device 300 receives the image captured by the hovering camera 100, along with information on the absolute position of the image, from the control terminal 200 to the communication unit 320.

[0124] The control unit 330 controls the operation of the information processing device 300. For example, the control unit 330 can control the display processing of characters, graphics, images, and other information on the display unit 310, and the transmission and reception of information with other devices (e.g., the control terminal 200) via the communication unit 320. The control unit 330 also includes an imaging position information acquisition unit 332, a damage position calculation unit 334, an image synthesis unit 336, and a damage data generation unit 338.

[0125] The imaging position information acquisition unit 332 acquires the imaging position information acquired by the hovering camera 100 when the hovering camera 100 images the bridge 1. The damage position calculation unit 334 detects the damaged portion of the bridge 1 from the image captured by the hovering camera 100 using image processing techniques such as pattern matching, and calculates the absolute position of the damaged portion using the imaging position information acquired by the imaging position information acquisition unit 332.

[0126] The image synthesis unit 336 performs image processing that stitches together still images captured by the hovering camera 100 to generate a single image. When stitching together the still images captured by the hovering camera 100, the image synthesis unit 336 may also use information about the position at which each still image was captured.

[0127] The damage location calculation unit 334 may use information about the imaging positions of the corner images (for example, each of the four corners) in the base image that forms the basis of the image stitched by the image synthesis unit 336 when calculating the damage location. As described above, the still images of the four corners in the base image that forms the basis of the stitched image are considered to have the least distortion, so the damage location calculation unit 334 can determine the damage location more accurately by using information about the imaging positions of the corner images in the base image that forms the basis of the stitched image.

[0128] The damage data generation unit 338 generates the damage data described above using the absolute position of the damaged part of the bridge 1 calculated by the damage position calculation unit 334. The damage data generation unit 338 may generate damage data for each still image, or it may generate one piece of damage data for each image stitched by the image synthesis unit 336.

[0129] The memory unit 340 stores various types of information. The information stored in the memory unit 340 may include, for example, still images captured by the imaging device 101 of the hovering camera 100, information on the absolute imaging position of the hovering camera 100 when the still images were captured, and information on damage data generated by the damage data generation unit 338.

[0130] An information processing device 300 according to one embodiment of this disclosure has the configuration shown in Figure 15, which enables the generation of damage data from still images captured by the hovering camera 100. Therefore, the information processing device 300 according to one embodiment of this disclosure can efficiently generate inspection results for the bridge 1, which is the structure to be inspected. As mentioned above, the damage data may be generated by the control terminal 200 instead of the information processing device 300. Therefore, the configuration of the control unit 330 of the information processing device 300 shown in Figure 15 may be provided by the control terminal 200. Furthermore, the inspection results for the bridge 1, which is the structure to be inspected, can be stored and utilized in public and private databases. As mentioned above, the damage data may be generated by the control terminal 200 instead of the information processing device 300. Therefore, the configuration of the control unit 330 of the information processing device 300 shown in Figure 15 may be provided by the control terminal 200.

[0131] The functional configuration example of the information processing device 300 according to one embodiment of this disclosure has been described above with reference to Figure 15. Next, an example of the operation of the information processing device 300 according to one embodiment of this disclosure will be described.

[0132] [1.5.2. Example of Operation] Figure 16 is a flowchart showing an example of operation of an information processing device 300 according to one embodiment of the present disclosure. Figure 16 shows an example of operation of the information processing device 300 according to one embodiment of the present disclosure when generating damage data by determining the absolute location of damage to the bridge girder 3 from a still image captured by a hovering camera 100. The operation example of the information processing device 300 according to one embodiment of the present disclosure will be described below using Figure 16.

[0133] The information processing device 300 first acquires an image linked to imaging location information, which is captured by the hovering camera 100 while it is flying around the periphery of the bridge 1 (step S301). After acquiring the image linked to imaging location information in step S301, the information processing device 300 then detects the damaged portion from the image using image processing techniques such as pattern matching (step S302). The damaged portion detection process in step S302 can be performed, for example, by the damage location calculation unit 334.

[0134] When a damaged area is detected from the image in step S302, the information processing device 300 then calculates the absolute position of the damaged area (step S303). The calculation process in step S303 can be performed, for example, by the damage position calculation unit 334. The information processing device 300 calculates the absolute position of the damaged area in step S303 based on the imaging position information of the still image captured by the hovering camera 100. When calculating the absolute position of the damaged area, the information processing device 300 may estimate the physical size of the imaging range of the hovering camera 100 based on the distance information from the hovering camera 100 to the imaging target (for example, the back or side of the bridge girder 3) and the field of view information of the imaging device 101. By estimating the physical relative position from the center of the captured image to the damaged area and adding the position coordinates of the origin of the captured image to this relative position, the information processing device 300 can determine the physical position information of the damaged area. The information processing device 300 then generates damage data including the absolute location of the damaged part (step S304). The damage data generation process in step S304 can be performed, for example, by a damage data generation unit 338.

[0135] An information processing device 300 according to one embodiment of the present disclosure can generate damage data from still images captured by a hovering camera 100 by performing the operations shown in Figure 16. Therefore, the information processing device 300 according to one embodiment of the present disclosure can efficiently generate inspection results for the bridge 1, which is the structure to be inspected. As mentioned above, the damage data may be generated by the control terminal 200 instead of the information processing device 300. Therefore, the operations shown in Figure 16 may be performed by the control terminal 200.

[0136] Figure 17 is a flowchart showing an example of the operation of a control terminal 200 according to one embodiment of the present disclosure. Figure 17 shows an example of the flight information generation process by the control terminal 200 using damage data generated by the information processing device 300. The operation example of the control terminal 200 according to one embodiment of the present disclosure will be described below using Figure 17.

[0137] The control terminal 200 displays a still image captured by the hovering camera 100, and when the user specifies the location of the damage on the still image (step S311), the control terminal 200 obtains the location of the damage at the specified location from the damage data generated by the information processing device 300 (step S312). The method of specifying the location of the damage is not limited, but for example, the user may specify the location of the damage by displaying a still image and having them touch the touch panel of the control terminal 200 with their finger.

[0138] After obtaining the location of damage specified by the user from the damage data, the control terminal 200 then generates flight information to fly the hovering camera 100 to the damage location obtained from the damage data and image the damage location (step S313). The processing in step S313 is performed, for example, by the flight information generation unit 232. The flight information generated by the control terminal 200 in step S313 is for detailed confirmation of the damage location, so it may be used to instruct the hovering camera 100 to narrow the interval between imaging locations or to perform special imaging at each imaging location, compared to the flight information generated by the control terminal 200 as described in Figure 5 above.

[0139] Once the flight information is generated in step S313, as shown in steps S104 and S105 of Figure 5, the control terminal 200 transmits the generated flight information to the hovering camera 100, and the hovering camera 100 performs flight and imaging processing based on the flight information. Then, as shown in steps S106 and S107 of Figure 5, once the hovering camera 100 has completed imaging processing at the last imaging point, it automatically flies back to the base station 600 and returns to the base station 600, and the control terminal 200 acquires the images captured by the hovering camera 100 from the hovering camera 100.

[0140] According to one embodiment of the present disclosure, the control terminal 200 can perform the operations shown in Figure 17 to generate flight information for the hovering camera 100 to take detailed images of the damaged portion of the bridge girder 3 using the damage data generated by the information processing device 300.

[0141] [1.6. Example of flight instructions using composite images] When the hovering camera 100 captures moving images or still images at predetermined intervals and transmits them to the control terminal 200, the control terminal 200 can perform a process (real-time stitching) to combine the moving images captured by the hovering camera 100 at predetermined frames or still images captured at predetermined intervals in real time. Using the combined image, the control terminal 200 can allow the user to specify the target position of the hovering camera 100. The control terminal 200 then transmits flight information to the hovering camera 100 to fly to the target position specified by the user.

[0142] The control terminal 200 combines images in this way, displays the combined image on the display unit 210, and generates flight information for flying to a specified target position based on the combined image, thereby enabling the user to generate a more intuitive flight path for the hovering camera 100. Furthermore, by generating a combined image from moving or still images captured by the hovering camera 100 and displaying the combined image on the display unit 210, the control terminal 200 can specify a flight path using detailed image information and current conditions that cannot be obtained from maps or aerial photographs, even in places where it is difficult to obtain aerial photographs, such as the back of a bridge, without requiring access to external databases such as map information. In addition, by generating a combined image from moving or still images captured by the hovering camera 100, the control terminal 200 can also specify a flight path for the hovering camera 100 indoors.

[0143] Figure 18 is an explanatory diagram showing how the hovering camera 100 is capturing images of the ground while it is in flight. Figure 19 is an explanatory diagram showing how the images captured by the hovering camera 100 are combined, and how the user is able to specify the flight path of the hovering camera 100 using the combined image.

[0144] Figure 19 shows a current position mark 252 indicating the current position of the hovering camera 100, superimposed on the composite image 251 obtained from aerial imaging. When generating the composite image 251, the control terminal 200 uses, for example, feature point matching, as will be described later.

[0145] When a user specifies a target position 253 for the hovering camera 100 on the composite image 251, as shown in Figure 19, the control terminal 200 generates flight information for the hovering camera 100 to fly along the flight path 254 from the current position mark 252 to the target position 253, and transmits it to the hovering camera 100. The hovering camera 100 flies based on the flight information transmitted from the control terminal 200.

[0146] Figure 20 is an explanatory diagram showing the hovering camera 100 taking images in the upward direction (the underside of the bridge) while it is in flight. Figure 21 is an explanatory diagram showing how the images taken by the hovering camera 100 are combined, and how the user is able to specify the flight path of the hovering camera 100 using the combined image.

[0147] Figure 21, similar to Figure 19, shows a current position mark 262 indicating the current position of the hovering camera 100, superimposed on the composite image 261 obtained by imaging the underside of the bridge. When generating the composite image 261, the control terminal 200 uses, for example, feature point matching, as will be described later.

[0148] When a user specifies a target position 263 for the hovering camera 100 on the composite image 261, as shown in Figure 21, the control terminal 200 generates flight information for the hovering camera 100 to fly along the flight path 264 from the current position mark 262 to the target position 263, and transmits it to the hovering camera 100. The hovering camera 100 flies based on the flight information transmitted from the control terminal 200.

[0149] First, an example of image generation by the control terminal 200 is shown. Figure 22 is an explanatory diagram showing the generation of an image by the control terminal 200 and its display on the display unit 210. Note that the processing described below may also be executed by, for example, the display control unit 234 of the control terminal 200. Therefore, the display control unit 234 can function as an example of the image processing unit of this disclosure.

[0150] Figure 22 shows how the control terminal 200 generates and displays a composite image using, for example, a group of images 271 taken at times t-5 to t-1 and an image 272 taken at time t. When the image 272 taken at time t is transmitted to the control terminal 200, the control terminal 200 compares the feature points of the image 272 taken at time t with the feature points of the image taken at time t-1 from the group of images 271. The control terminal 200 may also compare the feature points of the image 272 taken at time t with the feature points of a composite image already synthesized using the group of images 271.

[0151] Next, the control terminal 200 determines the rotation / translation parameters (or affine transformation parameters) that minimize the positional error of the feature points. Then, the control terminal 200 uses the new image 272 as a reference and combines (alpha blending) the composite images generated by the previous stitching process using the rotation / translation parameters (or affine transformation parameters). Through this combination, the control terminal 200 can generate a new composite image in which the image captured at time t is at the center of the composite image. Then, the control terminal 200 displays the composite image so that the center of the composite image, i.e., the center 273 of the image captured at time t, is at the position of the hovering camera 100.

[0152] Next, we will explain the process by which the control terminal 200 generates a flight path based on user input to the composite image generated in this way.

[0153] Figure 23 is an explanatory diagram illustrating the process by which the control terminal 200 generates a flight path based on user input to a composite image. When the user specifies a target position 276 to the composite image 275 generated by the control terminal 200, the direction from the current position of the hovering camera 100 to the target position can be determined. Since the composite image is generated based on images captured by the hovering camera 100, its direction to the target position represents the direction from the current position of the hovering camera 100.

[0154] Furthermore, the target position 276 may be specified by the user as a location that has not yet been captured by the hovering camera 100.

[0155] For example, suppose the hovering camera 100 is equipped with an imaging device facing downwards, such that the top of the captured image is in front of the hovering camera 100 and the left direction is to the left of the hovering camera 100. Therefore, when the user specifies a target position 276 for the composite image 275 shown in Figure 23, the target position 276 will be in the direction to the left rear when viewed from the current position of the hovering camera 100 (i.e., the center 273).

[0156] Therefore, the control terminal 200 determines a flight path 277 such that the hovering camera 100 flies in a direction to the left rear from the current position of the hovering camera 100, generates flight information to perform horizontal flight along that flight path 277, and transmits it to the hovering camera 100. The determination of the flight path 277 and the generation of flight information may be performed by, for example, the flight information generation unit 232. Based on the flight information transmitted from the control terminal 200, the hovering camera 100 flies horizontally to the left rear.

[0157] As the hovering camera 100 begins horizontal flight, the control terminal 200 can obtain a new image captured by the hovering camera 100. The control terminal 200 updates the composite image using the new image captured by the movement of the hovering camera 100. When updating the composite image, the control terminal 200 updates not only the orientation and position of the image, but also the target position. As the orientation, position, and target position are updated, the target position moves along with the position specified by the user. Figure 24 is an explanatory diagram showing an example of an updated composite image using a new image captured by the movement of the hovering camera 100 to the target position 276. In comparison with Figure 23, the position of the center 273 remains the same, while the target position 276 is approaching the center 273. In other words, Figure 24 shows that the target position 276 has moved from what is shown in Figure 23.

[0158] The control terminal 200 generates flight information by applying feedback control to the hovering camera 100 to bring the target position 276 closer to the center 273 by updating the newly captured and composited images, and transmits the generated flight information to the hovering camera 100.

[0159] The above example shows a method for moving the hovering camera 100 to the target position without rotation, but this disclosure is not limited to such example. When a target position is specified, the control terminal 200 may first generate flight information to rotate the hovering camera 100 toward the target position, and then generate flight information to move the hovering camera 100 toward the target position after it has been rotated. Even when the hovering camera 100 is rotated, the control terminal 200 performs feature point matching as described above, so the composite image can be rotated in conjunction with the rotation of the hovering camera 100.

[0160] Note that the direction of movement in the image varies depending on how the imaging device is attached to the hovering camera 100. Therefore, for example, if the imaging device 101 is mounted facing upwards on the hovering camera 100, the direction of movement will be reversed compared to when it is mounted facing downwards.

[0161] It is also possible to mount the imaging device 101 laterally on the hovering camera 100. When the imaging device 101 is mounted laterally on the hovering camera 100, the movement direction of the hovering camera 100 will be a combination of up and down and rotation, or up and down and left and right. Figure 25 is an explanatory diagram showing an example of a composite image when the imaging device 101 is mounted laterally on the hovering camera 100. Figure 25 shows a composite image 281 generated from images captured by the hovering camera 100. The area enclosed by reference numeral 282 indicates the image being captured by the hovering camera 100 from its current position.

[0162] Based on the above explanation, an example of the operation of a control terminal 200 according to one embodiment of this disclosure will be described.

[0163] Figure 26 is a flowchart showing an example of the operation of a control terminal 200 according to one embodiment of the present disclosure. Figure 26 shows an example of the operation of the control terminal 200 when generating a composite image from images captured by the hovering camera 100, generating flight information based on the composite image, and sending it to the hovering camera 100.

[0164] The control terminal 200 first performs an initialization process using the display control unit 234 (step S401). This initialization process takes the first captured image sent from the hovering camera 100 as the input image and uses that input image as the first composite image.

[0165] The control terminal 200 then uses the image captured by the hovering camera 100 as the input image, and the display control unit 234 performs feature point extraction on the input image and the composite image (step S402). After feature point extraction, the control terminal 200 then performs matching of the extracted feature points and calculates the amount of movement and rotation between the feature points using the display control unit 234 (step S403).

[0166] The control terminal 200 then uses the display control unit 234 to transform the composite image according to the amount of movement and rotation between the feature points obtained in step S403 (step S404). If a target position has already been specified during this transformation, the control terminal 200 also transforms the target position using the display control unit 234.

[0167] The control terminal 200 then combines the input image with the previously created composite image to create a new composite image, and the display control unit 234 performs the display processing of this composite image (step S405).

[0168] The control terminal 200 then uses the display control unit 234 to determine whether there has been user input to the display unit 210 on which the composite image is displayed (step S406). User input to the display unit 210 on which the composite image is displayed refers to, for example, user input to the touch panel provided on the display unit 210. If there has been user input to the touch panel provided on the display unit 210, the control terminal 200 uses the display control unit 234 to detect the coordinate position on the image entered by the user. If there has been user input to the composite image (step S406, Yes), the control terminal 200 uses the display control unit 234 to register the user's touch position as the target position (step S407). If there has been no user input to the composite image (step S406, No), the control terminal 200 skips the process in step S407.

[0169] The control terminal 200 then uses the display control unit 234 to determine whether a target position has been registered (step S408). If a target position has been registered (step S408, Yes), the control terminal 200 then uses the display control unit 234 to determine the direction of that target position from the current position (step S409).

[0170] Once the direction of the target position from the current position is determined, the control terminal 200 then converts the direction on the image to the direction of movement of the hovering camera 100, in accordance with the mounting of the imaging device 101 (step S410). The control terminal 200 then generates a command to move in the converted direction as flight information using the flight information generation unit 232 and sends it to the hovering camera 100 (step S411).

[0171] When the control terminal 200 sends flight information to the hovering camera 100, it returns to the feature point extraction process of step S402 described above.

[0172] On the other hand, if the result of the determination in step S408 is that the target position has not been registered (step S408, No), the control terminal 200 skips the processing in steps S409 to S411 and returns to the feature point extraction process in step S402.

[0173] In one embodiment of the present disclosure, the control terminal 200 can, by executing the above-described process, continuously synthesize images captured by the hovering camera 100 to generate a composite image, and generate flight information for the composite image to fly to a specified position, and send this information to the hovering camera 100.

[0174] In the example described above, the control terminal 200 generated flight information to control the flight of the hovering camera 100 in response to touch processing on the composite image, but this disclosure is not limited to this example.

[0175] For example, the control terminal 200 may generate flight information that controls the flight of the hovering camera 100 in response to operations such as reducing the size of the composite image by pinching in, enlarging it by pinching out, and rotating the composite image.

[0176] In other words, when a pinch-in operation is performed on the composite image, the control terminal 200 may generate flight information that moves the hovering camera 100 away from the target being imaged. Alternatively, when a pinch-in operation is performed on the composite image, the control terminal 200 may generate flight information that moves the hovering camera 100 closer to the target being imaged. Furthermore, when a rotation operation is performed on the composite image, the control terminal 200 may generate flight information that rotates the imaging device 101 of the hovering camera 100 while keeping it facing the target being imaged.

[0177] Furthermore, in the example described above, the control terminal 200 performed image synthesis processing on the images captured by the hovering camera 100 and sent flight instructions from the control terminal 200 to the hovering camera 100. However, it goes without saying that the technology of this embodiment can be applied to the control of mobile bodies in general, such as robots equipped with imaging devices.

[0178] In the example described above, the control terminal 200 performed image synthesis processing on the images captured by the hovering camera 100 and sent flight instructions from the control terminal 200 to the hovering camera 100. In this case, the hovering camera 100 transmits images captured by the hovering camera 100 to the control terminal 200 each time an image is captured, and the control terminal 200 transmits flight information generated by the control terminal 200, that is, information that allows the hovering camera 100 to directly interpret the direction of movement, to the hovering camera 100 each time the flight information is generated or updated.

[0179] However, this disclosure is not limited to such examples. For example, the hovering camera 100 may perform image processing to extract parameters from the image it captures and generate commands to control the flight of the hovering camera 100, while the control terminal 200 may only perform image synthesis processing on the image captured by the hovering camera 100 and accept input of a target position from the user. In this case, the hovering camera 100 transmits the image it captures to the control terminal 200 each time it captures an image, and the control terminal 200 transmits information about the target position specified by the user to the hovering camera 100 each time the user specifies a target position.

[0180] By having the hovering camera 100 extract parameters through image processing of the images it captures and generate commands to control its flight, the feedback control is completed entirely within the hovering camera 100. Consequently, the amount of communication required for information exchange between the control terminal 200 and the hovering camera 100 is reduced, and the hovering camera 100 can be safely flown even if communication between the control terminal 200 and the hovering camera 100 is interrupted.

[0181] Furthermore, in the example described above, the composite image was generated so that the most recently captured image by the hovering camera 100 was in the center of the screen, but this disclosure is not limited to such an example. That is, the display position of the composite image may be fixed, and images captured by the hovering camera 100 may be composited into the composite image. If the display position of the composite image is fixed, the target position is fixed, and the current position of the hovering camera 100 moves along with the update of the composite image. Also, if the display position of the composite image is fixed, and the composite image reaches the edge of the screen due to the update of the composite image, the entire composite image may be scrolled so that the most recently captured image fits on the screen, and the current position and target position may be updated.

[0182] The imaging device 101 may be attached to the hovering camera 100 so that its imaging direction is fixed, or it may be attached to the hovering camera 100 so that its imaging direction changes, for example, by a motor. If the imaging device 101 is attached to the hovering camera 100 so that its imaging direction changes, the imaging direction of the imaging device 101 may be detected by, for example, the control unit 110, and the composite image generation process or the process of specifying the movement direction of the hovering camera 100 may be performed according to the imaging direction detected by the control unit 110.

[0183] <2. Summary> As described above, according to one embodiment of the present disclosure, a hovering camera 100 that automatically flies based on set flight information and takes images of a structure to be inspected, and an inspection system 10 that can confirm the damage status of the structure based on the still images taken by the hovering camera 100 are provided.

[0184] In one embodiment of the present disclosure, the inspection system 10 uses information about the structure to be inspected when the control terminal 200 generates flight information to be transmitted to the hovering camera 100. By using information about the structure to be inspected, the control terminal 200 can generate flight information to fly the hovering camera 100 and efficiently inspect the structure to be inspected.

[0185] Furthermore, according to one embodiment of the present disclosure, a control terminal 200 is provided that can generate a composite image from images captured by the hovering camera 100 and generate flight information for moving the hovering camera 100 based on input to the composite image. The control terminal 200 according to one embodiment of the present disclosure generates a composite image from images captured by the hovering camera 100 and generates flight information for moving the hovering camera 100 based on input to the composite image, thereby enabling the user to intuitively operate the hovering camera 100. Accordingly, the control terminal 200 according to one embodiment of the present disclosure enables the user to operate the hovering camera 100 more easily without forcing the user to perform complex operations.

[0186] In the above embodiment, the image captured by the hovering camera 100 is a still image, and an example of an inspection system 10 for inspecting the damage state of the bridge 1 using that still image has been shown. However, this disclosure is not limited to this example. The hovering camera 100 may also capture the bridge 1 as a moving image while flying, and the information processing device 300 may generate damage data using the moving image captured by the hovering camera 100. The hovering camera 100 periodically acquires position information when performing the moving image capture process, and by linking the time of moving image capture with the time of acquisition of position information, the information processing device 300 can generate damage data using the moving image.

[0187] Each step in the process performed by each device described herein does not necessarily have to be processed chronologically in the order described in the sequence diagram or flowchart. For example, each step in the process performed by each device may be processed in a different order than that described in the flowchart, or may be processed in parallel.

[0188] Furthermore, it is possible to create computer programs that enable the hardware, such as the CPU, ROM, and RAM, built into each device to perform functions equivalent to those of the devices described above. Storage media containing these computer programs can also be provided. Additionally, a series of processes can be implemented using hardware or hardware circuits by configuring each functional block shown in the functional block diagram with hardware or hardware circuits. Furthermore, some or all of the functional blocks shown in the functional block diagram used in the above description may be implemented in a server device connected via a network such as the Internet. The configuration of each functional block shown in the functional block diagram used in the above description may be implemented in a single device or in a system where multiple devices cooperate. A system where multiple devices cooperate may include, for example, a combination of multiple server devices, or a combination of a server device and a terminal device.

[0189] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical idea set forth in the claims, and these will naturally also fall within the technical scope of the present disclosure.

[0190] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0191] Furthermore, the following configurations also fall within the technical scope of this disclosure. (1) The process involves generating an operational image from an image captured by a mobile device equipped with an imaging device, To generate movement information for moving the moving object in response to operations on the generated operation image, A control method including (2) The generation of the aforementioned movement information involves generating movement information for moving the moving object to a location based on a specified position designated for the operation image. The control method described in (1) above. (3) The generation of the aforementioned movement information involves determining the direction of movement of the moving body based on the installation state of the imaging device on the moving body during the generation of the aforementioned movement information. The control method described in (2) above. (4) The generation of the aforementioned movement information involves changing the orientation of the moving body so that it faces the location based on the specified position, and then generating movement information for moving the moving body to that location. The control method described in (2) or (3) above. (5) The generation of the aforementioned movement information involves generating movement information for moving the moving object so that it approaches the object being captured by the imaging device, based on the magnification process applied to the operating image. The control method described in any one of (1) to (4) above. (6) The generation of the aforementioned movement information involves generating movement information for moving the moving object away from the object being captured by the imaging device, based on the reduction process applied to the operating image. The control method described in any one of (1) to (5) above. (7) The generation of the aforementioned movement information involves generating movement information for moving the moving body so that the imaging device rotates while remaining facing the object being imaged, based on the rotation processing applied to the operating image. The control method described in any one of (1) to (6) above. (8) The process of generating the aforementioned operating image involves generating the operating image such that the center of the most recently captured image by the moving object is located at the center of the screen. The control method described in any one of (1) to (7) above. (9) The control method according to any one of (1) to (8) above, wherein the moving body is a flying device. (10) An image processing unit that generates an operating image from an image captured by a mobile device equipped with an imaging device, A movement information generation unit generates movement information for moving the moving object in response to an operation on the operation image generated by the image processing unit, A control device equipped with the following features. [Explanation of Symbols]

[0192] 10 Inspection System 100 Hovering Cameras 101 Imaging device 104a~104d Rotor 108a~108d Motor 110 Control Unit 120 Communications Department 130 Sensor section 132 Location information acquisition unit 140 Storage section 150 batteries 200 control terminals 300 Information Processing Devices 400 wireless relay nodes 500 location estimation nodes 600 base stations 700 charging stations

Claims

1. To generate movement information for moving a moving object in response to an operation that specifies a region on a control image generated from an captured image, This includes detecting damaged portions of a structure from an image taken by the moving body that has moved based on the generated movement information, the image of which includes the structure in at least a portion of the image, The aforementioned operating image is generated from an image captured while maintaining a constant distance from the structure. Control method.

2. To generate movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image, This includes detecting damaged portions of a structure from an image taken by the moving body that has moved based on the generated movement information, the image of which includes the structure in at least a portion of the image, The operation of specifying the area includes selecting the span, which is the distance between the supports of the structure to be inspected by the mobile body. Control method.

3. To generate movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image, This includes detecting damaged portions of a structure from an image taken by the moving body that has moved based on the generated movement information, the image of which includes the structure in at least a portion of the image, The generated movement information includes a movement path that repeatedly rises and falls relative to the imaging target surface, or moves back and forth along the imaging target surface. Control method.

4. The aforementioned operating image is a composite image generated based on a plurality of images captured by the moving object. The control method according to claim 1, 2, or 3.

5. The operation of specifying the aforementioned area includes tracing on the display screen. The control method according to claim 1, 2, or 3.

6. The generated movement information includes information that sets the positions for imaging the target to be imaged at equal intervals. The control method according to claim 1, 2, or 3.

7. The generated movement information includes information that keeps the distance from the imaging target surface constant. The control method according to claim 1, 2, or 3.

8. The detection of the damaged portion is performed using image processing technology. The control method according to claim 1, 2, or 3.

9. By detecting the damaged portion, the location information of the damage is calculated. The control method according to claim 1, 2, or 3.

10. The damage location information is calculated by calculating a relative value to the location information of the moving object when the image was captured. The control method according to claim 9.

11. The damage location information includes one of the following: the image acquisition position of the image, the relative value between the image acquisition position and the damaged portion of the image, the position and relative value of the four corners that form the basis of the operation image at the time of acquisition, or an absolute value consisting of latitude, longitude, and altitude. The control method according to claim 9.

12. The calculated damage location information is transmitted to the server device. The control method according to claim 9.

13. A movement information generation unit generates movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image, A damage detection unit detects damaged portions of a structure from an image taken by the moving body that has moved based on the generated movement information, the image which includes the structure in at least a portion of the image; Equipped with, The aforementioned operating image is generated from an image captured while maintaining a constant distance from the structure. Control device.

14. A movement information generation unit that generates movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image, A damage detection unit detects damaged portions of a structure from an image taken by the moving body that has moved based on the generated movement information, the image which includes the structure in at least a portion of the image; Equipped with, The operation of specifying the area includes selecting the span, which is the distance between the supports of the structure to be inspected by the mobile body. Control device.

15. A movement information generation unit that generates movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image, A damage detection unit detects damaged portions of a structure from an image taken by the moving body that has moved based on the generated movement information, the image which includes the structure in at least a portion of the image; Equipped with, The generated movement information includes a movement path that repeatedly rises and falls relative to the imaging target surface, or moves back and forth along the imaging target surface. Control device.

16. To generate movement information for moving a moving object in response to an operation that specifies a region on a control image generated from an captured image, Based on the generated movement information, the moving body moves and, from an image that includes the structure in at least a portion of the image, the damaged portion of the structure is detected. Have the computer run it, The aforementioned operating image is generated from an image captured while maintaining a constant distance from the structure. program.

17. To generate movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image, Based on the generated movement information, the moving body moves and, from an image that includes the structure in at least a portion of the image, the damaged portion of the structure is detected. Have the computer run it, The operation of specifying the area includes selecting the span, which is the distance between the supports of the structure to be inspected by the mobile body. program.

18. To generate movement information for moving a moving object in response to an operation that specifies a region on an operation image generated from an captured image, Based on the generated movement information, the moving body moves and, from an image that includes the structure in at least a portion of the image, the damaged portion of the structure is detected. Have the computer run it, The generated movement information includes a movement path that repeatedly rises and falls relative to the imaging target surface, or moves back and forth along the imaging target surface. program.

Images